[spark] branch master updated (8a4378c -> a4382f7)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8a4378c [SPARK-29686][ML] LinearSVC should persist instances if needed add a4382f7 [SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds No new revisions were added by this update. Summary of changes: .../spark/unsafe/types/CalendarInterval.java | 39 +++-- .../spark/unsafe/types/CalendarIntervalSuite.java | 65 +- .../sql/catalyst/expressions/UnsafeArrayData.java | 5 +- .../spark/sql/catalyst/expressions/UnsafeRow.java | 5 +- .../catalyst/expressions/codegen/UnsafeWriter.java | 5 +- .../apache/spark/sql/vectorized/ColumnVector.java | 21 --- .../catalyst/analysis/StreamingJoinHelper.scala| 3 +- .../sql/catalyst/expressions/TimeWindow.scala | 3 +- .../expressions/collectionOperations.scala | 37 .../catalyst/expressions/datetimeExpressions.scala | 14 ++--- .../spark/sql/catalyst/expressions/hash.scala | 8 ++- .../catalyst/expressions/intervalExpressions.scala | 4 +- .../spark/sql/catalyst/expressions/literals.scala | 2 +- .../spark/sql/catalyst/util/DateTimeUtils.scala| 6 +- .../spark/sql/catalyst/util/IntervalUtils.scala| 30 +- .../org/apache/spark/sql/RandomDataGenerator.scala | 3 +- .../sql/catalyst/analysis/TypeCoercionSuite.scala | 2 +- .../spark/sql/catalyst/expressions/CastSuite.scala | 12 ++-- .../expressions/DateExpressionsSuite.scala | 42 +- .../expressions/ExpressionSQLBuilderSuite.scala| 6 +- .../expressions/IntervalExpressionsSuite.scala | 32 ++- .../expressions/LiteralExpressionSuite.scala | 2 +- .../catalyst/expressions/LiteralGenerator.scala| 4 +- .../catalyst/optimizer/FilterPushdownSuite.scala | 8 +-- .../optimizer/LeftSemiAntiJoinPushDownSuite.scala | 1 - .../catalyst/parser/ExpressionParserSuite.scala| 9 +-- .../sql/catalyst/util/DateTimeUtilsSuite.scala | 34 +-- .../sql/catalyst/util/IntervalUtilsSuite.scala | 42 +++--- .../spark/sql/catalyst/util/UnsafeArraySuite.scala | 3 +- .../execution/vectorized/ColumnVectorUtils.java| 3 +- .../execution/vectorized/WritableColumnVector.java | 7 ++- .../org/apache/spark/sql/execution/Columnar.scala | 3 +- .../spark/sql/execution/streaming/Triggers.scala | 3 +- .../sql-tests/results/ansi/interval.sql.out| 4 +- .../resources/sql-tests/results/date_part.sql.out | 16 +++--- .../resources/sql-tests/results/datetime.sql.out | 4 +- .../resources/sql-tests/results/literals.sql.out | 4 +- .../sql-tests/results/postgreSQL/interval.sql.out | 4 +- .../sql-tests/results/postgreSQL/timestamp.sql.out | 12 ++-- .../org/apache/spark/sql/DateFunctionsSuite.scala | 16 +++--- .../scala/org/apache/spark/sql/SQLQuerySuite.scala | 14 ++--- .../execution/vectorized/ColumnarBatchSuite.scala | 28 +++--- 42 files changed, 337 insertions(+), 228 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (a4382f7 -> 14337f6)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a4382f7 [SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds add 14337f6 [SPARK-29643][SQL] ALTER TABLE/VIEW (DROP PARTITION) should look up catalog/table like v2 commands No new revisions were added by this update. Summary of changes: .../apache/spark/sql/catalyst/parser/SqlBase.g4| 4 +- .../spark/sql/catalyst/parser/AstBuilder.scala | 26 +++ .../sql/catalyst/plans/logical/statements.scala| 10 + .../spark/sql/catalyst/parser/DDLParserSuite.scala | 50 ++ .../catalyst/analysis/ResolveSessionCatalog.scala | 9 .../spark/sql/execution/SparkSqlParser.scala | 25 --- .../spark/sql/connector/DataSourceV2SQLSuite.scala | 11 + .../sql/execution/command/DDLParserSuite.scala | 39 - 8 files changed, 107 insertions(+), 67 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (14337f6 -> f53be0a)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 14337f6 [SPARK-29643][SQL] ALTER TABLE/VIEW (DROP PARTITION) should look up catalog/table like v2 commands add f53be0a [SPARK-29109][SQL][TESTS] Port window.sql (Part 3) No new revisions were added by this update. Summary of changes: .../sql-tests/inputs/postgreSQL/window_part3.sql | 451 + .../results/postgreSQL/window_part3.sql.out| 409 +++ 2 files changed, 860 insertions(+) create mode 100644 sql/core/src/test/resources/sql-tests/inputs/postgreSQL/window_part3.sql create mode 100644 sql/core/src/test/resources/sql-tests/results/postgreSQL/window_part3.sql.out - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f53be0a -> 39fff92)
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from f53be0a [SPARK-29109][SQL][TESTS] Port window.sql (Part 3)
add 39fff92 [SPARK-29452][WEBUI] Improve Storage tab tooltip
No new revisions were added by this update.
Summary of changes:
.../org/apache/spark/ui/storage/StoragePage.scala | 14 +++-
.../storage/ToolTips.scala}| 25 ++
.../apache/spark/ui/storage/StoragePageSuite.scala | 17 ++-
3 files changed, 50 insertions(+), 6 deletions(-)
copy
core/src/main/scala/org/apache/spark/{deploy/master/ApplicationState.scala =>
ui/storage/ToolTips.scala} (60%)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-29651][SQL][2.4] Fix parsing of interval seconds fraction
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 9b17a68 [SPARK-29651][SQL][2.4] Fix parsing of interval seconds
fraction
9b17a68 is described below
commit 9b17a682e6ad25f396a2c9ccfbe7c402a403ba4a
Author: Maxim Gekk
AuthorDate: Fri Nov 1 22:07:26 2019 +0800
[SPARK-29651][SQL][2.4] Fix parsing of interval seconds fraction
### What changes were proposed in this pull request?
In the PR, I propose to extract parsing of the seconds interval units to
the private method `parseNanos` in `CalendarInterval` and modify the code to
correctly parse the fractional part of the seconds unit of intervals in the
cases:
- When the fractional part has less than 9 digits
- The seconds unit is negative
This is a back port of the commit
https://github.com/apache/spark/commit/3206a9987001d78cf2f48509a93d73af86f51cfe.
### Why are the changes needed?
The changes are needed to fix the issues:
```sql
spark-sql> select interval 10.123456 seconds;
interval 10 seconds 123 microseconds
```
The correct result must be `interval 10 seconds 123 milliseconds 456
microseconds`
```sql
spark-sql> select interval -10.123456789 seconds;
interval -9 seconds -876 milliseconds -544 microseconds
```
but the whole interval should be negated, and the result must be `interval
-10 seconds -123 milliseconds -456 microseconds`, taking into account the
truncation to microseconds.
### Does this PR introduce any user-facing change?
Yes. After changes:
```sql
spark-sql> select interval 10.123456 seconds;
interval 10 seconds 123 milliseconds 456 microseconds
spark-sql> select interval -10.123456789 seconds;
interval -10 seconds -123 milliseconds -456 microseconds
```
### How was this patch tested?
By existing test suite, `literals.sql` and new tests in
`ExpressionParserSuite`.
Closes #26355 from MaxGekk/fix-interval-nanos-parsing-2.4.
Authored-by: Maxim Gekk
Signed-off-by: Wenchen Fan
---
.../spark/unsafe/types/CalendarInterval.java | 27 ++
.../spark/unsafe/types/CalendarIntervalSuite.java | 2 +-
.../catalyst/parser/ExpressionParserSuite.scala| 15 +++-
.../resources/sql-tests/results/literals.sql.out | 4 ++--
4 files changed, 39 insertions(+), 9 deletions(-)
diff --git
a/common/unsafe/src/main/java/org/apache/spark/unsafe/types/CalendarInterval.java
b/common/unsafe/src/main/java/org/apache/spark/unsafe/types/CalendarInterval.java
index 1818fef..b75a1f5 100644
---
a/common/unsafe/src/main/java/org/apache/spark/unsafe/types/CalendarInterval.java
+++
b/common/unsafe/src/main/java/org/apache/spark/unsafe/types/CalendarInterval.java
@@ -19,6 +19,7 @@ package org.apache.spark.unsafe.types;
import java.io.Serializable;
import java.util.Locale;
+import java.util.concurrent.TimeUnit;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
@@ -177,11 +178,10 @@ public final class CalendarInterval implements
Serializable {
long minutes = toLongWithRange("minute", m.group(4), 0, 59);
long seconds = toLongWithRange("second", m.group(5), 0, 59);
// Hive allow nanosecond precision interval
-String nanoStr = m.group(7) == null ? null : (m.group(7) +
"0").substring(0, 9);
-long nanos = toLongWithRange("nanosecond", nanoStr, 0L, 9L);
+long secondsFraction = parseNanos(m.group(7), seconds < 0);
result = new CalendarInterval(0, sign * (
days * MICROS_PER_DAY + hours * MICROS_PER_HOUR + minutes *
MICROS_PER_MINUTE +
- seconds * MICROS_PER_SECOND + nanos / 1000L));
+ seconds * MICROS_PER_SECOND + secondsFraction));
} catch (Exception e) {
throw new IllegalArgumentException(
"Error parsing interval day-time string: " + e.getMessage(), e);
@@ -270,8 +270,8 @@ public final class CalendarInterval implements Serializable
{
} else if (parts.length == 2) {
long seconds = parts[0].equals("") ? 0L : toLongWithRange("second",
parts[0],
Long.MIN_VALUE / MICROS_PER_SECOND, Long.MAX_VALUE /
MICROS_PER_SECOND);
- long nanos = toLongWithRange("nanosecond", parts[1], 0L, 9L);
- return seconds * MICROS_PER_SECOND + nanos / 1000L;
+ long secondsFraction = parseNanos(parts[1], seconds < 0);
+ return seconds * MICROS_PER_SECOND + secondsFraction;
} else {
throw new IllegalArgumentException(
@@ -357,4 +357,21 @@ public final class CalendarInterval implements
Serializable {
sb.append(' ').append(value).append(' ').append(unit).append('s');
}
}
+
+ // Parses a string with nanoseconds, truncates the result and returns
microseconds
+ private stati
[spark] branch master updated (39fff92 -> 1e1b730)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 39fff92 [SPARK-29452][WEBUI] Improve Storage tab tooltip add 1e1b730 [MINOR][PYSPARK][DOCS] Fix typo in example documentation No new revisions were added by this update. Summary of changes: python/pyspark/sql/context.py| 2 +- python/pyspark/sql/dataframe.py | 112 +-- python/pyspark/sql/readwriter.py | 2 +- python/pyspark/sql/session.py| 20 +++ python/pyspark/sql/streaming.py | 9 ++-- 5 files changed, 78 insertions(+), 67 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (39fff92 -> 1e1b730)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 39fff92 [SPARK-29452][WEBUI] Improve Storage tab tooltip add 1e1b730 [MINOR][PYSPARK][DOCS] Fix typo in example documentation No new revisions were added by this update. Summary of changes: python/pyspark/sql/context.py| 2 +- python/pyspark/sql/dataframe.py | 112 +-- python/pyspark/sql/readwriter.py | 2 +- python/pyspark/sql/session.py| 20 +++ python/pyspark/sql/streaming.py | 9 ++-- 5 files changed, 78 insertions(+), 67 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1e1b730 -> 28ccd31)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1e1b730 [MINOR][PYSPARK][DOCS] Fix typo in example documentation add 28ccd31 [SPARK-29611][WEBUI] Sort Kafka metadata by the number of messages No new revisions were added by this update. Summary of changes: .../apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala| 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
