Github user misutoth closed the pull request at:
https://github.com/apache/spark/pull/22331
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user misutoth commented on the issue:
https://github.com/apache/spark/pull/22331
So I am considering this as the recommended way to read a file sink's
output. If there is a need to include the protocol in this PR as an alternative
we can still reopen
Github user misutoth commented on the issue:
https://github.com/apache/spark/pull/22532
Many thanks for the feedback. I will list the test scenarios that I had in
mind and collected while I implemented this item.
And sorry about the failure, seems like I did not rerun all
Github user misutoth commented on the issue:
https://github.com/apache/spark/pull/22331
@lw-lin , @marmbrus , in the meantime I found that you have been discussing
about having deterministic file names in a
[PR](https://github.com/apache/spark/pull/16987#issuecomment-282094165
Github user misutoth commented on the issue:
https://github.com/apache/spark/pull/22532
@janewangfb, @gatorsmile could you please possibly review this change?
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user misutoth commented on the issue:
https://github.com/apache/spark/pull/22331
@rxin could you please look into this change?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
GitHub user misutoth opened a pull request:
https://github.com/apache/spark/pull/22532
[SPARK-20845][SQL] Support specification of column names in INSERT INTO
command.
## What changes were proposed in this pull request?
One can specify a list of columns for an INSERT
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/22331#discussion_r219521421
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StagingFileCommitProtocol.scala
---
@@ -0,0 +1,141 @@
+/*
+ * Licensed
GitHub user misutoth opened a pull request:
https://github.com/apache/spark/pull/22331
Tests for idempotency of FileStreamSink - Work in Progress
## What changes were proposed in this pull request?
Reproduce File Sink duplication in driver failure scenario to help
Github user misutoth closed the pull request at:
https://github.com/apache/spark/pull/21591
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
GitHub user misutoth opened a pull request:
https://github.com/apache/spark/pull/21591
[SQL][WIP] Added column name listing option to INSERT INTO
## What changes were proposed in this pull request?
Added possibility to specify column list to INSERT INTO. Source column list
Github user misutoth commented on the issue:
https://github.com/apache/spark/pull/20674
Seems like this functionality is not missed very painfully, so closing the
PR.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user misutoth closed the pull request at:
https://github.com/apache/spark/pull/20674
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user misutoth commented on the issue:
https://github.com/apache/spark/pull/20853
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20853#discussion_r175664716
--- Diff: core/src/main/scala/org/apache/spark/deploy/DependencyUtils.scala
---
@@ -137,16 +138,32 @@ private[deploy] object DependencyUtils {
def
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20853#discussion_r175576249
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -245,6 +245,19 @@ object SparkSubmit extends CommandLineUtils with
Logging
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20853#discussion_r175575718
--- Diff: core/src/main/scala/org/apache/spark/deploy/DependencyUtils.scala
---
@@ -137,16 +137,29 @@ private[deploy] object DependencyUtils {
def
Github user misutoth commented on the issue:
https://github.com/apache/spark/pull/20853
> Maybe just let the exception propagate? That's what a lot of this code
does... then you don't need to change this file at all.
@vanzin I want to present an error on the CLI. This is w
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20853#discussion_r175546041
--- Diff:
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala ---
@@ -105,11 +105,17 @@ class SparkSubmitSuite
// Necessary
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20853#discussion_r175543984
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -245,6 +245,19 @@ object SparkSubmit extends CommandLineUtils with
Logging
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20853#discussion_r175541331
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -245,6 +245,19 @@ object SparkSubmit extends CommandLineUtils with
Logging
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20853#discussion_r175540130
--- Diff:
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala ---
@@ -606,9 +612,12 @@ class SparkSubmitSuite
}
test
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20853#discussion_r175539256
--- Diff:
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala ---
@@ -105,11 +105,17 @@ class SparkSubmitSuite
// Necessary
GitHub user misutoth opened a pull request:
https://github.com/apache/spark/pull/20853
[SPARK-23729][SS] Glob resolution is done without the fragment part which
is meant to be the remote name
## What changes were proposed in this pull request?
Firstly, glob resolution
Github user misutoth commented on the issue:
https://github.com/apache/spark/pull/20674
@rxin @marmbrus @gatorsmile do you think this might be useful? I would
appreciate your comments.
---
-
To unsubscribe, e-mail
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r171992166
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -1313,131 +1313,165 @@ object functions
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r171991788
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -1500,31 +1534,35 @@ object functions
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r171991691
--- Diff: python/pyspark/sql/functions.py ---
@@ -173,16 +172,26 @@ def _():
_functions_2_1 = {
# unary math functions
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r171991733
--- Diff: python/pyspark/sql/functions.py ---
@@ -173,16 +172,26 @@ def _():
_functions_2_1 = {
# unary math functions
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r171991647
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
---
@@ -512,7 +529,11 @@ case class Rint(child
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20674#discussion_r171386636
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -2239,6 +2239,34 @@ class Dataset[T] private[sql
Github user misutoth commented on the issue:
https://github.com/apache/spark/pull/20618
I included java.lang.Math references in functions.R
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user misutoth commented on the issue:
https://github.com/apache/spark/pull/20618
As discussed in email R documentation is reorganized and math functions are
grouped as part of SPARK-20889. Because of this grouping I dont think this
change is really applicable on R. @srowen
Github user misutoth commented on the issue:
https://github.com/apache/spark/pull/20618
@felixcheung, I have started a mail thread on d...@spark.apache.org with
title _Help needed in R documentation generation_ because I did not feel it is
directly related to this PR. Thanks for your
Github user misutoth commented on the issue:
https://github.com/apache/spark/pull/20618
Sorry, I missed these comments. As I understood we fix all of them here. I
am just struggling with the R documentation: it seems the generated doc is
incorrect even if I just take the latest
GitHub user misutoth opened a pull request:
https://github.com/apache/spark/pull/20674
[SPARK-23465][SQL] Introduce new function to rename columns using an
algoritm
## What changes were proposed in this pull request?
Add an additional convenient method to rename multiple
Github user misutoth commented on the issue:
https://github.com/apache/spark/pull/20618
Sure, lets do that, no problem.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user misutoth commented on the issue:
https://github.com/apache/spark/pull/20618
Looking back I guess we can expect a couple of comments on R and Python
side too, though I will target a lower number of them. :) So I am a little bit
favoring moving functions.* update
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r169313329
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
---
@@ -548,7 +579,14 @@ case class Sqrt(child
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r169313008
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
---
@@ -262,6 +272,10 @@ case class Cos(child
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r169312315
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
---
@@ -262,6 +272,10 @@ case class Cos(child
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r169311993
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
---
@@ -252,7 +255,14 @@ case class Ceil(child
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r169311232
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2005,71 +1967,63 @@ object functions {
def signum(columnName: String
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r169291756
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -1313,131 +1313,165 @@ object functions
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r169287876
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
---
@@ -521,7 +542,15 @@ case class Signum(child
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r169287270
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
---
@@ -262,6 +273,11 @@ case class Cos(child
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r169187041
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
---
@@ -804,7 +858,6 @@ case class Pow(left
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r169186706
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
---
@@ -538,8 +559,14 @@ case class Sinh(child
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r169186651
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
---
@@ -512,16 +522,27 @@ case class Rint(child
Github user misutoth commented on the issue:
https://github.com/apache/spark/pull/20618
Thanks @srowen and @HyukjinKwon for your comments so far ...
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r169100931
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
---
@@ -770,7 +837,14 @@ case class Unhex(child
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r168578156
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -1313,131 +1313,178 @@ object functions
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r168531352
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
---
@@ -770,7 +837,14 @@ case class Unhex(child
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r168526385
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -1313,131 +1313,178 @@ object functions
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r168525504
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -1313,131 +1313,178 @@ object functions
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20618#discussion_r168521316
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
---
@@ -262,6 +285,11 @@ case class Cos(child
GitHub user misutoth opened a pull request:
https://github.com/apache/spark/pull/20618
[SPARK-23329][SQL] Fix documentation of trigonometric functions
## What changes were proposed in this pull request?
Provide more details in trigonometric function documentations
GitHub user misutoth opened a pull request:
https://github.com/apache/lucene-solr/pull/311
SOLR-11873: Use a time based expiration cache for one off HDFS FileSyâ¦
â¦stem instances in all functions.
You can merge this pull request into a Git repository by running:
$ git pull
58 matches
Mail list logo
