Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20611#discussion_r207320333
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -303,94 +303,44 @@ case class LoadDataCommand
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@gatorsmile Yes there is a change in the behavior, As i mentioned above in
descriptions now we will be able to support wildcard even in the folder level
for local file systems. Previous
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20611#discussion_r206961528
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -303,94 +303,44 @@ case class LoadDataCommand
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20611#discussion_r206411960
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -303,94 +303,44 @@ case class LoadDataCommand
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20611#discussion_r206037303
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -303,94 +303,44 @@ case class LoadDataCommand
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
ok, i missed it, let me have a look. Thanks
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/21775
@HyukjinKwon @gatorsmile All issues has been addressed, please let me know
how this patch looks like. Thanks
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16677#discussion_r204362254
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala
---
@@ -231,6 +231,12 @@ object
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16677#discussion_r204361301
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/TakeOrderedAndProjectSuite.scala
---
@@ -22,6 +22,7 @@ import scala.util.Random
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21775#discussion_r204287713
--- Diff: docs/sql-programming-guide.md ---
@@ -1843,6 +1843,7 @@ working with timestamps in `pandas_udf`s to get the
best performance, see
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21775#discussion_r204286227
--- Diff: docs/sql-programming-guide.md ---
@@ -1843,6 +1843,7 @@ working with timestamps in `pandas_udf`s to get the
best performance, see
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/21775
@HyukjinKwon
seems to be a limitation as of now even from hive, better we can follow the
hive behavior unless the limitation has been resolved from hive.
>> Hive-2526 is th
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/21775
sure, i will update the PR based on the comments, Thanks for suggestions.
---
-
To unsubscribe, e-mail: reviews-unsubscr
GitHub user sujith71955 opened a pull request:
https://github.com/apache/spark/pull/21775
[SPARK-24812][SQL] Last Access Time in the table description is not valid
## What changes were proposed in this pull request?
Last Access Time will always displayed wrong date Wed Dec 31 15
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@srowen
The only possible objection I can see is that the behavior for paths that
contains a ? or * would now change :- There is no behavior change currently,
currently user cannot
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20611#discussion_r202429058
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -303,94 +303,44 @@ case class LoadDataCommand
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@srowen Thanks for the review. all comments has been addressed from my
side. let me know for any clarifications
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20611#discussion_r202255494
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -303,94 +303,49 @@ case class LoadDataCommand
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20611#discussion_r202255324
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -303,94 +303,49 @@ case class LoadDataCommand
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20611#discussion_r202255185
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
---
@@ -1912,11 +1912,58 @@ class SQLQuerySuite extends
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20611#discussion_r202255230
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -303,94 +303,49 @@ case class LoadDataCommand
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
This PR is open from many days, we need this fix in our version code also,
can someone please review and provide me suggestions if any
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@HyukjinKwon Thanks for the re look.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20611#discussion_r196670750
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -303,94 +303,50 @@ case class LoadDataCommand
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@gatorsmile @jiangxb1987 @wzhfy @HyukjinKwon @vinodkc please review and
let me know for any suggestions. i think all the gaps which we discussed as
part of the PR is been addressed
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@kevinyu98 Now my PR addresses the scenario what you mentioned.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
As @kevinyu98 mentioned below usecase where '?' is been used in the load
command will fail as when we create a Path instance with uri , the chars
followed by ? will be removed as part
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@kevinyu98 Even below kind of queries cannot run as exist() api while
validation cannot identify wild chars, this is the reason in my PR i am using
globStatus() API
load data inpath 'hdfs
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@kevinyu98val uri = Utils.resolveURI(path) , if you wont mention any
scheme , this api will return
---
-
To unsubscribe
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@kevinyu98 Thanks for you feedback, i will test once all my scenario with
your fix, but here one more point i need to mention is like my fix has also has
impact in the logic of local path
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@kevinyu98
Spark supports wildcards characters in both local and non local scenarios,
the only restriction spark imposes is using wildcards in folder names of the
local file-system path
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
Currently all the load command use-cases with wildcard in a load path are
blocked for the users
---
-
To unsubscribe, e
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@wzhfy @HyukjinKwon @jiangxb1987 @gatorsmile @vinodkc, Please let me know
for any suggestions regarding this PR. I covered all scenarios related to this
PR. Working fine
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@wzhfy @HyukjinKwon @jiangxb1987 @gatorsmile @vinodkc
All test-cases has been passed, please let me know for any other
suggestions.
With current solution following improvement has
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20611#discussion_r181543985
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -304,45 +304,14 @@ case class LoadDataCommand
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@wzhfy i am working on it, when i ran locally few test-cases were failing,
correcting the same. once done i will update. Thanks
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20611#discussion_r179030611
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -385,7 +385,9 @@ case class LoadDataCommand
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20611#discussion_r179030399
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -385,7 +385,9 @@ case class LoadDataCommand
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@wzhfy Actually its not a behavior change , i think this condition was
working in other spark 2.x versions. Some time back a PR is been added for
load command hdfs path validation, in this PR
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@jiangxb1987 This usecase has dependency on hdfs file path and i could not
find any testcases which validates or executes any scenarios against hdfs file
path. can you please suggest me some
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@wzhfy updated as per comments
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@wzhfy @gatorsmile @jiangxb1987 due to some inconsistency in the previous
test report due to inconsistent hdfs file paths i had updated it, i added the
test results in one more section Test
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@wzhfy @gatorsmile @jiangxb1987 updated the code based on comments, also
added one more use-case to verify the normal case of data-loading without
specifying wildcard in file path. let me
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20611#discussion_r174868215
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -385,8 +385,12 @@ case class LoadDataCommand
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20611#discussion_r174791662
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -385,8 +385,12 @@ case class LoadDataCommand
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
Yeah there is a bit confusing, i updated the snapshot, data validation and
consistency i verified
---
-
To unsubscribe, e
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@wzhfy @gatorsmile @jiangxb1987 updated the test results and PR title based
on zhenhuas comment, please review and let me know if still any improvement
area is present for this particular PR
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@wzhfy sure will do it. Thanks for suggestions
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
@wzhfy
Hive supports specifying wildcard in both file and folder level, please
find test report attached below
Use-case related to Hive File level wild card support
seems to be an invalid failure
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/20611
Please review and let me know for any suggestions. Thanks
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
GitHub user sujith71955 opened a pull request:
https://github.com/apache/spark/pull/20611
[SPARK-23425][SQL]When wild card is been used in load command system is
throwing analysis exception
## What changes were proposed in this pull request?
A validaton logic is been added
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/19823
Thanks all for the review and guidance.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19823#discussion_r154187840
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
---
@@ -2392,5 +2392,14 @@ abstract class DDLSuite extends
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19823#discussion_r153693386
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
---
@@ -2392,5 +2392,13 @@ abstract class DDLSuite extends
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19823#discussion_r153693359
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
---
@@ -2392,5 +2392,13 @@ abstract class DDLSuite extends
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/19823
@gatorsmile @HyukjinKwon @srowen Please review as i modified the code as
per provided comments. thanks
---
-
To unsubscribe
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/19823
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/19823
Basically this validation stands good for both cases where scheme can come
as null and not null, i will update the logic as Sean told. Thanks
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/19823
Thanks for the comments guys, i am working on it.,will update the PR based
on comments.
---
-
To unsubscribe, e-mail
Github user sujith71955 commented on the issue:
https://github.com/apache/spark/pull/19823
