Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-218323375
LGTM pending jenkins
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12616
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-218324187
Thanks! Merging to master and branch 2.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-218323829
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-218323831
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-218323662
**[Test build #58285 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58285/consoleFull)**
for PR 12616 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-218307642
**[Test build #58285 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58285/consoleFull)**
for PR 12616 at commit
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/12616#discussion_r62741347
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
---
@@ -33,12 +35,14 @@ import
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/12616#discussion_r62741272
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
---
@@ -444,6 +445,79 @@ class FileStreamSourceSuite extends
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/12616#discussion_r62741054
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
---
@@ -97,21 +102,30 @@ class FileStreamSource(
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/12616#discussion_r62740742
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
---
@@ -97,21 +102,30 @@ class FileStreamSource(
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/12616#discussion_r62739836
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
---
@@ -444,6 +445,79 @@ class FileStreamSourceSuite extends
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/12616#discussion_r62737973
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
---
@@ -33,12 +35,14 @@ import
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/12616#discussion_r62738019
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
---
@@ -97,21 +102,30 @@ class FileStreamSource(
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/12616#discussion_r62737571
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
---
@@ -97,21 +102,30 @@ class FileStreamSource(
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/12616#discussion_r62737506
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
---
@@ -97,21 +102,30 @@ class FileStreamSource(
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-218260826
@yhuai @marmbrus
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-218041367
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-218041368
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-218041227
**[Test build #58191 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58191/consoleFull)**
for PR 12616 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-218040367
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-218040366
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-218040169
**[Test build #58189 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58189/consoleFull)**
for PR 12616 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-218030218
**[Test build #58191 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58191/consoleFull)**
for PR 12616 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-218029616
**[Test build #58189 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58189/consoleFull)**
for PR 12616 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-215576137
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-215576141
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-215575885
**[Test build #57275 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57275/consoleFull)**
for PR 12616 at commit
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-215562885
@marmbrus Please take a look once again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/12616#discussion_r61498520
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala
---
@@ -439,6 +442,105 @@ class FileStreamSourceSuite extends
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/12616#discussion_r61498450
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
---
@@ -98,20 +109,49 @@ class FileStreamSource(
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/12616#discussion_r61498388
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
---
@@ -98,20 +109,49 @@ class FileStreamSource(
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/12616#discussion_r61498403
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
---
@@ -98,20 +109,49 @@ class FileStreamSource(
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/12616#discussion_r61498250
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
---
@@ -55,10 +62,12 @@ class FileStreamSource(
*/
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/12616#discussion_r61498117
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
---
@@ -55,10 +62,12 @@ class FileStreamSource(
*/
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/12616#discussion_r61498063
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -175,25 +175,11 @@ case class DataSource(
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-215553704
**[Test build #57275 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57275/consoleFull)**
for PR 12616 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-215297169
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-215297170
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-215296860
**[Test build #57200 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57200/consoleFull)**
for PR 12616 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-215282807
**[Test build #57200 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57200/consoleFull)**
for PR 12616 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-215281704
**[Test build #57199 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57199/consoleFull)**
for PR 12616 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-215281708
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-215281707
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-215281481
**[Test build #57199 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57199/consoleFull)**
for PR 12616 at commit
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-214622436
This is also supported by hadoop. For example, we can pass the globs in
sc.textFile("/dir/*/*"). I was wondering, if it will be implemented again.
---
If your
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-213567878
I.e. something like `/dir/*/*`. We either do it in DataSource or
HDFSFileCatalog.
---
If your project is set up for it, you can reply to this email and have your
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-213561278
Globbing? Can you explain that a bit. Code links?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-213559435
Why not just support globbing like we do in batch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-213550362
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-213550360
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-213550352
**[Test build #56719 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56719/consoleFull)**
for PR 12616 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-213550024
**[Test build #56719 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56719/consoleFull)**
for PR 12616 at commit
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/12616#issuecomment-213549288
@marmbrus @zsxwing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/12616
[SPARK-14837][SQL][STREAMING] Added support in file stream source for
reading new files added to subdirs
## What changes were proposed in this pull request?
Currently, file stream source can only
55 matches
Mail list logo
