[GitHub] AmplabJenkins removed a comment on issue #23708: [SPARK-26745][SQL][BRANCH-2.4] Revert count optimization in JSON datasource by SPARK-24959

[GitBox]

AmplabJenkins removed a comment on issue #23708: [SPARK-26745][SQL][BRANCH-2.4] 
Revert count optimization in JSON datasource by SPARK-24959 
URL: https://github.com/apache/spark/pull/23708#issuecomment-459249904
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7621/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23708: [SPARK-26745][SQL][BRANCH-2.4] Revert count optimization in JSON datasource by SPARK-24959

[GitBox]

AmplabJenkins removed a comment on issue #23708: [SPARK-26745][SQL][BRANCH-2.4] 
Revert count optimization in JSON datasource by SPARK-24959 
URL: https://github.com/apache/spark/pull/23708#issuecomment-459249898
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] SparkQA commented on issue #23601: [SPARK-26673][SQL] File source V2 writes: create framework and migrate ORC

[GitBox]

SparkQA commented on issue #23601: [SPARK-26673][SQL] File source V2 writes: 
create framework and migrate ORC
URL: https://github.com/apache/spark/pull/23601#issuecomment-459250376
 
 
   **[Test build #101937 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101937/testReport)**
 for PR 23601 at commit 
[`8a6a9b6`](https://github.com/apache/spark/commit/8a6a9b6fb4292da457d5395dbf2fa05736dea397).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] SparkQA commented on issue #23708: [SPARK-26745][SQL][BRANCH-2.4] Revert count optimization in JSON datasource by SPARK-24959

[GitBox]

SparkQA commented on issue #23708: [SPARK-26745][SQL][BRANCH-2.4] Revert count 
optimization in JSON datasource by SPARK-24959 
URL: https://github.com/apache/spark/pull/23708#issuecomment-459250368
 
 
   **[Test build #101936 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101936/testReport)**
 for PR 23708 at commit 
[`84cb068`](https://github.com/apache/spark/commit/84cb0686aaa05c9f8710efdc5d119ebfa47f986c).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23601: [SPARK-26673][SQL] File source V2 writes: create framework and migrate ORC

[GitBox]

AmplabJenkins removed a comment on issue #23601: [SPARK-26673][SQL] File source 
V2 writes: create framework and migrate ORC
URL: https://github.com/apache/spark/pull/23601#issuecomment-459249925
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #22996: [SPARK-25997][ML]add Python example code for Power Iteration Clustering in spark.ml

[GitBox]

AmplabJenkins removed a comment on issue #22996: [SPARK-25997][ML]add Python 
example code for Power Iteration Clustering in spark.ml
URL: https://github.com/apache/spark/pull/22996#issuecomment-459249948
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23601: [SPARK-26673][SQL] File source V2 writes: create framework and migrate ORC

[GitBox]

AmplabJenkins removed a comment on issue #23601: [SPARK-26673][SQL] File source 
V2 writes: create framework and migrate ORC
URL: https://github.com/apache/spark/pull/23601#issuecomment-459249933
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7622/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #22996: [SPARK-25997][ML]add Python example code for Power Iteration Clustering in spark.ml

[GitBox]

AmplabJenkins removed a comment on issue #22996: [SPARK-25997][ML]add Python 
example code for Power Iteration Clustering in spark.ml
URL: https://github.com/apache/spark/pull/22996#issuecomment-459249952
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7623/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] asfgit closed pull request #23664: [MINOR][DOCS] Add a note that 'spark.executor.pyspark.memory' is dependent on 'resource'

[GitBox]

asfgit closed pull request #23664: [MINOR][DOCS] Add a note that 
'spark.executor.pyspark.memory' is dependent on 'resource'
URL: https://github.com/apache/spark/pull/23664
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #22996: [SPARK-25997][ML]add Python example code for Power Iteration Clustering in spark.ml

[GitBox]

AmplabJenkins commented on issue #22996: [SPARK-25997][ML]add Python example 
code for Power Iteration Clustering in spark.ml
URL: https://github.com/apache/spark/pull/22996#issuecomment-459249952
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7623/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23601: [SPARK-26673][SQL] File source V2 writes: create framework and migrate ORC

[GitBox]

AmplabJenkins commented on issue #23601: [SPARK-26673][SQL] File source V2 
writes: create framework and migrate ORC
URL: https://github.com/apache/spark/pull/23601#issuecomment-459249933
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7622/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23708: [SPARK-26745][SQL][BRANCH-2.4] Revert count optimization in JSON datasource by SPARK-24959

[GitBox]

AmplabJenkins commented on issue #23708: [SPARK-26745][SQL][BRANCH-2.4] Revert 
count optimization in JSON datasource by SPARK-24959 
URL: https://github.com/apache/spark/pull/23708#issuecomment-459249898
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #22996: [SPARK-25997][ML]add Python example code for Power Iteration Clustering in spark.ml

[GitBox]

AmplabJenkins commented on issue #22996: [SPARK-25997][ML]add Python example 
code for Power Iteration Clustering in spark.ml
URL: https://github.com/apache/spark/pull/22996#issuecomment-459249948
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23601: [SPARK-26673][SQL] File source V2 writes: create framework and migrate ORC

[GitBox]

AmplabJenkins commented on issue #23601: [SPARK-26673][SQL] File source V2 
writes: create framework and migrate ORC
URL: https://github.com/apache/spark/pull/23601#issuecomment-459249925
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23708: [SPARK-26745][SQL][BRANCH-2.4] Revert count optimization in JSON datasource by SPARK-24959

[GitBox]

AmplabJenkins commented on issue #23708: [SPARK-26745][SQL][BRANCH-2.4] Revert 
count optimization in JSON datasource by SPARK-24959 
URL: https://github.com/apache/spark/pull/23708#issuecomment-459249904
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7621/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] HyukjinKwon commented on issue #23664: [MINOR][DOCS] Add a note that 'spark.executor.pyspark.memory' is dependent on 'resource'

[GitBox]

HyukjinKwon commented on issue #23664: [MINOR][DOCS] Add a note that 
'spark.executor.pyspark.memory' is dependent on 'resource'
URL: https://github.com/apache/spark/pull/23664#issuecomment-459249166
 
 
   Merged to master.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] HyukjinKwon opened a new pull request #23708: [SPARK-26745][SQL][BRANCH-2.4] Revert count optimization in JSON datasource by SPARK-24959

[GitBox]

HyukjinKwon opened a new pull request #23708: [SPARK-26745][SQL][BRANCH-2.4] 
Revert count optimization in JSON datasource by SPARK-24959 
URL: https://github.com/apache/spark/pull/23708
 
 
   ## What changes were proposed in this pull request?
   
   This PR reverts JSON count optimization part of #21909.
   
   We cannot distinguish the cases below without parsing:
   
   ```
   [{...}, {...}]
   ```
   
   ```
   []
   ```
   
   ```
   {...}
   ```
   
   ```bash
   # empty string
   ```
   
   when we `count()`. One line (input: IN) can be, 0 record, 1 record and 
multiple records and this is dependent on each input.
   
   See also https://github.com/apache/spark/pull/23665#discussion_r251276720.
   
   ## How was this patch tested?
   
   Manually tested.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] gatorsmile commented on issue #23692: [SPARK-26448][SQL][followup] should not normalize grouping expressions for final aggregate

[GitBox]

gatorsmile commented on issue #23692: [SPARK-26448][SQL][followup] should not 
normalize grouping expressions for final aggregate
URL: https://github.com/apache/spark/pull/23692#issuecomment-459248742
 
 
   LGTM since the current hack will be removed soon.
   
   On Wed, Jan 30, 2019 at 11:47 PM Wenchen Fan 
   wrote:
   
   > *@cloud-fan* commented on this pull request.
   > --
   >
   > In
   > 
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
   > :
   >
   > > @@ -331,8 +332,17 @@ abstract class SparkStrategies extends 
QueryPlanner[SparkPlan] {
   >
   >  val stateVersion = 
conf.getConf(SQLConf.STREAMING_AGGREGATION_STATE_FORMAT_VERSION)
   >
   > +// Ideally this should be done in `NormalizeFloatingNumbers`, but 
we do it here because
   > +// `groupingExpressions` is not extracted during logical phase.
   >
   > yea
   >
   > —
   > You are receiving this because you were mentioned.
   >
   >
   > Reply to this email directly, view it on GitHub
   > , or mute
   > the thread
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] cloud-fan closed pull request #23699: [FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods and override toString method in Avro

[GitBox]

cloud-fan closed pull request #23699: [FollowUp][SPARK-26716][SPARK-26765][SQL] 
Clean up schema validation methods and override toString method in Avro
URL: https://github.com/apache/spark/pull/23699
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] cloud-fan commented on a change in pull request #23692: [SPARK-26448][SQL][followup] should not normalize grouping expressions for final aggregate

[GitBox]

cloud-fan commented on a change in pull request #23692: 
[SPARK-26448][SQL][followup] should not normalize grouping expressions for 
final aggregate
URL: https://github.com/apache/spark/pull/23692#discussion_r252562555
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
 ##
 @@ -331,8 +332,17 @@ abstract class SparkStrategies extends 
QueryPlanner[SparkPlan] {
 
 val stateVersion = 
conf.getConf(SQLConf.STREAMING_AGGREGATION_STATE_FORMAT_VERSION)
 
+// Ideally this should be done in `NormalizeFloatingNumbers`, but we 
do it here because
+// `groupingExpressions` is not extracted during logical phase.
 
 Review comment:
   yea


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] cloud-fan commented on issue #23699: [FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods and override toString method in Avro

[GitBox]

cloud-fan commented on issue #23699: [FollowUp][SPARK-26716][SPARK-26765][SQL] 
Clean up schema validation methods and override toString method in Avro
URL: https://github.com/apache/spark/pull/23699#issuecomment-459248060
 
 
   thanks, merging to master!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] gatorsmile commented on a change in pull request #23692: [SPARK-26448][SQL][followup] should not normalize grouping expressions for final aggregate

[GitBox]

gatorsmile commented on a change in pull request #23692: 
[SPARK-26448][SQL][followup] should not normalize grouping expressions for 
final aggregate
URL: https://github.com/apache/spark/pull/23692#discussion_r252560399
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
 ##
 @@ -331,8 +332,17 @@ abstract class SparkStrategies extends 
QueryPlanner[SparkPlan] {
 
 val stateVersion = 
conf.getConf(SQLConf.STREAMING_AGGREGATION_STATE_FORMAT_VERSION)
 
+// Ideally this should be done in `NormalizeFloatingNumbers`, but we 
do it here because
+// `groupingExpressions` is not extracted during logical phase.
 
 Review comment:
   This will be refactored after 
https://issues.apache.org/jira/browse/SPARK-25914?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] HyukjinKwon commented on a change in pull request #23667: [SPARK-26745][SQL] Revert count optimization in JSON datasource by SPARK-24959

[GitBox]

HyukjinKwon commented on a change in pull request #23667: [SPARK-26745][SQL] 
Revert count optimization in JSON datasource by SPARK-24959
URL: https://github.com/apache/spark/pull/23667#discussion_r252559932
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala
 ##
 @@ -188,11 +188,19 @@ class UnivocityParser(
 }
   }
 
+  private val doParse = if (requiredSchema.nonEmpty) {
+(input: String) => convert(tokenizer.parseLine(input))
+  } else {
+// If `columnPruning` enabled and partition attributes scanned only,
+// `schema` gets empty.
+(_: String) => InternalRow.empty
 
 Review comment:
   Yes, it does for CSV when multiline is off and, for miltiline mode it 
executes a different code path.
   
   `UnivocityParser.parseStream` -> `UnivocityParser.convert`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] HyukjinKwon commented on a change in pull request #23667: [SPARK-26745][SQL] Revert count optimization in JSON datasource by SPARK-24959

[GitBox]

HyukjinKwon commented on a change in pull request #23667: [SPARK-26745][SQL] 
Revert count optimization in JSON datasource by SPARK-24959
URL: https://github.com/apache/spark/pull/23667#discussion_r252559932
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala
 ##
 @@ -188,11 +188,19 @@ class UnivocityParser(
 }
   }
 
+  private val doParse = if (requiredSchema.nonEmpty) {
+(input: String) => convert(tokenizer.parseLine(input))
+  } else {
+// If `columnPruning` enabled and partition attributes scanned only,
+// `schema` gets empty.
+(_: String) => InternalRow.empty
 
 Review comment:
   Yes, it does for CSV and for miltiline mode, it executes a different code 
path.
   
   `UnivocityParser.parseStream` -> `UnivocityParser.convert`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23699: [FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods and override toString method in Avro

[GitBox]

AmplabJenkins removed a comment on issue #23699: 
[FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods 
and override toString method in Avro
URL: https://github.com/apache/spark/pull/23699#issuecomment-459244500
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23699: [FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods and override toString method in Avro

[GitBox]

AmplabJenkins commented on issue #23699: 
[FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods 
and override toString method in Avro
URL: https://github.com/apache/spark/pull/23699#issuecomment-459244506
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101930/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23699: [FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods and override toString method in Avro

[GitBox]

AmplabJenkins removed a comment on issue #23699: 
[FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods 
and override toString method in Avro
URL: https://github.com/apache/spark/pull/23699#issuecomment-459244506
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101930/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23699: [FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods and override toString method in Avro

[GitBox]

AmplabJenkins commented on issue #23699: 
[FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods 
and override toString method in Avro
URL: https://github.com/apache/spark/pull/23699#issuecomment-459244500
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] SparkQA removed a comment on issue #23699: [FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods and override toString method in Avro

[GitBox]

SparkQA removed a comment on issue #23699: 
[FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods 
and override toString method in Avro
URL: https://github.com/apache/spark/pull/23699#issuecomment-459201259
 
 
   **[Test build #101930 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101930/testReport)**
 for PR 23699 at commit 
[`30a318b`](https://github.com/apache/spark/commit/30a318b100f43ccfc30e2ef32c3b09c0e7096aaf).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] SparkQA commented on issue #23699: [FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods and override toString method in Avro

[GitBox]

SparkQA commented on issue #23699: [FollowUp][SPARK-26716][SPARK-26765][SQL] 
Clean up schema validation methods and override toString method in Avro
URL: https://github.com/apache/spark/pull/23699#issuecomment-459244028
 
 
   **[Test build #101930 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101930/testReport)**
 for PR 23699 at commit 
[`30a318b`](https://github.com/apache/spark/commit/30a318b100f43ccfc30e2ef32c3b09c0e7096aaf).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23699: [FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods and override toString method in Avro

[GitBox]

AmplabJenkins removed a comment on issue #23699: 
[FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods 
and override toString method in Avro
URL: https://github.com/apache/spark/pull/23699#issuecomment-459242098
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23699: [FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods and override toString method in Avro

[GitBox]

AmplabJenkins removed a comment on issue #23699: 
[FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods 
and override toString method in Avro
URL: https://github.com/apache/spark/pull/23699#issuecomment-459242101
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101928/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23699: [FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods and override toString method in Avro

[GitBox]

AmplabJenkins commented on issue #23699: 
[FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods 
and override toString method in Avro
URL: https://github.com/apache/spark/pull/23699#issuecomment-459242101
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101928/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23699: [FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods and override toString method in Avro

[GitBox]

AmplabJenkins commented on issue #23699: 
[FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods 
and override toString method in Avro
URL: https://github.com/apache/spark/pull/23699#issuecomment-459242098
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] SparkQA commented on issue #23699: [FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods and override toString method in Avro

[GitBox]

SparkQA commented on issue #23699: [FollowUp][SPARK-26716][SPARK-26765][SQL] 
Clean up schema validation methods and override toString method in Avro
URL: https://github.com/apache/spark/pull/23699#issuecomment-459241611
 
 
   **[Test build #101928 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101928/testReport)**
 for PR 23699 at commit 
[`30a318b`](https://github.com/apache/spark/commit/30a318b100f43ccfc30e2ef32c3b09c0e7096aaf).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] SparkQA removed a comment on issue #23699: [FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods and override toString method in Avro

[GitBox]

SparkQA removed a comment on issue #23699: 
[FollowUp][SPARK-26716][SPARK-26765][SQL] Clean up schema validation methods 
and override toString method in Avro
URL: https://github.com/apache/spark/pull/23699#issuecomment-459200285
 
 
   **[Test build #101928 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101928/testReport)**
 for PR 23699 at commit 
[`30a318b`](https://github.com/apache/spark/commit/30a318b100f43ccfc30e2ef32c3b09c0e7096aaf).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] rednaxelafx commented on issue #23701: [SPARK-26741][SQL] Allow using aggregate expressions in ORDER BY clause

[GitBox]

rednaxelafx commented on issue #23701: [SPARK-26741][SQL] Allow using aggregate 
expressions in ORDER BY clause
URL: https://github.com/apache/spark/pull/23701#issuecomment-459240165
 
 
   Indeed, we should probably fail that query instead of throwing a weird error 
in codegen.
   
   This is an equivalent example in PG10:
   ```sql
   create table t1 as select unnest(array[0, 1, 2]) as v;
   create table t2 as select unnest(array[10, 11, 12]) as v;
   
   select * from
 ((select v, max(v) from t1 group by v)
 union all
 (select v, max(v) from t2 group by v)) tt
   order by max(tt.v)
   ```
   which fails with:
   ```
   Query Error: error: column "tt.v" must appear in the GROUP BY clause or be 
used in an aggregate function
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] cloud-fan commented on a change in pull request #23667: [SPARK-26745][SQL] Revert count optimization in JSON datasource by SPARK-24959

[GitBox]

cloud-fan commented on a change in pull request #23667: [SPARK-26745][SQL] 
Revert count optimization in JSON datasource by SPARK-24959
URL: https://github.com/apache/spark/pull/23667#discussion_r252552978
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala
 ##
 @@ -188,11 +188,19 @@ class UnivocityParser(
 }
   }
 
+  private val doParse = if (requiredSchema.nonEmpty) {
+(input: String) => convert(tokenizer.parseLine(input))
+  } else {
+// If `columnPruning` enabled and partition attributes scanned only,
+// `schema` gets empty.
+(_: String) => InternalRow.empty
 
 Review comment:
   It's too long ago and I can't remember the details. Does it mean we still 
have this count optimization for CSV? does it work in multiline mode?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] felixcheung commented on a change in pull request #23520: [SPARK-26603][K8s] Update minikube backend

[GitBox]

felixcheung commented on a change in pull request #23520: [SPARK-26603][K8s] 
Update minikube backend
URL: https://github.com/apache/spark/pull/23520#discussion_r252552495
 
 

 ##
 File path: 
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/minikube/Minikube.scala
 ##
 @@ -52,13 +66,43 @@ private[spark] object Minikube extends Logging {
 val kubernetesConf = new ConfigBuilder()
   .withApiVersion("v1")
   .withMasterUrl(kubernetesMaster)
-  .withCaCertFile(Paths.get(userHome, ".minikube", 
"ca.crt").toFile.getAbsolutePath)
-  .withClientCertFile(Paths.get(userHome, ".minikube", 
"apiserver.crt").toFile.getAbsolutePath)
-  .withClientKeyFile(Paths.get(userHome, ".minikube", 
"apiserver.key").toFile.getAbsolutePath)
+  .withCaCertFile(Paths.get(userHome, MINIKUBE_PATH, 
"ca.crt").toFile.getAbsolutePath)
+  .withClientCertFile(Paths.get(userHome, MINIKUBE_PATH, 
"apiserver.crt").toFile.getAbsolutePath)
+  .withClientKeyFile(Paths.get(userHome, MINIKUBE_PATH, 
"apiserver.key").toFile.getAbsolutePath)
   .build()
 new DefaultKubernetesClient(kubernetesConf)
   }
 
+  // Covers minikube status output after Minikube V0.30.
+  private def getIfNewMinikubeStatus(statusString: Seq[String]): 
MinikubeStatus.Value = {
+val hostString = statusString.find(_.contains(s"$HOST_PREFIX "))
+val kubeletString = statusString.find(_.contains(s"$KUBELET_PREFIX "))
+val apiserverString = statusString.find(_.contains(s"$APISERVER_PREFIX "))
+val kubectlString = statusString.find(_.contains(s"$KUBECTL_PREFIX "))
+
+if (hostString.isEmpty || kubeletString.isEmpty
+  || apiserverString.isEmpty || kubectlString.isEmpty) {
+  MinikubeStatus.NONE
+} else {
+  val status1 = hostString.get.replaceFirst(s"$HOST_PREFIX ", "")
+  val status2 = kubeletString.get.replaceFirst(s"$KUBELET_PREFIX ", "")
+  val status3 = apiserverString.get.replaceFirst(s"$APISERVER_PREFIX ", "")
+  val status4 = kubectlString.get.replaceFirst(s"$KUBECTL_PREFIX ", "")
+  if (!status4.contains("Correctly Configured:")) {
 
 Review comment:
   is searching this specific word a bit fragile?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] viirya commented on a change in pull request #23438: [SPARK-26525][SHUFFLE]Fast release ShuffleBlockFetcherIterator on completion of the iteration

[GitBox]

viirya commented on a change in pull request #23438: [SPARK-26525][SHUFFLE]Fast 
release ShuffleBlockFetcherIterator on completion of the iteration
URL: https://github.com/apache/spark/pull/23438#discussion_r252551515
 
 

 ##
 File path: 
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
 ##
 @@ -509,6 +511,11 @@ final class ShuffleBlockFetcherIterator(
 (currentResult.blockId, new BufferReleasingInputStream(input, this))
   }
 
+  def toCompletionIterator: Iterator[(BlockId, InputStream)] = {
+CompletionIterator[(BlockId, InputStream), this.type](this,
 
 Review comment:
   Although `CompletionIterator` frees the reference to the 
`ShuffleBlockFetcherIterator` in `ShuffleFetchCompletionListener`, doesn't this 
`CompletionIterator` also internally keep another reference to the 
`ShuffleBlockFetcherIterator`? Will it be released early too?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23117: [SPARK-7721][INFRA] Run and generate test coverage report from Python via Jenkins

[GitBox]

AmplabJenkins removed a comment on issue #23117: [SPARK-7721][INFRA] Run and 
generate test coverage report from Python via Jenkins
URL: https://github.com/apache/spark/pull/23117#issuecomment-459235207
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101927/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23117: [SPARK-7721][INFRA] Run and generate test coverage report from Python via Jenkins

[GitBox]

AmplabJenkins removed a comment on issue #23117: [SPARK-7721][INFRA] Run and 
generate test coverage report from Python via Jenkins
URL: https://github.com/apache/spark/pull/23117#issuecomment-459235197
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23117: [SPARK-7721][INFRA] Run and generate test coverage report from Python via Jenkins

[GitBox]

AmplabJenkins commented on issue #23117: [SPARK-7721][INFRA] Run and generate 
test coverage report from Python via Jenkins
URL: https://github.com/apache/spark/pull/23117#issuecomment-459235207
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101927/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] cloud-fan commented on issue #23701: [SPARK-26741][SQL] Allow using aggregate expressions in ORDER BY clause

[GitBox]

cloud-fan commented on issue #23701: [SPARK-26741][SQL] Allow using aggregate 
expressions in ORDER BY clause
URL: https://github.com/apache/spark/pull/23701#issuecomment-459235128
 
 
   ```
   val df1 = spark.range(10).groupBy('id).agg(max('id) as 'm)
   val df2 = spark.range(10, 20).groupBy('id).agg(max('id) as 'm)
   val df = df1.unionAll(df2).orderBy(max('id))
   ```
   It seems to me we should fail this query. If we write this query in SQL, 
`select * from (... union ...) order by max(id)`, does this work in other 
databases?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23117: [SPARK-7721][INFRA] Run and generate test coverage report from Python via Jenkins

[GitBox]

AmplabJenkins commented on issue #23117: [SPARK-7721][INFRA] Run and generate 
test coverage report from Python via Jenkins
URL: https://github.com/apache/spark/pull/23117#issuecomment-459235197
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] SparkQA removed a comment on issue #23117: [SPARK-7721][INFRA] Run and generate test coverage report from Python via Jenkins

[GitBox]

SparkQA removed a comment on issue #23117: [SPARK-7721][INFRA] Run and generate 
test coverage report from Python via Jenkins
URL: https://github.com/apache/spark/pull/23117#issuecomment-459188865
 
 
   **[Test build #101927 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101927/testReport)**
 for PR 23117 at commit 
[`426ef11`](https://github.com/apache/spark/commit/426ef1182dffb3bd01561af7d714d8ce3bb6ddde).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23664: [MINOR][DOCS] Add a note that 'spark.executor.pyspark.memory' is dependent on 'resource'

[GitBox]

AmplabJenkins removed a comment on issue #23664: [MINOR][DOCS] Add a note that 
'spark.executor.pyspark.memory' is dependent on 'resource'
URL: https://github.com/apache/spark/pull/23664#issuecomment-459234721
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] SparkQA commented on issue #23117: [SPARK-7721][INFRA] Run and generate test coverage report from Python via Jenkins

[GitBox]

SparkQA commented on issue #23117: [SPARK-7721][INFRA] Run and generate test 
coverage report from Python via Jenkins
URL: https://github.com/apache/spark/pull/23117#issuecomment-459234798
 
 
   **[Test build #101927 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101927/testReport)**
 for PR 23117 at commit 
[`426ef11`](https://github.com/apache/spark/commit/426ef1182dffb3bd01561af7d714d8ce3bb6ddde).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23664: [MINOR][DOCS] Add a note that 'spark.executor.pyspark.memory' is dependent on 'resource'

[GitBox]

AmplabJenkins commented on issue #23664: [MINOR][DOCS] Add a note that 
'spark.executor.pyspark.memory' is dependent on 'resource'
URL: https://github.com/apache/spark/pull/23664#issuecomment-459234724
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101934/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23664: [MINOR][DOCS] Add a note that 'spark.executor.pyspark.memory' is dependent on 'resource'

[GitBox]

AmplabJenkins removed a comment on issue #23664: [MINOR][DOCS] Add a note that 
'spark.executor.pyspark.memory' is dependent on 'resource'
URL: https://github.com/apache/spark/pull/23664#issuecomment-459234724
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101934/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] SparkQA removed a comment on issue #23664: [MINOR][DOCS] Add a note that 'spark.executor.pyspark.memory' is dependent on 'resource'

[GitBox]

SparkQA removed a comment on issue #23664: [MINOR][DOCS] Add a note that 
'spark.executor.pyspark.memory' is dependent on 'resource'
URL: https://github.com/apache/spark/pull/23664#issuecomment-459232766
 
 
   **[Test build #101934 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101934/testReport)**
 for PR 23664 at commit 
[`7fa33ed`](https://github.com/apache/spark/commit/7fa33ed39053a98d36502cf8f9237004ee968c69).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] SparkQA commented on issue #23664: [MINOR][DOCS] Add a note that 'spark.executor.pyspark.memory' is dependent on 'resource'

[GitBox]

SparkQA commented on issue #23664: [MINOR][DOCS] Add a note that 
'spark.executor.pyspark.memory' is dependent on 'resource'
URL: https://github.com/apache/spark/pull/23664#issuecomment-459234672
 
 
   **[Test build #101934 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101934/testReport)**
 for PR 23664 at commit 
[`7fa33ed`](https://github.com/apache/spark/commit/7fa33ed39053a98d36502cf8f9237004ee968c69).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23664: [MINOR][DOCS] Add a note that 'spark.executor.pyspark.memory' is dependent on 'resource'

[GitBox]

AmplabJenkins commented on issue #23664: [MINOR][DOCS] Add a note that 
'spark.executor.pyspark.memory' is dependent on 'resource'
URL: https://github.com/apache/spark/pull/23664#issuecomment-459234721
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] felixcheung edited a comment on issue #23707: [SPARK-26793][CORE]Remove spark.shuffle.manager

[GitBox]

felixcheung edited a comment on issue #23707: [SPARK-26793][CORE]Remove 
spark.shuffle.manager
URL: https://github.com/apache/spark/pull/23707#issuecomment-459233992
 
 
   @10110346 wait, there's actually a bunch of ongoing work in this space that 
depends on this extensibility.
   
   please start a discussion on dev@ before any further action. for reference, 
related work (one effort, there are others that plug-in to shuffle manager) 
https://issues.apache.org/jira/browse/SPARK-25299
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] felixcheung commented on issue #23707: [SPARK-26793][CORE]Remove spark.shuffle.manager

[GitBox]

felixcheung commented on issue #23707: [SPARK-26793][CORE]Remove 
spark.shuffle.manager
URL: https://github.com/apache/spark/pull/23707#issuecomment-459233992
 
 
   @10110346 wait, there's actually a bunch of ongoing work in this space that 
depends on this extensibility.
   
   please start a discussion on dev@ more any further action. related work (one 
effort, there are other that plug-in to shuffle manager) 
https://issues.apache.org/jira/browse/SPARK-25299
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23674: [SPARK-26745][SQL] JsonSuite test case: empty line -> 0 record count

[GitBox]

AmplabJenkins removed a comment on issue #23674: [SPARK-26745][SQL] JsonSuite 
test case: empty line -> 0 record count
URL: https://github.com/apache/spark/pull/23674#issuecomment-459233504
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7620/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] HyukjinKwon closed pull request #23665: [SPARK-26745][SQL] Skip empty lines in JSON-derived DataFrames when skipParsing optimization in effect

[GitBox]

HyukjinKwon closed pull request #23665: [SPARK-26745][SQL] Skip empty lines in 
JSON-derived DataFrames when skipParsing optimization in effect
URL: https://github.com/apache/spark/pull/23665
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] SparkQA commented on issue #23674: [SPARK-26745][SQL] JsonSuite test case: empty line -> 0 record count

[GitBox]

SparkQA commented on issue #23674: [SPARK-26745][SQL] JsonSuite test case: 
empty line -> 0 record count
URL: https://github.com/apache/spark/pull/23674#issuecomment-459233786
 
 
   **[Test build #101935 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101935/testReport)**
 for PR 23674 at commit 
[`739bbf0`](https://github.com/apache/spark/commit/739bbf0a2ebe5306e700e6d8b1971018b2709303).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] HyukjinKwon commented on issue #23665: [SPARK-26745][SQL] Skip empty lines in JSON-derived DataFrames when skipParsing optimization in effect

[GitBox]

HyukjinKwon commented on issue #23665: [SPARK-26745][SQL] Skip empty lines in 
JSON-derived DataFrames when skipParsing optimization in effect
URL: https://github.com/apache/spark/pull/23665#issuecomment-459233888
 
 
   Closing this since that's reverted.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23674: [SPARK-26745][SQL] JsonSuite test case: empty line -> 0 record count

[GitBox]

AmplabJenkins removed a comment on issue #23674: [SPARK-26745][SQL] JsonSuite 
test case: empty line -> 0 record count
URL: https://github.com/apache/spark/pull/23674#issuecomment-459233503
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] HyukjinKwon commented on a change in pull request #23674: [SPARK-26745][SQL] JsonSuite test case: empty line -> 0 record count

[GitBox]

HyukjinKwon commented on a change in pull request #23674: [SPARK-26745][SQL] 
JsonSuite test case: empty line -> 0 record count
URL: https://github.com/apache/spark/pull/23674#discussion_r252550120
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala
 ##
 @@ -2426,6 +2426,17 @@ class JsonSuite extends QueryTest with SharedSQLContext 
with TestJsonData {
 countForMalformedJSON(0, Seq(""))
   }
 
+  test("SPARK-26745: count() for non-multiline input with empty lines") {
+val df = spark.read.json(testFile("test-data/with-empty-line.json"))
+val withEmptyLineData = Row(1, 2, 3) :: Row(4, 5, 6) :: Row(7, 8, 9) :: Nil
 
 Review comment:
   Actually, can you write a test without a separate file? you could do, for 
instance,
   
   ```scala
   withTempPath { path =>
 Seq("", """{"a":4}""", "", ...).toDF.repartition(1)
   .write.text(path.getCanonicalPath())
 assert(spark.read.json(path.getCanonicalPath()).count() === 3)
   }
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23649: [SPARK-26726] Synchronize the amount of memory used by the broadcast variable to the UI display

[GitBox]

AmplabJenkins removed a comment on issue #23649: [SPARK-26726] Synchronize the 
amount of memory used by the broadcast variable to the UI display
URL: https://github.com/apache/spark/pull/23649#issuecomment-459233603
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23649: [SPARK-26726] Synchronize the amount of memory used by the broadcast variable to the UI display

[GitBox]

AmplabJenkins commented on issue #23649: [SPARK-26726] Synchronize the amount 
of memory used by the broadcast variable to the UI display
URL: https://github.com/apache/spark/pull/23649#issuecomment-459233603
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23649: [SPARK-26726] Synchronize the amount of memory used by the broadcast variable to the UI display

[GitBox]

AmplabJenkins removed a comment on issue #23649: [SPARK-26726] Synchronize the 
amount of memory used by the broadcast variable to the UI display
URL: https://github.com/apache/spark/pull/23649#issuecomment-459233605
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101925/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23649: [SPARK-26726] Synchronize the amount of memory used by the broadcast variable to the UI display

[GitBox]

AmplabJenkins commented on issue #23649: [SPARK-26726] Synchronize the amount 
of memory used by the broadcast variable to the UI display
URL: https://github.com/apache/spark/pull/23649#issuecomment-459233605
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101925/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23674: [SPARK-26745][SQL] JsonSuite test case: empty line -> 0 record count

[GitBox]

AmplabJenkins commented on issue #23674: [SPARK-26745][SQL] JsonSuite test 
case: empty line -> 0 record count
URL: https://github.com/apache/spark/pull/23674#issuecomment-459233503
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23674: [SPARK-26745][SQL] JsonSuite test case: empty line -> 0 record count

[GitBox]

AmplabJenkins commented on issue #23674: [SPARK-26745][SQL] JsonSuite test 
case: empty line -> 0 record count
URL: https://github.com/apache/spark/pull/23674#issuecomment-459233504
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7620/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] HyukjinKwon commented on a change in pull request #23674: [SPARK-26745][SQL] JsonSuite test case: empty line -> 0 record count

[GitBox]

HyukjinKwon commented on a change in pull request #23674: [SPARK-26745][SQL] 
JsonSuite test case: empty line -> 0 record count
URL: https://github.com/apache/spark/pull/23674#discussion_r252549650
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala
 ##
 @@ -2426,6 +2426,17 @@ class JsonSuite extends QueryTest with SharedSQLContext 
with TestJsonData {
 countForMalformedJSON(0, Seq(""))
   }
 
+  test("SPARK-26745: count() for non-multiline input with empty lines") {
+val df = spark.read.json(testFile("test-data/with-empty-line.json"))
+val withEmptyLineData = Row(1, 2, 3) :: Row(4, 5, 6) :: Row(7, 8, 9) :: Nil
 
 Review comment:
   Just simply checking the count alone should be okay.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] SparkQA removed a comment on issue #23649: [SPARK-26726] Synchronize the amount of memory used by the broadcast variable to the UI display

[GitBox]

SparkQA removed a comment on issue #23649: [SPARK-26726] Synchronize the amount 
of memory used by the broadcast variable to the UI display
URL: https://github.com/apache/spark/pull/23649#issuecomment-459184323
 
 
   **[Test build #101925 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101925/testReport)**
 for PR 23649 at commit 
[`2024976`](https://github.com/apache/spark/commit/2024976b473665ff5ebde3f1ae68ce180fd54ae5).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] HyukjinKwon commented on a change in pull request #23674: [SPARK-26745][SQL] JsonSuite test case: empty line -> 0 record count

[GitBox]

HyukjinKwon commented on a change in pull request #23674: [SPARK-26745][SQL] 
JsonSuite test case: empty line -> 0 record count
URL: https://github.com/apache/spark/pull/23674#discussion_r252549650
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala
 ##
 @@ -2426,6 +2426,17 @@ class JsonSuite extends QueryTest with SharedSQLContext 
with TestJsonData {
 countForMalformedJSON(0, Seq(""))
   }
 
+  test("SPARK-26745: count() for non-multiline input with empty lines") {
+val df = spark.read.json(testFile("test-data/with-empty-line.json"))
+val withEmptyLineData = Row(1, 2, 3) :: Row(4, 5, 6) :: Row(7, 8, 9) :: Nil
 
 Review comment:
   Just simply checking the count alone should be okay.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] SparkQA commented on issue #23649: [SPARK-26726] Synchronize the amount of memory used by the broadcast variable to the UI display

[GitBox]

SparkQA commented on issue #23649: [SPARK-26726] Synchronize the amount of 
memory used by the broadcast variable to the UI display
URL: https://github.com/apache/spark/pull/23649#issuecomment-459233227
 
 
   **[Test build #101925 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101925/testReport)**
 for PR 23649 at commit 
[`2024976`](https://github.com/apache/spark/commit/2024976b473665ff5ebde3f1ae68ce180fd54ae5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] HyukjinKwon commented on a change in pull request #23674: [SPARK-26745][SQL] JsonSuite test case: empty line -> 0 record count

[GitBox]

HyukjinKwon commented on a change in pull request #23674: [SPARK-26745][SQL] 
JsonSuite test case: empty line -> 0 record count
URL: https://github.com/apache/spark/pull/23674#discussion_r252549571
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala
 ##
 @@ -2426,6 +2426,17 @@ class JsonSuite extends QueryTest with SharedSQLContext 
with TestJsonData {
 countForMalformedJSON(0, Seq(""))
   }
 
+  test("SPARK-26745: count() for non-multiline input with empty lines") {
+val df = spark.read.json(testFile("test-data/with-empty-line.json"))
+val withEmptyLineData = Row(1, 2, 3) :: Row(4, 5, 6) :: Row(7, 8, 9) :: Nil
+// important to do this .count() first, prior to 
caching/persisting/computing/collecting, to
+// test the non-parsed-count pathway
+assert(df.count() === withEmptyLineData.length,
+   "JSON DataFrame unparsed-count should exclude whitespace-only 
lines")
+df.createOrReplaceTempView("jsonWithEmptyLineTable")
+checkAnswer(sql("select * from jsonWithEmptyLineTable"), withEmptyLineData)
 
 Review comment:
   I think we don't need to test sql syntax again here. We could remove two 
lines.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] asfgit closed pull request #23667: [SPARK-26745][SQL] Revert count optimization in JSON datasource by SPARK-24959

[GitBox]

asfgit closed pull request #23667: [SPARK-26745][SQL] Revert count optimization 
in JSON datasource by SPARK-24959
URL: https://github.com/apache/spark/pull/23667
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] HyukjinKwon commented on a change in pull request #23674: [SPARK-26745][SQL] JsonSuite test case: empty line -> 0 record count

[GitBox]

HyukjinKwon commented on a change in pull request #23674: [SPARK-26745][SQL] 
JsonSuite test case: empty line -> 0 record count
URL: https://github.com/apache/spark/pull/23674#discussion_r252549423
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala
 ##
 @@ -2426,6 +2426,17 @@ class JsonSuite extends QueryTest with SharedSQLContext 
with TestJsonData {
 countForMalformedJSON(0, Seq(""))
   }
 
+  test("SPARK-26745: count() for non-multiline input with empty lines") {
+val df = spark.read.json(testFile("test-data/with-empty-line.json"))
+val withEmptyLineData = Row(1, 2, 3) :: Row(4, 5, 6) :: Row(7, 8, 9) :: Nil
+// important to do this .count() first, prior to 
caching/persisting/computing/collecting, to
+// test the non-parsed-count pathway
+assert(df.count() === withEmptyLineData.length,
+   "JSON DataFrame unparsed-count should exclude whitespace-only 
lines")
 
 Review comment:
   nit:
   
   ```
   assert(df.count() === withEmptyLineData.length,
 "JSON DataFrame unparsed-count should exclude whitespace-only lines")
   ```
   
   indentation


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23664: [MINOR][DOCS] Add a note that 'spark.executor.pyspark.memory' is dependent on 'resource'

[GitBox]

AmplabJenkins removed a comment on issue #23664: [MINOR][DOCS] Add a note that 
'spark.executor.pyspark.memory' is dependent on 'resource'
URL: https://github.com/apache/spark/pull/23664#issuecomment-459232424
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7619/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] HyukjinKwon commented on issue #23674: [SPARK-26745][SQL] JsonSuite test case: empty line -> 0 record count

[GitBox]

HyukjinKwon commented on issue #23674: [SPARK-26745][SQL] JsonSuite test case: 
empty line -> 0 record count
URL: https://github.com/apache/spark/pull/23674#issuecomment-459232901
 
 
   ok to test


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23674: [SPARK-26745][SQL] JsonSuite test case: empty line -> 0 record count

[GitBox]

AmplabJenkins removed a comment on issue #23674: [SPARK-26745][SQL] JsonSuite 
test case: empty line -> 0 record count
URL: https://github.com/apache/spark/pull/23674#issuecomment-458129627
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] SparkQA commented on issue #23664: [MINOR][DOCS] Add a note that 'spark.executor.pyspark.memory' is dependent on 'resource'

[GitBox]

SparkQA commented on issue #23664: [MINOR][DOCS] Add a note that 
'spark.executor.pyspark.memory' is dependent on 'resource'
URL: https://github.com/apache/spark/pull/23664#issuecomment-459232766
 
 
   **[Test build #101934 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101934/testReport)**
 for PR 23664 at commit 
[`7fa33ed`](https://github.com/apache/spark/commit/7fa33ed39053a98d36502cf8f9237004ee968c69).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] HyukjinKwon commented on issue #23667: [SPARK-26745][SQL] Revert count optimization in JSON datasource by SPARK-24959

[GitBox]

HyukjinKwon commented on issue #23667: [SPARK-26745][SQL] Revert count 
optimization in JSON datasource by SPARK-24959
URL: https://github.com/apache/spark/pull/23667#issuecomment-459232624
 
 
   Merged to master.
   
   I am going to open a backport soon.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23664: [MINOR][DOCS] Add a note that 'spark.executor.pyspark.memory' is dependent on 'resource'

[GitBox]

AmplabJenkins removed a comment on issue #23664: [MINOR][DOCS] Add a note that 
'spark.executor.pyspark.memory' is dependent on 'resource'
URL: https://github.com/apache/spark/pull/23664#issuecomment-459232418
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23664: [MINOR][DOCS] Add a note that 'spark.executor.pyspark.memory' is dependent on 'resource'

[GitBox]

AmplabJenkins commented on issue #23664: [MINOR][DOCS] Add a note that 
'spark.executor.pyspark.memory' is dependent on 'resource'
URL: https://github.com/apache/spark/pull/23664#issuecomment-459232424
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7619/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23664: [MINOR][DOCS] Add a note that 'spark.executor.pyspark.memory' is dependent on 'resource'

[GitBox]

AmplabJenkins commented on issue #23664: [MINOR][DOCS] Add a note that 
'spark.executor.pyspark.memory' is dependent on 'resource'
URL: https://github.com/apache/spark/pull/23664#issuecomment-459232418
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] cloud-fan commented on issue #23601: [SPARK-26673][SQL] File source V2 writes: create framework and migrate ORC

[GitBox]

cloud-fan commented on issue #23601: [SPARK-26673][SQL] File source V2 writes: 
create framework and migrate ORC
URL: https://github.com/apache/spark/pull/23601#issuecomment-459232006
 
 
   LGTM except a few minor comments


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] cloud-fan commented on a change in pull request #23601: [SPARK-26673][SQL] File source V2 writes: create framework and migrate ORC

[GitBox]

cloud-fan commented on a change in pull request #23601: [SPARK-26673][SQL] File 
source V2 writes: create framework and migrate ORC
URL: https://github.com/apache/spark/pull/23601#discussion_r252548464
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala
 ##
 @@ -56,18 +56,25 @@ case class WriteToDataSourceV2Exec(batchWrite: BatchWrite, 
query: SparkPlan)
 val writerFactory = batchWrite.createBatchWriterFactory()
 val useCommitCoordinator = batchWrite.useCommitCoordinator
 val rdd = query.execute()
-val messages = new Array[WriterCommitMessage](rdd.partitions.length)
+// SPARK-23271 If we are attempting to write a zero partition rdd, create 
a dummy single
+// partition rdd to make sure we at least set up one write task to write 
the metadata.
 
 Review comment:
   It's ok for now, but we should improve it later:
   1. use a config to do it, it seems only file source need it
   2. or do it in `FileBatchWrite.commit`. If commit messages are empty, write 
a metadata file.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] cloud-fan commented on a change in pull request #23601: [SPARK-26673][SQL] File source V2 writes: create framework and migrate ORC

[GitBox]

cloud-fan commented on a change in pull request #23601: [SPARK-26673][SQL] File 
source V2 writes: create framework and migrate ORC
URL: https://github.com/apache/spark/pull/23601#discussion_r252547696
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileWriteBuilder.scala
 ##
 @@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.v2
+
+import java.util.{Optional, UUID}
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.fs.Path
+import org.apache.hadoop.mapreduce.Job
+import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat
+
+import org.apache.spark.internal.io.{FileCommitProtocol, 
HadoopMapReduceCommitProtocol}
+import org.apache.spark.sql.{AnalysisException, SaveMode, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{CaseInsensitiveMap, DateTimeUtils}
+import org.apache.spark.sql.execution.datasources.{BasicWriteJobStatsTracker, 
DataSource, OutputWriterFactory, WriteJobDescription}
+import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.sources.v2.{DataSourceOptions, SupportsBatchWrite}
+import org.apache.spark.sql.sources.v2.writer.{BatchWrite, SupportsSaveMode, 
WriteBuilder}
+import org.apache.spark.sql.types.{DataType, StructType}
+import org.apache.spark.util.SerializableConfiguration
+
+abstract class FileWriteBuilder(options: DataSourceOptions)
+  extends WriteBuilder with SupportsSaveMode {
+  private var schema: StructType = _
+  private var queryId: String = _
+  private var mode: SaveMode = _
+
+  override def withInputDataSchema(schema: StructType): WriteBuilder = {
+this.schema = schema
+this
+  }
+
+  override def withQueryId(queryId: String): WriteBuilder = {
+this.queryId = queryId
+this
+  }
+
+  override def mode(mode: SaveMode): WriteBuilder = {
+this.mode = mode
+this
+  }
+
+  override def buildForBatch(): BatchWrite = {
 
 Review comment:
   this method is too long, could be better if we can separate it into multiple 
methods


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23704: [SPARK-26677][BUILD][WIP][test-maven] Update Parquet to 1.10.1 with notEq pushdown fix.

[GitBox]

AmplabJenkins removed a comment on issue #23704: 
[SPARK-26677][BUILD][WIP][test-maven] Update Parquet to 1.10.1 with notEq 
pushdown fix.
URL: https://github.com/apache/spark/pull/23704#issuecomment-459230801
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23704: [SPARK-26677][BUILD][WIP][test-maven] Update Parquet to 1.10.1 with notEq pushdown fix.

[GitBox]

AmplabJenkins commented on issue #23704: [SPARK-26677][BUILD][WIP][test-maven] 
Update Parquet to 1.10.1 with notEq pushdown fix.
URL: https://github.com/apache/spark/pull/23704#issuecomment-459230806
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101924/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23704: [SPARK-26677][BUILD][WIP][test-maven] Update Parquet to 1.10.1 with notEq pushdown fix.

[GitBox]

AmplabJenkins removed a comment on issue #23704: 
[SPARK-26677][BUILD][WIP][test-maven] Update Parquet to 1.10.1 with notEq 
pushdown fix.
URL: https://github.com/apache/spark/pull/23704#issuecomment-459230806
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101924/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23704: [SPARK-26677][BUILD][WIP][test-maven] Update Parquet to 1.10.1 with notEq pushdown fix.

[GitBox]

AmplabJenkins commented on issue #23704: [SPARK-26677][BUILD][WIP][test-maven] 
Update Parquet to 1.10.1 with notEq pushdown fix.
URL: https://github.com/apache/spark/pull/23704#issuecomment-459230801
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] SparkQA removed a comment on issue #23704: [SPARK-26677][BUILD][WIP][test-maven] Update Parquet to 1.10.1 with notEq pushdown fix.

[GitBox]

SparkQA removed a comment on issue #23704: 
[SPARK-26677][BUILD][WIP][test-maven] Update Parquet to 1.10.1 with notEq 
pushdown fix.
URL: https://github.com/apache/spark/pull/23704#issuecomment-459183146
 
 
   **[Test build #101924 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101924/testReport)**
 for PR 23704 at commit 
[`b5ba00c`](https://github.com/apache/spark/commit/b5ba00ccbf076529aa63d04a164806c2e5e7bbb3).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] SparkQA commented on issue #23679: [SPARK-23516][CORE] Avoid repeated release/acquire on storage memory

[GitBox]

SparkQA commented on issue #23679: [SPARK-23516][CORE] Avoid repeated 
release/acquire on storage memory
URL: https://github.com/apache/spark/pull/23679#issuecomment-459230417
 
 
   **[Test build #101933 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101933/testReport)**
 for PR 23679 at commit 
[`fd3bb3d`](https://github.com/apache/spark/commit/fd3bb3dcf566636e2b56274f0e1e4ff2efb6e17d).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] SparkQA commented on issue #23704: [SPARK-26677][BUILD][WIP][test-maven] Update Parquet to 1.10.1 with notEq pushdown fix.

[GitBox]

SparkQA commented on issue #23704: [SPARK-26677][BUILD][WIP][test-maven] Update 
Parquet to 1.10.1 with notEq pushdown fix.
URL: https://github.com/apache/spark/pull/23704#issuecomment-459230366
 
 
   **[Test build #101924 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101924/testReport)**
 for PR 23704 at commit 
[`b5ba00c`](https://github.com/apache/spark/commit/b5ba00ccbf076529aa63d04a164806c2e5e7bbb3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23679: [SPARK-23516][CORE] Avoid repeated release/acquire on storage memory

[GitBox]

AmplabJenkins removed a comment on issue #23679: [SPARK-23516][CORE] Avoid 
repeated release/acquire on storage memory
URL: https://github.com/apache/spark/pull/23679#issuecomment-459230088
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7618/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23679: [SPARK-23516][CORE] Avoid repeated release/acquire on storage memory

[GitBox]

AmplabJenkins removed a comment on issue #23679: [SPARK-23516][CORE] Avoid 
repeated release/acquire on storage memory
URL: https://github.com/apache/spark/pull/23679#issuecomment-459230082
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23679: [SPARK-23516][CORE] Avoid repeated release/acquire on storage memory

[GitBox]

AmplabJenkins commented on issue #23679: [SPARK-23516][CORE] Avoid repeated 
release/acquire on storage memory
URL: https://github.com/apache/spark/pull/23679#issuecomment-459230082
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23695: [SPARK-26780][CORE]Improve shuffle read using ReadAheadInputStream

[GitBox]

AmplabJenkins removed a comment on issue #23695: [SPARK-26780][CORE]Improve 
shuffle read using ReadAheadInputStream 
URL: https://github.com/apache/spark/pull/23695#issuecomment-459229908
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23679: [SPARK-23516][CORE] Avoid repeated release/acquire on storage memory

[GitBox]

AmplabJenkins commented on issue #23679: [SPARK-23516][CORE] Avoid repeated 
release/acquire on storage memory
URL: https://github.com/apache/spark/pull/23679#issuecomment-459230088
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7618/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins removed a comment on issue #23695: [SPARK-26780][CORE]Improve shuffle read using ReadAheadInputStream

[GitBox]

AmplabJenkins removed a comment on issue #23695: [SPARK-26780][CORE]Improve 
shuffle read using ReadAheadInputStream 
URL: https://github.com/apache/spark/pull/23695#issuecomment-459229910
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101922/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23695: [SPARK-26780][CORE]Improve shuffle read using ReadAheadInputStream

[GitBox]

AmplabJenkins commented on issue #23695: [SPARK-26780][CORE]Improve shuffle 
read using ReadAheadInputStream 
URL: https://github.com/apache/spark/pull/23695#issuecomment-459229908
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] AmplabJenkins commented on issue #23695: [SPARK-26780][CORE]Improve shuffle read using ReadAheadInputStream

[GitBox]

AmplabJenkins commented on issue #23695: [SPARK-26780][CORE]Improve shuffle 
read using ReadAheadInputStream 
URL: https://github.com/apache/spark/pull/23695#issuecomment-459229910
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101922/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org