[jira] [Commented] (SPARK-5484) Pregel should checkpoint periodically to avoid StackOverflowError

2016-09-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498029#comment-15498029 ] Apache Spark commented on SPARK-5484: - User 'dding3' has created a pull request for this issue:

[jira] [Commented] (SPARK-17559) PeriodicGraphCheckpointer didnot persist edges as expected in some cases

2016-09-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497917#comment-15497917 ] Apache Spark commented on SPARK-17559: -- User 'dding3' has created a pull request for this issue:

[jira] [Commented] (SPARK-17551) support null ordering for DataFrame API

2016-09-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497874#comment-15497874 ] Apache Spark commented on SPARK-17551: -- User 'petermaxlee' has created a pull request for this

[jira] [Commented] (SPARK-17570) Avoid Hash and Exchange in Sort Merge join if bucketing factor is multiple for tables

2016-09-16 Thread Tejas Patil (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497729#comment-15497729 ] Tejas Patil commented on SPARK-17570: - [~cloud_fan] , [~hvanhovell] : I have been looking into the

[jira] [Assigned] (SPARK-17569) Don't recheck existence of files when generating File Relation resolution in StructuredStreaming

2016-09-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17569: Assignee: (was: Apache Spark) > Don't recheck existence of files when generating File

[jira] [Commented] (SPARK-17569) Don't recheck existence of files when generating File Relation resolution in StructuredStreaming

2016-09-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497680#comment-15497680 ] Apache Spark commented on SPARK-17569: -- User 'brkyvz' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17569) Don't recheck existence of files when generating File Relation resolution in StructuredStreaming

2016-09-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17569: Assignee: Apache Spark > Don't recheck existence of files when generating File Relation

[jira] [Assigned] (SPARK-17567) Broken link to Spark paper

2016-09-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17567: Assignee: Apache Spark > Broken link to Spark paper > -- > >

[jira] [Created] (SPARK-17570) Avoid Hash and Exchange in Sort Merge join if bucketing factor is multiple for tables

2016-09-16 Thread Tejas Patil (JIRA)
Tejas Patil created SPARK-17570: --- Summary: Avoid Hash and Exchange in Sort Merge join if bucketing factor is multiple for tables Key: SPARK-17570 URL: https://issues.apache.org/jira/browse/SPARK-17570

[jira] [Assigned] (SPARK-17567) Broken link to Spark paper

2016-09-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17567: Assignee: (was: Apache Spark) > Broken link to Spark paper >

[jira] [Commented] (SPARK-17567) Broken link to Spark paper

2016-09-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497670#comment-15497670 ] Apache Spark commented on SPARK-17567: -- User 'keypointt' has created a pull request for this issue:

[jira] [Created] (SPARK-17569) Don't recheck existence of files when generating File Relation resolution in StructuredStreaming

2016-09-16 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-17569: --- Summary: Don't recheck existence of files when generating File Relation resolution in StructuredStreaming Key: SPARK-17569 URL: https://issues.apache.org/jira/browse/SPARK-17569

[jira] [Assigned] (SPARK-4563) Allow spark driver to bind to different ip then advertise ip

2016-09-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-4563: --- Assignee: (was: Apache Spark) > Allow spark driver to bind to different ip then

[jira] [Commented] (SPARK-4563) Allow spark driver to bind to different ip then advertise ip

2016-09-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497532#comment-15497532 ] Apache Spark commented on SPARK-4563: - User 'vanzin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-4563) Allow spark driver to bind to different ip then advertise ip

2016-09-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-4563: --- Assignee: Apache Spark > Allow spark driver to bind to different ip then advertise ip >

[jira] [Assigned] (SPARK-17568) Add spark-submit option for user to override ivy settings used to resolve packages/artifacts

2016-09-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17568: Assignee: Apache Spark > Add spark-submit option for user to override ivy settings used

[jira] [Assigned] (SPARK-17568) Add spark-submit option for user to override ivy settings used to resolve packages/artifacts

2016-09-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17568: Assignee: (was: Apache Spark) > Add spark-submit option for user to override ivy

[jira] [Commented] (SPARK-17568) Add spark-submit option for user to override ivy settings used to resolve packages/artifacts

2016-09-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497502#comment-15497502 ] Apache Spark commented on SPARK-17568: -- User 'BryanCutler' has created a pull request for this

[jira] [Created] (SPARK-17568) Add spark-submit option for user to override ivy settings used to resolve packages/artifacts

2016-09-16 Thread Bryan Cutler (JIRA)
Bryan Cutler created SPARK-17568: Summary: Add spark-submit option for user to override ivy settings used to resolve packages/artifacts Key: SPARK-17568 URL: https://issues.apache.org/jira/browse/SPARK-17568

[jira] [Commented] (SPARK-17565) Janino exception when calculating metrics for large generated class

2016-09-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497368#comment-15497368 ] Marcelo Vanzin commented on SPARK-17565: The fix for SPARK-17549 adds a workaround for this issue

[jira] [Updated] (SPARK-17549) InMemoryRelation doesn't scale to large tables

2016-09-16 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-17549: - Assignee: Marcelo Vanzin > InMemoryRelation doesn't scale to large tables >

[jira] [Resolved] (SPARK-17549) InMemoryRelation doesn't scale to large tables

2016-09-16 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-17549. -- Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 Issue resolved by pull request

[jira] [Updated] (SPARK-17561) DataFrameWriter documentation formatting problems

2016-09-16 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-17561: Fix Version/s: (was: 2.0.1) > DataFrameWriter documentation formatting problems >

[jira] [Resolved] (SPARK-17561) DataFrameWriter documentation formatting problems

2016-09-16 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-17561. - Resolution: Fixed Assignee: Sean Owen Fix Version/s: 2.1.0

[jira] [Created] (SPARK-17567) Broken link to Spark paper

2016-09-16 Thread Ondrej Galbavy (JIRA)
Ondrej Galbavy created SPARK-17567: -- Summary: Broken link to Spark paper Key: SPARK-17567 URL: https://issues.apache.org/jira/browse/SPARK-17567 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-17566) "--master yarn --deploy-mode cluster" gives "Launching Python applications through spark-submit is currently only supported for local files"

2016-09-16 Thread Zhenhua Xu (JIRA)
Zhenhua Xu created SPARK-17566: -- Summary: "--master yarn --deploy-mode cluster" gives "Launching Python applications through spark-submit is currently only supported for local files" Key: SPARK-17566 URL:

[jira] [Commented] (SPARK-14948) Exception when joining DataFrames derived form the same DataFrame

2016-09-16 Thread Mijung Kim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497086#comment-15497086 ] Mijung Kim commented on SPARK-14948: I have come across the same problem. A leeway for this problem

[jira] [Commented] (SPARK-14358) Change SparkListener from a trait to an abstract class, and remove JavaSparkListener

2016-09-16 Thread Oleksiy Sayankin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496924#comment-15496924 ] Oleksiy Sayankin commented on SPARK-14358: -- [~rxin], please see my comments here:

[jira] [Commented] (SPARK-17565) Janino exception when calculating metrics for large generated class

2016-09-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496860#comment-15496860 ] Marcelo Vanzin commented on SPARK-17565: Note the code was generated with a recent 2.1 snapshot,

[jira] [Updated] (SPARK-17565) Janino exception when calculating metrics for large generated class

2016-09-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-17565: --- Attachment: generated_code.txt > Janino exception when calculating metrics for large

[jira] [Created] (SPARK-17565) Janino exception when calculating metrics for large generated class

2016-09-16 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-17565: -- Summary: Janino exception when calculating metrics for large generated class Key: SPARK-17565 URL: https://issues.apache.org/jira/browse/SPARK-17565 Project:

[jira] [Commented] (SPARK-5484) Pregel should checkpoint periodically to avoid StackOverflowError

2016-09-16 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496857#comment-15496857 ] ding commented on SPARK-5484: - Thank you for your kindly reminder. However as the code is almost ready, I will

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

2016-09-16 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496823#comment-15496823 ] Reynold Xin commented on SPARK-16534: - [~maver1ck] you don't need to wait on this do you? You can

[jira] [Updated] (SPARK-17079) broadcast decision based on cbo

2016-09-16 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-17079: - Description: We decide if broadcast join should be used based on the cardinality and size of

[jira] [Commented] (SPARK-14358) Change SparkListener from a trait to an abstract class, and remove JavaSparkListener

2016-09-16 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496798#comment-15496798 ] Reynold Xin commented on SPARK-14358: - [~osayankin] we thought about creating a compatibility package

[jira] [Commented] (SPARK-17564) Flaky RequestTimeoutIntegrationSuite, furtherRequestsDelay

2016-09-16 Thread Adam Roberts (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496699#comment-15496699 ] Adam Roberts commented on SPARK-17564: -- callback1.failure is sometimes null and due to this failing

[jira] [Updated] (SPARK-17564) Flaky RequestTimeoutIntegrationSuite, furtherRequestsDelay

2016-09-16 Thread Adam Roberts (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Roberts updated SPARK-17564: - Description: Could be related to [SPARK-10680] This is the test and one fix would be to

[jira] [Resolved] (SPARK-17563) Add org/apache/spark/JavaSparkListener to make Spark-2.0.0 work with Hive-2.X.X

2016-09-16 Thread Oleksiy Sayankin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin resolved SPARK-17563. -- Resolution: Won't Fix > Add org/apache/spark/JavaSparkListener to make Spark-2.0.0

[jira] [Commented] (SPARK-17563) Add org/apache/spark/JavaSparkListener to make Spark-2.0.0 work with Hive-2.X.X

2016-09-16 Thread Oleksiy Sayankin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496622#comment-15496622 ] Oleksiy Sayankin commented on SPARK-17563: -- Created

[jira] [Commented] (SPARK-17563) Add org/apache/spark/JavaSparkListener to make Spark-2.0.0 work with Hive-2.X.X

2016-09-16 Thread Oleksiy Sayankin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496614#comment-15496614 ] Oleksiy Sayankin commented on SPARK-17563: -- After three hours of fixing I have found out that

[jira] [Commented] (SPARK-17545) Spark SQL Catalyst doesn't handle ISO 8601 date without colon in offset

2016-09-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496579#comment-15496579 ] Hyukjin Kwon commented on SPARK-17545: -- Thank you both for your feedback. I am okay with fixing this

[jira] [Commented] (SPARK-17545) Spark SQL Catalyst doesn't handle ISO 8601 date without colon in offset

2016-09-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496525#comment-15496525 ] Sean Owen commented on SPARK-17545: --- I think it depends on how hard it is to handle the quirk and

[jira] [Commented] (SPARK-17545) Spark SQL Catalyst doesn't handle ISO 8601 date without colon in offset

2016-09-16 Thread Nathan Beyer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496505#comment-15496505 ] Nathan Beyer commented on SPARK-17545: -- I agree, the data is quirky and it's not how I'd personally

[jira] [Updated] (SPARK-17564) Flaky RequestTimeoutIntegrationSuite, furtherRequestsDelay

2016-09-16 Thread Adam Roberts (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Roberts updated SPARK-17564: - Description: Could be related to [SPARK-10680] This is the test and one fix would be to

[jira] [Updated] (SPARK-17564) Flaky RequestTimeoutIntegrationSuite, furtherRequestsDelay

2016-09-16 Thread Adam Roberts (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Roberts updated SPARK-17564: - Description: Could be related to [SPARK-10680] This is the test and one fix would be to

[jira] [Updated] (SPARK-17564) Flaky RequestTimeoutIntegrationSuite, furtherRequestsDelay

2016-09-16 Thread Adam Roberts (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Roberts updated SPARK-17564: - Component/s: Tests > Flaky RequestTimeoutIntegrationSuite, furtherRequestsDelay >

[jira] [Created] (SPARK-17564) Flaky RequestTimeoutIntegrationSuite, furtherRequestsDelay

2016-09-16 Thread Adam Roberts (JIRA)
Adam Roberts created SPARK-17564: Summary: Flaky RequestTimeoutIntegrationSuite, furtherRequestsDelay Key: SPARK-17564 URL: https://issues.apache.org/jira/browse/SPARK-17564 Project: Spark

[jira] [Issue Comment Deleted] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Jianfei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Wang updated SPARK-17562: - Comment: was deleted (was: 中秋快乐!谢谢。) > I think a little code is unnecessary to exist in >

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Jianfei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496324#comment-15496324 ] Jianfei Wang commented on SPARK-17562: -- 中秋快乐!谢谢。 > I think a little code is unnecessary to exist in

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Jianfei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496323#comment-15496323 ] Jianfei Wang commented on SPARK-17562: -- 中秋快乐!谢谢。 > I think a little code is unnecessary to exist in

[jira] [Commented] (SPARK-2352) [MLLIB] Add Artificial Neural Network (ANN) to Spark

2016-09-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496326#comment-15496326 ] Sean Owen commented on SPARK-2352: -- Maybe, but, that's also too broad to be actionable. There are still

[jira] [Issue Comment Deleted] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Jianfei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Wang updated SPARK-17562: - Comment: was deleted (was: 中秋快乐!谢谢。) > I think a little code is unnecessary to exist in >

[jira] [Commented] (SPARK-2352) [MLLIB] Add Artificial Neural Network (ANN) to Spark

2016-09-16 Thread Alessio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496312#comment-15496312 ] Alessio commented on SPARK-2352: Dear Sean, I am well aware of the MultiLayer Perceptron classifier. But

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496302#comment-15496302 ] Wenchen Fan commented on SPARK-17562: - I'm not familiar with this part, cc [~joshrosen] > I think a

[jira] [Comment Edited] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Jianfei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496283#comment-15496283 ] Jianfei Wang edited comment on SPARK-17562 at 9/16/16 1:03 PM: --- this func

[jira] [Comment Edited] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Jianfei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496283#comment-15496283 ] Jianfei Wang edited comment on SPARK-17562 at 9/16/16 1:03 PM: --- this func

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Jianfei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496283#comment-15496283 ] Jianfei Wang commented on SPARK-17562: -- this func is to revert writes that haven't been committed

[jira] [Commented] (SPARK-16121) ListingFileCatalog does not list in parallel anymore

2016-09-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496276#comment-15496276 ] Sean Owen commented on SPARK-16121: --- You can look at the branch/tag yourself:

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496251#comment-15496251 ] Sean Owen commented on SPARK-17562: --- If you do that, you don't call revertPartialWritesAndClose() then.

[jira] [Comment Edited] (SPARK-17563) Add org/apache/spark/JavaSparkListener to make Spark-2.0.0 work with Hive-2.X.X

2016-09-16 Thread Oleksiy Sayankin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496232#comment-15496232 ] Oleksiy Sayankin edited comment on SPARK-17563 at 9/16/16 12:40 PM:

[jira] [Commented] (SPARK-17563) Add org/apache/spark/JavaSparkListener to make Spark-2.0.0 work with Hive-2.X.X

2016-09-16 Thread Oleksiy Sayankin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496232#comment-15496232 ] Oleksiy Sayankin commented on SPARK-17563: -- Well I can change JavaSparkListener -->

[jira] [Commented] (SPARK-16121) ListingFileCatalog does not list in parallel anymore

2016-09-16 Thread Gaurav Shah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496234#comment-15496234 ] Gaurav Shah commented on SPARK-16121: - [~mengxr] was this fixed in 2.0.0 or is it planned for 2.0.1,

[jira] [Commented] (SPARK-17561) DataFrameWriter documentation formatting problems

2016-09-16 Thread Jagadeesan A S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496233#comment-15496233 ] Jagadeesan A S commented on SPARK-17561: It's okay.. :) > DataFrameWriter documentation

[jira] [Commented] (SPARK-17561) DataFrameWriter documentation formatting problems

2016-09-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496186#comment-15496186 ] Sean Owen commented on SPARK-17561: --- Oh, whoops I didn't actually see this until now. Sorry about that,

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Jianfei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496183#comment-15496183 ] Jianfei Wang commented on SPARK-17562: -- if 0 object is written, we should just set the success flag

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496167#comment-15496167 ] Sean Owen commented on SPARK-17562: --- Can you explain why? it looks possible that 0 objects are written.

[jira] [Commented] (SPARK-17563) Add org/apache/spark/JavaSparkListener to make Spark-2.0.0 work with Hive-2.X.X

2016-09-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496162#comment-15496162 ] Sean Owen commented on SPARK-17563: --- JobMetricsListener is not part of Spark, right? It needs to change

[jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Jianfei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496147#comment-15496147 ] Jianfei Wang commented on SPARK-17562: -- [~cloud_fan] can you check this? thank you! > I think a

[jira] [Updated] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Jianfei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Wang updated SPARK-17562: - Description: In ExternalSorter.spillMemoryIteratorToDisk, I think the code below will never be

[jira] [Updated] (SPARK-17563) Add org/apache/spark/JavaSparkListener to make Spark-2.0.0 work with Hive-2.X.X

2016-09-16 Thread Oleksiy Sayankin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated SPARK-17563: - Summary: Add org/apache/spark/JavaSparkListener to make Spark-2.0.0 work with Hive-2.X.X

[jira] [Created] (SPARK-17563) Add org/apache/spark/JavaSparkListener to make Spark-2.0.0 work with Hive-2.0.0

2016-09-16 Thread Oleksiy Sayankin (JIRA)
Oleksiy Sayankin created SPARK-17563: Summary: Add org/apache/spark/JavaSparkListener to make Spark-2.0.0 work with Hive-2.0.0 Key: SPARK-17563 URL: https://issues.apache.org/jira/browse/SPARK-17563

[jira] [Created] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

2016-09-16 Thread Jianfei Wang (JIRA)
Jianfei Wang created SPARK-17562: Summary: I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk Key: SPARK-17562 URL: https://issues.apache.org/jira/browse/SPARK-17562

[jira] [Commented] (SPARK-14358) Change SparkListener from a trait to an abstract class, and remove JavaSparkListener

2016-09-16 Thread Oleksiy Sayankin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496130#comment-15496130 ] Oleksiy Sayankin commented on SPARK-14358: -- Hive-2.X.X uses JavaSparkListener {code} package

[jira] [Updated] (SPARK-17426) Current TreeNode.toJSON may trigger OOM under some corner cases

2016-09-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-17426: Assignee: Sean Zhong > Current TreeNode.toJSON may trigger OOM under some corner cases >

[jira] [Updated] (SPARK-17426) Current TreeNode.toJSON may trigger OOM under some corner cases

2016-09-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-17426: Fix Version/s: (was: 2.2.0) 2.1.0 > Current TreeNode.toJSON may trigger OOM

[jira] [Resolved] (SPARK-17426) Current TreeNode.toJSON may trigger OOM under some corner cases

2016-09-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-17426. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 14990

[jira] [Assigned] (SPARK-17561) DataFrameWriter documentation formatting problems

2016-09-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17561: Assignee: (was: Apache Spark) > DataFrameWriter documentation formatting problems >

[jira] [Commented] (SPARK-17561) DataFrameWriter documentation formatting problems

2016-09-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496022#comment-15496022 ] Apache Spark commented on SPARK-17561: -- User 'srowen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17561) DataFrameWriter documentation formatting problems

2016-09-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17561: Assignee: Apache Spark > DataFrameWriter documentation formatting problems >

[jira] [Commented] (SPARK-5484) Pregel should checkpoint periodically to avoid StackOverflowError

2016-09-16 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495984#comment-15495984 ] Takeshi Yamamuro commented on SPARK-5484: - This component seems very inactive, so I think there is

[jira] [Commented] (SPARK-17560) SQLContext tables returns table names in lower case only

2016-09-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495968#comment-15495968 ] Sean Owen commented on SPARK-17560: --- You just set it with --conf like other options. However I don't

[jira] [Commented] (SPARK-17560) SQLContext tables returns table names in lower case only

2016-09-16 Thread Aseem Bansal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495960#comment-15495960 ] Aseem Bansal commented on SPARK-17560: -- Can you share where this option needs to be set? Maybe I can

[jira] [Commented] (SPARK-17560) SQLContext tables returns table names in lower case only

2016-09-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495955#comment-15495955 ] Sean Owen commented on SPARK-17560: --- It doesn't exist in Spark 1.4, and is an undocumented option.

[jira] [Updated] (SPARK-17561) DataFrameWriter documentation formatting problems

2016-09-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-17561: -- Affects Version/s: 2.0.0 Priority: Trivial (was: Major) Component/s: SQL See

[jira] [Commented] (SPARK-13210) NPE in Sort

2016-09-16 Thread Tzach Zohar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495921#comment-15495921 ] Tzach Zohar commented on SPARK-13210: - I've just seen this happening on Spark 1.6.2 - very similar

[jira] [Commented] (SPARK-17560) SQLContext tables returns table names in lower case only

2016-09-16 Thread Aseem Bansal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495906#comment-15495906 ] Aseem Bansal commented on SPARK-17560: -- Looked through

[jira] [Commented] (SPARK-17561) DataFrameWriter documentation formatting problems

2016-09-16 Thread Jagadeesan A S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495899#comment-15495899 ] Jagadeesan A S commented on SPARK-17561: [~anshbansal] Thanks for reporting. Started working on

[jira] [Updated] (SPARK-17561) DataFrameWriter documentation formatting problems

2016-09-16 Thread Aseem Bansal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aseem Bansal updated SPARK-17561: - Description: I visited this page

[jira] [Updated] (SPARK-17561) DataFrameWriter documentation formatting problems

2016-09-16 Thread Aseem Bansal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aseem Bansal updated SPARK-17561: - Description: I visited this page

[jira] [Updated] (SPARK-17561) DataFrameWriter documentation formatting problems

2016-09-16 Thread Aseem Bansal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aseem Bansal updated SPARK-17561: - Attachment: screenshot-1.png > DataFrameWriter documentation formatting problems >

[jira] [Created] (SPARK-17561) DataFrameWriter documentation formatting problems

2016-09-16 Thread Aseem Bansal (JIRA)
Aseem Bansal created SPARK-17561: Summary: DataFrameWriter documentation formatting problems Key: SPARK-17561 URL: https://issues.apache.org/jira/browse/SPARK-17561 Project: Spark Issue

[jira] [Commented] (SPARK-17560) SQLContext tables returns table names in lower case only

2016-09-16 Thread Aseem Bansal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495862#comment-15495862 ] Aseem Bansal commented on SPARK-17560: -- No I did not. Where? > SQLContext tables returns table

[jira] [Comment Edited] (SPARK-17560) SQLContext tables returns table names in lower case only

2016-09-16 Thread Aseem Bansal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495862#comment-15495862 ] Aseem Bansal edited comment on SPARK-17560 at 9/16/16 9:38 AM: --- No I did

[jira] [Resolved] (SPARK-17534) Increase timeouts for DirectKafkaStreamSuite tests

2016-09-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17534. --- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15094

[jira] [Updated] (SPARK-17543) Missing log4j config file for tests in common/network-shuffle

2016-09-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-17543: -- Assignee: Jagadeesan A S > Missing log4j config file for tests in common/network-shuffle >

[jira] [Resolved] (SPARK-17543) Missing log4j config file for tests in common/network-shuffle

2016-09-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17543. --- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15108

[jira] [Commented] (SPARK-17560) SQLContext tables returns table names in lower case only

2016-09-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495832#comment-15495832 ] Sean Owen commented on SPARK-17560: --- Did you set spark.sql.caseSensitive=true? > SQLContext tables

[jira] [Created] (SPARK-17560) SQLContext tables returns table names in lower case only

2016-09-16 Thread Aseem Bansal (JIRA)
Aseem Bansal created SPARK-17560: Summary: SQLContext tables returns table names in lower case only Key: SPARK-17560 URL: https://issues.apache.org/jira/browse/SPARK-17560 Project: Spark

[jira] [Resolved] (SPARK-6385) ISO 8601 timestamp parsing does not support arbitrary precision second fractions

2016-09-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-6385. -- Resolution: Duplicate > ISO 8601 timestamp parsing does not support arbitrary precision second >

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

2016-09-16 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495496#comment-15495496 ] Maciej Bryński commented on SPARK-16534: [~rxin] I understand your point of view, but I think

  1   2   >