[jira] [Commented] (SPARK-27121) Resolve Scala compiler failure for Java 9+ in REPL

[ https://issues.apache.org/jira/browse/SPARK-27121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804555#comment-16804555 ] Sean Owen commented on SPARK-27121: --- Yeah you're right. I got the tests to pass without much change.

[jira] [Commented] (SPARK-24630) SPIP: Support SQLStreaming in Spark

[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804544#comment-16804544 ] Jackey Lee commented on SPARK-24630: [~KevinZwx] More discussions are in this doc.  

[jira] [Commented] (SPARK-24630) SPIP: Support SQLStreaming in Spark

[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804535#comment-16804535 ] Kevin Zhang commented on SPARK-24630: - Hi, is there any update about the progress? > SPIP: Support

[jira] [Assigned] (SPARK-27252) Make current_date() independent from time zones

[ https://issues.apache.org/jira/browse/SPARK-27252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27252: --- Assignee: Maxim Gekk > Make current_date() independent from time zones >

[jira] [Resolved] (SPARK-27252) Make current_date() independent from time zones

[ https://issues.apache.org/jira/browse/SPARK-27252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27252. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24185

[jira] [Resolved] (SPARK-27293) Setting random seed produces different results in RandomForestRegressor

[ https://issues.apache.org/jira/browse/SPARK-27293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27293. -- Resolution: Not A Problem > Setting random seed produces different results in

[jira] [Resolved] (SPARK-27292) Spark Job Fails with Unknown Error writing to S3 from AWS EMR

[ https://issues.apache.org/jira/browse/SPARK-27292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27292. -- Resolution: Invalid Yes, questions should go to the mailing list. > Spark Job Fails with

[jira] [Updated] (SPARK-27298) Dataset except operation gives different results(dataset count) on Spark 2.3.0 Windows and Spark 2.3.0 Linux environment

[ https://issues.apache.org/jira/browse/SPARK-27298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27298: - Priority: Major (was: Critical) > Dataset except operation gives different results(dataset

[jira] [Commented] (SPARK-27298) Dataset except operation gives different results(dataset count) on Spark 2.3.0 Windows and Spark 2.3.0 Linux environment

[ https://issues.apache.org/jira/browse/SPARK-27298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804483#comment-16804483 ] Hyukjin Kwon commented on SPARK-27298: -- Please avoid to set Critical+ which is usually reserved for

[jira] [Updated] (SPARK-27318) Join operation on bucketing table fails with base adaptive enabled

[ https://issues.apache.org/jira/browse/SPARK-27318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27318: - Description: Join Operation on bucketed table is failing. Steps to reproduce the issue.

[jira] [Updated] (SPARK-27318) Join operation on bucketing table fails with base adaptive enabled

[ https://issues.apache.org/jira/browse/SPARK-27318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27318: - Description: Join Operation on bucketed table is failing. Steps to reproduce the issue.

[jira] [Commented] (SPARK-27216) Kryo serialization with RoaringBitmap

[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804447#comment-16804447 ] Andrew Allen commented on SPARK-27216: -- Thanks for this bug report, [~cltlfcjin]. We we running

[jira] [Created] (SPARK-27321) SortMerge join on partition keys causes shuffle

Lekshmi Nair created SPARK-27321: Summary: SortMerge join on partition keys causes shuffle Key: SPARK-27321 URL: https://issues.apache.org/jira/browse/SPARK-27321 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-27098) Flaky missing file parts when writing to Ceph without error

[ https://issues.apache.org/jira/browse/SPARK-27098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804294#comment-16804294 ] Martin Loncaric edited comment on SPARK-27098 at 3/28/19 8:53 PM: -- An

[jira] [Commented] (SPARK-27098) Flaky missing file parts when writing to Ceph without error

[ https://issues.apache.org/jira/browse/SPARK-27098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804294#comment-16804294 ] Martin Loncaric commented on SPARK-27098: - An INFO of this format also appears for each missing

[jira] [Commented] (SPARK-27121) Resolve Scala compiler failure for Java 9+ in REPL

[ https://issues.apache.org/jira/browse/SPARK-27121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804184#comment-16804184 ] Imran Rashid commented on SPARK-27121: -- I think this might just be an issue with the test setup.

[jira] [Commented] (SPARK-27296) User Defined Aggregating Functions (UDAFs) have a major efficiency problem

[ https://issues.apache.org/jira/browse/SPARK-27296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804176#comment-16804176 ] Erik Erlandson commented on SPARK-27296: My initial proposal would be to alter the logic

[jira] [Updated] (SPARK-27275) Potential corruption in EncryptedMessage.transferTo

[ https://issues.apache.org/jira/browse/SPARK-27275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-27275: Fix Version/s: 2.4.2 2.3.4 > Potential corruption in

[jira] [Updated] (SPARK-27293) Setting random seed produces different results in RandomForestRegressor

[ https://issues.apache.org/jira/browse/SPARK-27293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-27293: - Component/s: ML > Setting random seed produces different results in RandomForestRegressor >

[jira] [Updated] (SPARK-27293) Setting random seed produces different results in RandomForestRegressor

[ https://issues.apache.org/jira/browse/SPARK-27293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-27293: - Description: I am interested in finding out if there is a bug in the implementation of

[jira] [Commented] (SPARK-27293) I am interested in finding out if there is a bug in the implementation of RandomForests. The Issue is when applying a seed and getting different results than other peo

[ https://issues.apache.org/jira/browse/SPARK-27293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804124#comment-16804124 ] Bryan Cutler commented on SPARK-27293: -- Setting the seed like in your example for randomSplit and

[jira] [Updated] (SPARK-27293) Setting random seed produces different results in RandomForestRegressor

[ https://issues.apache.org/jira/browse/SPARK-27293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-27293: - Summary: Setting random seed produces different results in RandomForestRegressor (was: I am

[jira] [Updated] (SPARK-26771) Make .unpersist(), .destroy() consistently non-blocking by default

[ https://issues.apache.org/jira/browse/SPARK-26771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-26771: -- Docs Text: The RDD and DataFrame .unpersist() method, and Broadcast .destroy() method, take an

[jira] [Resolved] (SPARK-26914) ThriftServer scheduler pool may be unpredictably when using fair schedule mode

[ https://issues.apache.org/jira/browse/SPARK-26914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26914. --- Resolution: Fixed Assignee: zhoukang Fix Version/s: 3.0.0 Resolved by

[jira] [Resolved] (SPARK-27283) BigDecimal arithmetic losing precision

[ https://issues.apache.org/jira/browse/SPARK-27283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mats resolved SPARK-27283. -- Resolution: Not A Bug Closing because this seems to be a design decision with no direct conflicts with

[jira] [Created] (SPARK-27320) Converting seq to array in AggregationIterator to improve its access performance

eaton created SPARK-27320: - Summary: Converting seq to array in AggregationIterator to improve its access performance Key: SPARK-27320 URL: https://issues.apache.org/jira/browse/SPARK-27320 Project: Spark

[jira] [Commented] (SPARK-27258) The value of "spark.app.name" or "--name" starts with number , which causes resourceName does not match regular expression

[ https://issues.apache.org/jira/browse/SPARK-27258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16803777#comment-16803777 ] hehuiyuan commented on SPARK-27258: --- I have submitted a PR . Resolve the name of Pod & Service that

[jira] [Commented] (SPARK-27283) BigDecimal arithmetic losing precision

[ https://issues.apache.org/jira/browse/SPARK-27283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16803767#comment-16803767 ] Marco Gaido commented on SPARK-27283: - {quote} I guess I'm mostly frustrated that the SQL standard

[jira] [Commented] (SPARK-27283) BigDecimal arithmetic losing precision

[ https://issues.apache.org/jira/browse/SPARK-27283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16803759#comment-16803759 ] Mats commented on SPARK-27283: -- What MySQL seems to be doing is to fail with an error when the double is

[jira] [Created] (SPARK-27319) Filter out dir based on PathFilter before listing them

Xianyang Liu created SPARK-27319: Summary: Filter out dir based on PathFilter before listing them Key: SPARK-27319 URL: https://issues.apache.org/jira/browse/SPARK-27319 Project: Spark Issue

[jira] [Created] (SPARK-27318) Join operation on bucketing table fails with base adaptive enabled

Supritha created SPARK-27318: Summary: Join operation on bucketing table fails with base adaptive enabled Key: SPARK-27318 URL: https://issues.apache.org/jira/browse/SPARK-27318 Project: Spark

[jira] [Commented] (SPARK-27290) remove unneed sort under Aggregate

[ https://issues.apache.org/jira/browse/SPARK-27290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16803737#comment-16803737 ] Xiaoju Wu commented on SPARK-27290: --- [~ekoifman] HashAggregate can not benefit from sorted input but

[jira] [Created] (SPARK-27317) New graph component user guide (algorithms)

Xiangrui Meng created SPARK-27317: - Summary: New graph component user guide (algorithms) Key: SPARK-27317 URL: https://issues.apache.org/jira/browse/SPARK-27317 Project: Spark Issue Type:

[jira] [Created] (SPARK-27316) Query-based PropertyGraph algorithms

Xiangrui Meng created SPARK-27316: - Summary: Query-based PropertyGraph algorithms Key: SPARK-27316 URL: https://issues.apache.org/jira/browse/SPARK-27316 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-27315) GraphX-based PropertyGraph algorithms

Xiangrui Meng created SPARK-27315: - Summary: GraphX-based PropertyGraph algorithms Key: SPARK-27315 URL: https://issues.apache.org/jira/browse/SPARK-27315 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-27314) Deduplicate exprIds for Union.

Takuya Ueshin created SPARK-27314: - Summary: Deduplicate exprIds for Union. Key: SPARK-27314 URL: https://issues.apache.org/jira/browse/SPARK-27314 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-27313) DataFrame-based PropertyGraph algorithms

Xiangrui Meng created SPARK-27313: - Summary: DataFrame-based PropertyGraph algorithms Key: SPARK-27313 URL: https://issues.apache.org/jira/browse/SPARK-27313 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-27312) PropertyGraph <-> GraphX conversions

Xiangrui Meng created SPARK-27312: - Summary: PropertyGraph <-> GraphX conversions Key: SPARK-27312 URL: https://issues.apache.org/jira/browse/SPARK-27312 Project: Spark Issue Type: Story

[jira] [Updated] (SPARK-27309) PropertyGraph SparkSession adapter (Scala/Java)

[ https://issues.apache.org/jira/browse/SPARK-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27309: -- Description: Create SparkSession adapter to simply ProperGraph creation (Scala implicits?

[jira] [Updated] (SPARK-27309) PropertyGraph SparkSession adapter (Scala/Java)

[ https://issues.apache.org/jira/browse/SPARK-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27309: -- Summary: PropertyGraph SparkSession adapter (Scala/Java) (was: ProperGraph SparkSession

[jira] [Updated] (SPARK-27300) Create the new graph projects in Spark and set up build/test

[ https://issues.apache.org/jira/browse/SPARK-27300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27300: -- Description: * Create graph projects in Spark repo that works with both maven and sbt. * Add

[jira] [Created] (SPARK-27311) New graph component user guide (PropertyGraph / Query)

Xiangrui Meng created SPARK-27311: - Summary: New graph component user guide (PropertyGraph / Query) Key: SPARK-27311 URL: https://issues.apache.org/jira/browse/SPARK-27311 Project: Spark

[jira] [Created] (SPARK-27310) PropertyGraph SparkSession adapter (Python)

Xiangrui Meng created SPARK-27310: - Summary: PropertyGraph SparkSession adapter (Python) Key: SPARK-27310 URL: https://issues.apache.org/jira/browse/SPARK-27310 Project: Spark Issue Type:

[jira] [Created] (SPARK-27309) ProperGraph SparkSession adapter (Scala/Java)

Xiangrui Meng created SPARK-27309: - Summary: ProperGraph SparkSession adapter (Scala/Java) Key: SPARK-27309 URL: https://issues.apache.org/jira/browse/SPARK-27309 Project: Spark Issue Type:

[jira] [Created] (SPARK-27307) PropertyGraph save/load (Python)

Xiangrui Meng created SPARK-27307: - Summary: PropertyGraph save/load (Python) Key: SPARK-27307 URL: https://issues.apache.org/jira/browse/SPARK-27307 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-27308) PropertyGraph Cypher query (Python)

Xiangrui Meng created SPARK-27308: - Summary: PropertyGraph Cypher query (Python) Key: SPARK-27308 URL: https://issues.apache.org/jira/browse/SPARK-27308 Project: Spark Issue Type: Story

[jira] [Updated] (SPARK-27306) PropertyGraph construction (Python)

[ https://issues.apache.org/jira/browse/SPARK-27306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27306: -- Description: Python API and test for SPARK-27303. (was: Python API for SPARK-27303.) >

[jira] [Created] (SPARK-27306) PropertyGraph construction (Python)

Xiangrui Meng created SPARK-27306: - Summary: PropertyGraph construction (Python) Key: SPARK-27306 URL: https://issues.apache.org/jira/browse/SPARK-27306 Project: Spark Issue Type: Story

[jira] [Updated] (SPARK-25994) SPIP: Property Graphs, Cypher Queries, and Algorithms

[ https://issues.apache.org/jira/browse/SPARK-25994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25994: -- Epic Name: Property Graph / Cypher / Algorithms > SPIP: Property Graphs, Cypher Queries, and

[jira] [Created] (SPARK-27305) PropertyGraph Cypher query (Java/Scala)

Xiangrui Meng created SPARK-27305: - Summary: PropertyGraph Cypher query (Java/Scala) Key: SPARK-27305 URL: https://issues.apache.org/jira/browse/SPARK-27305 Project: Spark Issue Type: Story

[jira] [Updated] (SPARK-27300) Create the new graph projects in Spark and set up build/test

[ https://issues.apache.org/jira/browse/SPARK-27300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27300: -- Summary: Create the new graph projects in Spark and set up build/test (was: Create the new

[jira] [Created] (SPARK-27304) PropertyGraph save/load (Scala/Java)

Xiangrui Meng created SPARK-27304: - Summary: PropertyGraph save/load (Scala/Java) Key: SPARK-27304 URL: https://issues.apache.org/jira/browse/SPARK-27304 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-27303) PropertyGraph construction (Scala/Java)

Xiangrui Meng created SPARK-27303: - Summary: PropertyGraph construction (Scala/Java) Key: SPARK-27303 URL: https://issues.apache.org/jira/browse/SPARK-27303 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-27302) Design: graph algorithms based on property graphs

Xiangrui Meng created SPARK-27302: - Summary: Design: graph algorithms based on property graphs Key: SPARK-27302 URL: https://issues.apache.org/jira/browse/SPARK-27302 Project: Spark Issue

[jira] [Updated] (SPARK-27301) DStreamCheckpointData failed to clean up because it's fileSystem cached

[ https://issues.apache.org/jira/browse/SPARK-27301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-27301: - Description: The cached FileSystem's token will expire if no tokens explicitly are add into it.

[jira] [Created] (SPARK-27301) DStreamCheckpointData failed to clean up because it's fileSystem cached

Kent Yao created SPARK-27301: Summary: DStreamCheckpointData failed to clean up because it's fileSystem cached Key: SPARK-27301 URL: https://issues.apache.org/jira/browse/SPARK-27301 Project: Spark

[jira] [Created] (SPARK-27300) Create the new graph projects in Spark

Xiangrui Meng created SPARK-27300: - Summary: Create the new graph projects in Spark Key: SPARK-27300 URL: https://issues.apache.org/jira/browse/SPARK-27300 Project: Spark Issue Type: Story

[jira] [Updated] (SPARK-27299) Design: Property graph construction, save/load, and query APIs

[ https://issues.apache.org/jira/browse/SPARK-27299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27299: -- Description: Design doc for property graph and Cypher queries. * Construct a property graph.

[jira] [Updated] (SPARK-27299) Design: Property graph construction, save/load, and query APIs

[ https://issues.apache.org/jira/browse/SPARK-27299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27299: -- Description: Design doc for property graph and Cypher queries. * Construct a property graph.

[jira] [Created] (SPARK-27299) Design: Property graph construction, save/load, and query APIs

Xiangrui Meng created SPARK-27299: - Summary: Design: Property graph construction, save/load, and query APIs Key: SPARK-27299 URL: https://issues.apache.org/jira/browse/SPARK-27299 Project: Spark

[jira] [Updated] (SPARK-25994) SPIP: Property Graphs, Cypher Queries, and Algorithms

[ https://issues.apache.org/jira/browse/SPARK-25994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25994: -- Issue Type: Epic (was: New Feature) > SPIP: Property Graphs, Cypher Queries, and Algorithms

[jira] [Updated] (SPARK-25994) SPIP: Property Graphs, Cypher Queries, and Algorithms

[ https://issues.apache.org/jira/browse/SPARK-25994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25994: -- Component/s: (was: GraphX) Graph > SPIP: Property Graphs, Cypher

[jira] [Comment Edited] (SPARK-24130) Data Source V2: Join Push Down

[ https://issues.apache.org/jira/browse/SPARK-24130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780735#comment-16780735 ] William Wong edited comment on SPARK-24130 at 3/28/19 7:31 AM: --- Hi