[jira] [Resolved] (SPARK-26637) Makes GetArrayItem nullability more precise

2019-01-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-26637. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23566 [https://gith

[jira] [Assigned] (SPARK-26637) Makes GetArrayItem nullability more precise

2019-01-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-26637: --- Assignee: Takeshi Yamamuro > Makes GetArrayItem nullability more precise >

[jira] [Created] (SPARK-26699) Dataset column discrepancies between Parquet

2019-01-22 Thread Lakshmi Praveena (JIRA)
Lakshmi Praveena created SPARK-26699: Summary: Dataset column discrepancies between Parquet Key: SPARK-26699 URL: https://issues.apache.org/jira/browse/SPARK-26699 Project: Spark Issue T

[jira] [Commented] (SPARK-26678) Empty values end up as quoted empty strings in CSV files

2019-01-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749598#comment-16749598 ] Hyukjin Kwon commented on SPARK-26678: -- We should distinguish empty string and miss

[jira] [Resolved] (SPARK-26678) Empty values end up as quoted empty strings in CSV files

2019-01-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26678. -- Resolution: Not A Problem > Empty values end up as quoted empty strings in CSV files > ---

[jira] [Updated] (SPARK-26699) Dataset column output discrepancies

2019-01-22 Thread Lakshmi Praveena (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lakshmi Praveena updated SPARK-26699: - Summary: Dataset column output discrepancies (was: Dataset column discrepancies betwee

[jira] [Created] (SPARK-26698) Use ConfigEntry for hardcoded configs for memory and storage categories

2019-01-22 Thread SongYadong (JIRA)
SongYadong created SPARK-26698: -- Summary: Use ConfigEntry for hardcoded configs for memory and storage categories Key: SPARK-26698 URL: https://issues.apache.org/jira/browse/SPARK-26698 Project: Spark

[jira] [Assigned] (SPARK-26698) Use ConfigEntry for hardcoded configs for memory and storage categories

2019-01-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26698: Assignee: (was: Apache Spark) > Use ConfigEntry for hardcoded configs for memory and

[jira] [Assigned] (SPARK-26698) Use ConfigEntry for hardcoded configs for memory and storage categories

2019-01-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26698: Assignee: Apache Spark > Use ConfigEntry for hardcoded configs for memory and storage cat

[jira] [Resolved] (SPARK-19478) JDBC Sink

2019-01-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-19478. -- Resolution: Not A Problem Im resolving per https://github.com/apache/spark/pull/17190#issueco

[jira] [Commented] (SPARK-26679) Deconflict spark.executor.pyspark.memory and spark.python.worker.memory

2019-01-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749557#comment-16749557 ] Hyukjin Kwon commented on SPARK-26679: -- [~rdblue], if there's no case we can curren

[jira] [Assigned] (SPARK-26696) Dataset encoder should be publicly accessible

2019-01-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26696: Assignee: (was: Apache Spark) > Dataset encoder should be publicly accessible > -

[jira] [Assigned] (SPARK-26677) Incorrect results of not(eqNullSafe) when data read from Parquet file

2019-01-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26677: Assignee: (was: Apache Spark) > Incorrect results of not(eqNullSafe) when data read f

[jira] [Assigned] (SPARK-26677) Incorrect results of not(eqNullSafe) when data read from Parquet file

2019-01-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26677: Assignee: Apache Spark > Incorrect results of not(eqNullSafe) when data read from Parquet

[jira] [Commented] (SPARK-26677) Incorrect results of not(eqNullSafe) when data read from Parquet file

2019-01-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749526#comment-16749526 ] Hyukjin Kwon commented on SPARK-26677: -- Im gonna open a PR soon. > Incorrect resul

[jira] [Assigned] (SPARK-26697) ShuffleBlockFetcherIterator can log block sizes in addition to num blocks

2019-01-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26697: Assignee: Apache Spark > ShuffleBlockFetcherIterator can log block sizes in addition to n

[jira] [Assigned] (SPARK-26697) ShuffleBlockFetcherIterator can log block sizes in addition to num blocks

2019-01-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26697: Assignee: (was: Apache Spark) > ShuffleBlockFetcherIterator can log block sizes in ad

[jira] [Created] (SPARK-26697) ShuffleBlockFetcherIterator can log block sizes in addition to num blocks

2019-01-22 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-26697: Summary: ShuffleBlockFetcherIterator can log block sizes in addition to num blocks Key: SPARK-26697 URL: https://issues.apache.org/jira/browse/SPARK-26697 Project: Sp

[jira] [Updated] (SPARK-26677) Incorrect results of not(eqNullSafe) when data read from Parquet file

2019-01-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26677: - Priority: Blocker (was: Major) > Incorrect results of not(eqNullSafe) when data read from Parqu

[jira] [Updated] (SPARK-26677) Incorrect results of not(eqNullSafe) when data read from Parquet file

2019-01-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26677: - Labels: correctness (was: ) > Incorrect results of not(eqNullSafe) when data read from Parquet

[jira] [Commented] (SPARK-26696) Dataset encoder should be publicly accessible

2019-01-22 Thread Simeon Simeonov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749518#comment-16749518 ] Simeon Simeonov commented on SPARK-26696: - [PR with improvement|https://github.c

[jira] [Assigned] (SPARK-26696) Dataset encoder should be publicly accessible

2019-01-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26696: Assignee: Apache Spark > Dataset encoder should be publicly accessible >

[jira] [Created] (SPARK-26696) Dataset encoder should be publicly accessible

2019-01-22 Thread Simeon Simeonov (JIRA)
Simeon Simeonov created SPARK-26696: --- Summary: Dataset encoder should be publicly accessible Key: SPARK-26696 URL: https://issues.apache.org/jira/browse/SPARK-26696 Project: Spark Issue Typ

[jira] [Commented] (SPARK-26668) One Kafka broker serve is down,the spark streaming start consuming delay

2019-01-22 Thread Chang Quanyou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749484#comment-16749484 ] Chang Quanyou commented on SPARK-26668: --- NO, I try it use 2.2.2 version; Not works

[jira] [Assigned] (SPARK-26695) data source V2 API refactoring (continuous read)

2019-01-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26695: Assignee: Wenchen Fan (was: Apache Spark) > data source V2 API refactoring (continuous r

[jira] [Assigned] (SPARK-26695) data source V2 API refactoring (continuous read)

2019-01-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26695: Assignee: Apache Spark (was: Wenchen Fan) > data source V2 API refactoring (continuous r

[jira] [Created] (SPARK-26695) data source V2 API refactoring (continuous read)

2019-01-22 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-26695: --- Summary: data source V2 API refactoring (continuous read) Key: SPARK-26695 URL: https://issues.apache.org/jira/browse/SPARK-26695 Project: Spark Issue Type: Su

[jira] [Updated] (SPARK-26677) Incorrect results of not(eqNullSafe) when data read from Parquet file

2019-01-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26677: - Component/s: (was: Spark Core) SQL > Incorrect results of not(eqNullSafe) w

[jira] [Commented] (SPARK-26427) Upgrade Apache ORC to 1.5.4

2019-01-22 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749400#comment-16749400 ] Dongjoon Hyun commented on SPARK-26427: --- It's only ORC dependency changes. {code}

[jira] [Updated] (SPARK-26677) Incorrect results of not(eqNullSafe) when data read from Parquet file

2019-01-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26677: - Priority: Major (was: Critical) > Incorrect results of not(eqNullSafe) when data read from Parq

[jira] [Commented] (SPARK-26668) One Kafka broker serve is down,the spark streaming start consuming delay

2019-01-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749396#comment-16749396 ] Hyukjin Kwon commented on SPARK-26668: -- [~quanyou.chang], are you able to test this

[jira] [Commented] (SPARK-26427) Upgrade Apache ORC to 1.5.4

2019-01-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749391#comment-16749391 ] Wenchen Fan commented on SPARK-26427: - does it include other transitive dependences

[jira] [Assigned] (SPARK-26228) OOM issue encountered when computing Gramian matrix

2019-01-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26228: - Assignee: Sean Owen > OOM issue encountered when computing Gramian matrix > --

[jira] [Resolved] (SPARK-26228) OOM issue encountered when computing Gramian matrix

2019-01-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26228. --- Resolution: Fixed Fix Version/s: 2.3.4 2.4.1 3.0.0 Issu

[jira] [Assigned] (SPARK-26694) Console progress bar not showing in 3.0

2019-01-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26694: Assignee: (was: Apache Spark) > Console progress bar not showing in 3.0 > ---

[jira] [Assigned] (SPARK-26694) Console progress bar not showing in 3.0

2019-01-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26694: Assignee: Apache Spark > Console progress bar not showing in 3.0 > --

[jira] [Assigned] (SPARK-26605) New executors failing with expired tokens in client mode

2019-01-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26605: -- Assignee: Marcelo Vanzin > New executors failing with expired tokens in client mode >

[jira] [Resolved] (SPARK-26605) New executors failing with expired tokens in client mode

2019-01-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26605. Resolution: Fixed Fix Version/s: 2.4.1 Issue resolved by pull request 23523 [https:

[jira] [Resolved] (SPARK-24484) Power Iteration Clustering is giving incorrect clustering results when there are mutiple leading eigen values.

2019-01-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-24484. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21627 [https://github.c

[jira] [Assigned] (SPARK-24484) Power Iteration Clustering is giving incorrect clustering results when there are mutiple leading eigen values.

2019-01-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-24484: - Assignee: shahid > Power Iteration Clustering is giving incorrect clustering results when there

[jira] [Commented] (SPARK-26694) Console progress bar not showing in 3.0

2019-01-22 Thread Ankur Gupta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749280#comment-16749280 ] Ankur Gupta commented on SPARK-26694: - I am working on this > Console progress bar

[jira] [Updated] (SPARK-21708) use sbt 1.x

2019-01-22 Thread PJ Fanning (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PJ Fanning updated SPARK-21708: --- Summary: use sbt 1.x (was: use sbt 1.0.0) > use sbt 1.x > --- > > Key: SPAR

[jira] [Commented] (SPARK-26427) Upgrade Apache ORC to 1.5.4

2019-01-22 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749223#comment-16749223 ] Dongjoon Hyun commented on SPARK-26427: --- Hi, [~smilegator] and [~cloud_fan]. In ge

[jira] [Comment Edited] (SPARK-23505) Flaky test: ParquetQuerySuite

2019-01-22 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749220#comment-16749220 ] Dongjoon Hyun edited comment on SPARK-23505 at 1/22/19 10:31 PM: -

[jira] [Commented] (SPARK-23505) Flaky test: ParquetQuerySuite

2019-01-22 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749220#comment-16749220 ] Dongjoon Hyun commented on SPARK-23505: --- I've monitoring the Jenkins. At least, fo

[jira] [Created] (SPARK-26694) Console progress bar not showing in 3.0

2019-01-22 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-26694: -- Summary: Console progress bar not showing in 3.0 Key: SPARK-26694 URL: https://issues.apache.org/jira/browse/SPARK-26694 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-26661) Show actual class name of the writing command in CTAS explain

2019-01-22 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-26661. --- Resolution: Fixed Assignee: Kris Mok Fix Version/s: 3.0.0 This is resolved v

[jira] [Created] (SPARK-26693) Large Numbers Truncated

2019-01-22 Thread Jason Blahovec (JIRA)
Jason Blahovec created SPARK-26693: -- Summary: Large Numbers Truncated Key: SPARK-26693 URL: https://issues.apache.org/jira/browse/SPARK-26693 Project: Spark Issue Type: Bug Compon

[jira] [Commented] (SPARK-26608) Remove Jenkins jobs for `branch-2.2`

2019-01-22 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749156#comment-16749156 ] shane knapp commented on SPARK-26608: - no, it's for the jenkins job builder configs

[jira] [Commented] (SPARK-26608) Remove Jenkins jobs for `branch-2.2`

2019-01-22 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749151#comment-16749151 ] Dongjoon Hyun commented on SPARK-26608: --- Thanks! Is the pending spark config PR fo

[jira] [Comment Edited] (SPARK-26608) Remove Jenkins jobs for `branch-2.2`

2019-01-22 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749137#comment-16749137 ] shane knapp edited comment on SPARK-26608 at 1/22/19 8:48 PM:

[jira] [Commented] (SPARK-26608) Remove Jenkins jobs for `branch-2.2`

2019-01-22 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749137#comment-16749137 ] shane knapp commented on SPARK-26608: - they're all disabled, and i'm waiting on the

[jira] [Commented] (SPARK-26608) Remove Jenkins jobs for `branch-2.2`

2019-01-22 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749122#comment-16749122 ] Dongjoon Hyun commented on SPARK-26608: --- Oh, sorry. I forgot to reply here. Yes, r

[jira] [Commented] (SPARK-26608) Remove Jenkins jobs for `branch-2.2`

2019-01-22 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749100#comment-16749100 ] shane knapp commented on SPARK-26608: - ping [~dongjoon] i'm assuming we'll want to

[jira] [Resolved] (SPARK-26685) Building Spark Images with latest Docker does not honour spark_uid build argument

2019-01-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26685. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23611 [https:

[jira] [Assigned] (SPARK-26685) Building Spark Images with latest Docker does not honour spark_uid build argument

2019-01-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26685: -- Assignee: Rob Vesse > Building Spark Images with latest Docker does not honour spark_

[jira] [Resolved] (SPARK-25887) Allow specifying Kubernetes context to use

2019-01-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25887. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22904 [https:

[jira] [Assigned] (SPARK-25887) Allow specifying Kubernetes context to use

2019-01-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25887: -- Assignee: Rob Vesse > Allow specifying Kubernetes context to use > --

[jira] [Comment Edited] (SPARK-26677) Incorrect results of not(eqNullSafe) when data read from Parquet file

2019-01-22 Thread ANAND CHINNAKANNAN (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748879#comment-16748879 ] ANAND CHINNAKANNAN edited comment on SPARK-26677 at 1/22/19 6:04 PM: -

[jira] [Comment Edited] (SPARK-26677) Incorrect results of not(eqNullSafe) when data read from Parquet file

2019-01-22 Thread ANAND CHINNAKANNAN (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748879#comment-16748879 ] ANAND CHINNAKANNAN edited comment on SPARK-26677 at 1/22/19 6:03 PM: -

[jira] [Updated] (SPARK-26668) One Kafka broker serve is down,the spark streaming start consuming delay

2019-01-22 Thread Chang Quanyou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Quanyou updated SPARK-26668: -- Attachment: batch_interval_6s.png batch_interval_6s_processing.png

[jira] [Commented] (SPARK-26668) One Kafka broker serve is down,the spark streaming start consuming delay

2019-01-22 Thread Chang Quanyou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748954#comment-16748954 ] Chang Quanyou commented on SPARK-26668: --- Thank you for replying, here are details:

[jira] [Resolved] (SPARK-26692) Structured Streaming: Aggregation + JOIN not working

2019-01-22 Thread Theo Diefenthal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Theo Diefenthal resolved SPARK-26692. - Resolution: Invalid Just read the crucial part of the doc again {code:java} https://spar

[jira] [Commented] (SPARK-24437) Memory leak in UnsafeHashedRelation

2019-01-22 Thread Dave DeCaprio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748948#comment-16748948 ] Dave DeCaprio commented on SPARK-24437: --- I'm actually running into a very similar

[jira] [Updated] (SPARK-26665) BlockTransferService.fetchBlockSync may hang forever

2019-01-22 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26665: - Fix Version/s: 2.3.4 > BlockTransferService.fetchBlockSync may hang forever > --

[jira] [Updated] (SPARK-26668) One Kafka broker serve is down,the spark streaming start consuming delay

2019-01-22 Thread Chang Quanyou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Quanyou updated SPARK-26668: -- Attachment: kafka_consumer.log executor.log 10.4.42.64_start.pn

[jira] [Updated] (SPARK-26665) BlockTransferService.fetchBlockSync may hang forever

2019-01-22 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26665: - Affects Version/s: 2.3.0 2.3.1 > BlockTransferService.fetchBlockSync may

[jira] [Updated] (SPARK-26665) BlockTransferService.fetchBlockSync may hang forever

2019-01-22 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26665: - Affects Version/s: 2.3.2 > BlockTransferService.fetchBlockSync may hang forever > --

[jira] [Commented] (SPARK-26688) Provide configuration of initially blacklisted YARN nodes

2019-01-22 Thread Attila Zsolt Piros (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748922#comment-16748922 ] Attila Zsolt Piros commented on SPARK-26688: As I know a node only could hav

[jira] [Created] (SPARK-26692) Structured Streaming: Aggregation + JOIN not working

2019-01-22 Thread Theo Diefenthal (JIRA)
Theo Diefenthal created SPARK-26692: --- Summary: Structured Streaming: Aggregation + JOIN not working Key: SPARK-26692 URL: https://issues.apache.org/jira/browse/SPARK-26692 Project: Spark Is

[jira] [Commented] (SPARK-26688) Provide configuration of initially blacklisted YARN nodes

2019-01-22 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748915#comment-16748915 ] Mridul Muralidharan commented on SPARK-26688: - What is the usecase for this

[jira] [Resolved] (SPARK-26665) BlockTransferService.fetchBlockSync may hang forever

2019-01-22 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-26665. -- Resolution: Fixed Fix Version/s: 3.0.0 2.4.1 > BlockTransferService.

[jira] [Commented] (SPARK-26649) Noop Streaming Sink using DSV2

2019-01-22 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748887#comment-16748887 ] Gabor Somogyi commented on SPARK-26649: --- Started to work on this. > Noop Streamin

[jira] [Updated] (SPARK-26691) WholeStageCodegen after InMemoryTableScan task takes significant time and time increases based on the input size

2019-01-22 Thread Vikash Kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikash Kumar updated SPARK-26691: - Summary: WholeStageCodegen after InMemoryTableScan task takes significant time and time increase

[jira] [Resolved] (SPARK-26657) Port DayWeek, DayOfWeek and WeekDay on Proleptic Gregorian calendar

2019-01-22 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-26657. --- Resolution: Fixed Assignee: Maxim Gekk Fix Version/s: 3.0.0 > Port D

[jira] [Comment Edited] (SPARK-26677) Incorrect results of not(eqNullSafe) when data read from Parquet file

2019-01-22 Thread ANAND CHINNAKANNAN (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748879#comment-16748879 ] ANAND CHINNAKANNAN edited comment on SPARK-26677 at 1/22/19 4:22 PM: -

[jira] [Commented] (SPARK-26677) Incorrect results of not(eqNullSafe) when data read from Parquet file

2019-01-22 Thread ANAND CHINNAKANNAN (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748879#comment-16748879 ] ANAND CHINNAKANNAN commented on SPARK-26677: I have done the analysis for th

[jira] [Assigned] (SPARK-16838) Add PMML export for ML KMeans in PySpark

2019-01-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-16838: - Assignee: Huaxin Gao > Add PMML export for ML KMeans in PySpark > -

[jira] [Updated] (SPARK-26691) WholeStageCodegen after InMemoryTableScan task takes more time and time increases based on the input size

2019-01-22 Thread Vikash Kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikash Kumar updated SPARK-26691: - Attachment: WholeStageCodegen.PNG > WholeStageCodegen after InMemoryTableScan task takes more ti

[jira] [Updated] (SPARK-26691) WholeStageCodegen after InMemoryTableScan task takes more time and time increases based on the input size

2019-01-22 Thread Vikash Kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikash Kumar updated SPARK-26691: - Attachment: DataScale_LkpPolicy_FirstRow_SF50_JobDetail.png DataScale_LkpPolicy_F

[jira] [Created] (SPARK-26691) WholeStageCodegen after InMemoryTableScan task takes more time and time increases based on the input size

2019-01-22 Thread Vikash Kumar (JIRA)
Vikash Kumar created SPARK-26691: Summary: WholeStageCodegen after InMemoryTableScan task takes more time and time increases based on the input size Key: SPARK-26691 URL: https://issues.apache.org/jira/browse/SPAR

[jira] [Resolved] (SPARK-16838) Add PMML export for ML KMeans in PySpark

2019-01-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-16838. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23592 [https://github.c

[jira] [Commented] (SPARK-26187) Stream-stream left outer join returns outer nulls for already matched rows

2019-01-22 Thread Pavel Chernikov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748808#comment-16748808 ] Pavel Chernikov commented on SPARK-26187: - [~sandeep.katta2007], initially I use

[jira] [Assigned] (SPARK-26680) StackOverflowError if Stream passed to groupBy

2019-01-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26680: Assignee: Apache Spark > StackOverflowError if Stream passed to groupBy > ---

[jira] [Assigned] (SPARK-26680) StackOverflowError if Stream passed to groupBy

2019-01-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26680: Assignee: (was: Apache Spark) > StackOverflowError if Stream passed to groupBy >

[jira] [Commented] (SPARK-26680) StackOverflowError if Stream passed to groupBy

2019-01-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748811#comment-16748811 ] Apache Spark commented on SPARK-26680: -- User 'bersprockets' has created a pull requ

[jira] [Commented] (SPARK-26668) One Kafka broker serve is down,the spark streaming start consuming delay

2019-01-22 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748809#comment-16748809 ] Gabor Somogyi commented on SPARK-26668: --- [~quanyou.chang] Is it DStreams or Strctu

[jira] [Comment Edited] (SPARK-26668) One Kafka broker serve is down,the spark streaming start consuming delay

2019-01-22 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748809#comment-16748809 ] Gabor Somogyi edited comment on SPARK-26668 at 1/22/19 3:03 PM: --

[jira] [Updated] (SPARK-26649) Noop Streaming Sink using DSV2

2019-01-22 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-26649: -- Issue Type: New Feature (was: Bug) > Noop Streaming Sink using DSV2 > ---

[jira] [Updated] (SPARK-24938) Understand usage of netty's onheap memory use, even with offheap pools

2019-01-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-24938: -- Priority: Minor (was: Major) > Understand usage of netty's onheap memory use, even with offheap pools

[jira] [Resolved] (SPARK-24938) Understand usage of netty's onheap memory use, even with offheap pools

2019-01-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-24938. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22114 [https://github.c

[jira] [Assigned] (SPARK-24938) Understand usage of netty's onheap memory use, even with offheap pools

2019-01-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-24938: - Assignee: Nihar Sheth > Understand usage of netty's onheap memory use, even with offheap pools

[jira] [Assigned] (SPARK-26688) Provide configuration of initially blacklisted YARN nodes

2019-01-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26688: Assignee: Apache Spark > Provide configuration of initially blacklisted YARN nodes >

[jira] [Assigned] (SPARK-26688) Provide configuration of initially blacklisted YARN nodes

2019-01-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26688: Assignee: (was: Apache Spark) > Provide configuration of initially blacklisted YARN n

[jira] [Updated] (SPARK-26680) StackOverflowError if Stream passed to groupBy

2019-01-22 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-26680: -- Affects Version/s: 2.3.2 > StackOverflowError if Stream passed to groupBy > --

[jira] [Updated] (SPARK-26680) StackOverflowError if Stream passed to groupBy

2019-01-22 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-26680: -- Affects Version/s: 2.4.0 > StackOverflowError if Stream passed to groupBy > --

[jira] [Resolved] (SPARK-26463) Use ConfigEntry for hardcoded configs for scheduler categories.

2019-01-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26463. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23416 [https://github.c

[jira] [Created] (SPARK-26690) Checkpoints of Dataframes are not visible in the SQL UI

2019-01-22 Thread Tom van Bussel (JIRA)
Tom van Bussel created SPARK-26690: -- Summary: Checkpoints of Dataframes are not visible in the SQL UI Key: SPARK-26690 URL: https://issues.apache.org/jira/browse/SPARK-26690 Project: Spark I

[jira] [Assigned] (SPARK-26463) Use ConfigEntry for hardcoded configs for scheduler categories.

2019-01-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26463: - Assignee: Kazuaki Ishizaki > Use ConfigEntry for hardcoded configs for scheduler categories. >

[jira] [Resolved] (SPARK-26616) Expose document frequency in IDFModel

2019-01-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26616. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23549 [https://github.c

  1   2   >