[jira] [Created] (SPARK-21555) GROUP BY don't work with expressions with NVL and nested objects

2017-07-28 Thread Vitaly Gerasimov (JIRA)
Vitaly Gerasimov created SPARK-21555: Summary: GROUP BY don't work with expressions with NVL and nested objects Key: SPARK-21555 URL: https://issues.apache.org/jira/browse/SPARK-21555 Project: Spa

[jira] [Commented] (SPARK-21543) Should not count executor initialize failed towards task failures

2017-07-28 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16104549#comment-16104549 ] zhoukang commented on SPARK-21543: -- First i will describe my case: There is a cluster wi

[jira] [Closed] (SPARK-21543) Should not count executor initialize failed towards task failures

2017-07-28 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang closed SPARK-21543. Resolution: Invalid > Should not count executor initialize failed towards task failures > -

[jira] [Created] (SPARK-21556) PySpark, Unable to save pipeline of non-spark transformers

2017-07-28 Thread Saif Addin (JIRA)
Saif Addin created SPARK-21556: -- Summary: PySpark, Unable to save pipeline of non-spark transformers Key: SPARK-21556 URL: https://issues.apache.org/jira/browse/SPARK-21556 Project: Spark Issue

[jira] [Commented] (SPARK-21548) Support insert into serial columns of table

2017-07-28 Thread LvDongrong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16104558#comment-16104558 ] LvDongrong commented on SPARK-21548: Think you very much, I've made a pr, and you can

[jira] [Updated] (SPARK-21555) GROUP BY don't work with expressions with NVL and nested objects

2017-07-28 Thread Vitaly Gerasimov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitaly Gerasimov updated SPARK-21555: - Description: {code} spark.read.json(spark.createDataset("""{"foo":{"foo1":"value"}}""" ::

[jira] [Updated] (SPARK-21555) GROUP BY don't work with expressions with NVL and nested objects

2017-07-28 Thread Vitaly Gerasimov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitaly Gerasimov updated SPARK-21555: - Description: {code} spark.read.json(spark.createDataset("""{"foo":{"foo1":"value"}}""" ::

[jira] [Updated] (SPARK-21555) GROUP BY don't work with expressions with NVL and nested objects

2017-07-28 Thread Vitaly Gerasimov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitaly Gerasimov updated SPARK-21555: - Description: {code} spark.read.json(spark.createDataset("""{"foo":{"foo1":"value"}}""" ::

[jira] [Created] (SPARK-21557) Debug issues for SparkML(scala.Predef$.any2ArrowAssoc)

2017-07-28 Thread prabir bhowmick (JIRA)
prabir bhowmick created SPARK-21557: --- Summary: Debug issues for SparkML(scala.Predef$.any2ArrowAssoc) Key: SPARK-21557 URL: https://issues.apache.org/jira/browse/SPARK-21557 Project: Spark

[jira] [Commented] (SPARK-21554) Spark Hive reporting pyspark.sql.utils.AnalysisException: u'Table not found: XXX' when run on yarn cluster

2017-07-28 Thread Subhod Lagade (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16104634#comment-16104634 ] Subhod Lagade commented on SPARK-21554: --- Thanks for quick reply @Hyukjin Kwon - We

[jira] [Resolved] (SPARK-21557) Debug issues for SparkML(scala.Predef$.any2ArrowAssoc)

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21557. --- Resolution: Invalid Fix Version/s: (was: 2.1.2) Target Version/s: (was: 2.2.0)

[jira] [Commented] (SPARK-21554) Spark Hive reporting pyspark.sql.utils.AnalysisException: u'Table not found: XXX' when run on yarn cluster

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16104640#comment-16104640 ] Sean Owen commented on SPARK-21554: --- The error here doesn't show the actual error. >

[jira] [Created] (SPARK-21558) Kinesis lease failover time should be increased or made configurable

2017-07-28 Thread JIRA
Clément MATHIEU created SPARK-21558: --- Summary: Kinesis lease failover time should be increased or made configurable Key: SPARK-21558 URL: https://issues.apache.org/jira/browse/SPARK-21558 Project: S

[jira] [Commented] (SPARK-21479) Outer join filter pushdown in null supplying table when condition is on one of the joined columns

2017-07-28 Thread Abhijit Bhole (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16104708#comment-16104708 ] Abhijit Bhole commented on SPARK-21479: --- So here is the actual use case - {code:j

[jira] [Updated] (SPARK-21553) Add the description of the default value of master parameter in the spark-shell

2017-07-28 Thread Donghui Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Donghui Xu updated SPARK-21553: --- Summary: Add the description of the default value of master parameter in the spark-shell (was: Added

[jira] [Created] (SPARK-21559) Remove Mesos Fine-grain mode

2017-07-28 Thread Stavros Kontopoulos (JIRA)
Stavros Kontopoulos created SPARK-21559: --- Summary: Remove Mesos Fine-grain mode Key: SPARK-21559 URL: https://issues.apache.org/jira/browse/SPARK-21559 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-21559) Remove Mesos fine-grained mode

2017-07-28 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-21559: Description: After discussing this with people from Mesosphere we agreed that it is

[jira] [Updated] (SPARK-21559) Remove Mesos fine-grained mode

2017-07-28 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-21559: Summary: Remove Mesos fine-grained mode (was: Remove Mesos Fine-grain mode) > Remo

[jira] [Updated] (SPARK-20919) Simplificaiton of CachedKafkaConsumer.

2017-07-28 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-20919: Summary: Simplificaiton of CachedKafkaConsumer. (was: Simplificaiton of CachedKafkaConsume

[jira] [Updated] (SPARK-20919) Simplificaiton of CachedKafkaConsumer.

2017-07-28 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-20919: Description: Using an object pool instead of a cache for recycling objects in Kafka consume

[jira] [Updated] (SPARK-21306) OneVsRest Conceals Columns That May Be Relevant To Underlying Classifier

2017-07-28 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-21306: Fix Version/s: (was: 2.1.2) (was: 2.0.3) > OneVsRest Conceals Columns Th

[jira] [Commented] (SPARK-21067) Thrift Server - CTAS fail with Unable to move source

2017-07-28 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16104918#comment-16104918 ] Wenchen Fan commented on SPARK-21067: - We have many tests for CREATE TABLE inside Spa

[jira] [Created] (SPARK-21560) Add hold mode for the LiveListenerBus

2017-07-28 Thread Li Yuanjian (JIRA)
Li Yuanjian created SPARK-21560: --- Summary: Add hold mode for the LiveListenerBus Key: SPARK-21560 URL: https://issues.apache.org/jira/browse/SPARK-21560 Project: Spark Issue Type: Improvement

[jira] [Issue Comment Deleted] (SPARK-14927) DataFrame. saveAsTable creates RDD partitions but not Hive partitions

2017-07-28 Thread SOMASUNDARAM SUDALAIMUTHU (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SOMASUNDARAM SUDALAIMUTHU updated SPARK-14927: -- Comment: was deleted (was: Is this fixed in 2.0 version ?) > DataFrame

[jira] [Commented] (SPARK-14927) DataFrame. saveAsTable creates RDD partitions but not Hive partitions

2017-07-28 Thread SOMASUNDARAM SUDALAIMUTHU (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105004#comment-16105004 ] SOMASUNDARAM SUDALAIMUTHU commented on SPARK-14927: --- Is this fixed in 2

[jira] [Resolved] (SPARK-21553) Add the description of the default value of master parameter in the spark-shell

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21553. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18755 [https://github.co

[jira] [Updated] (SPARK-21553) Add the description of the default value of master parameter in the spark-shell

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21553: -- Priority: Trivial (was: Minor) > Add the description of the default value of master parameter in the

[jira] [Assigned] (SPARK-21553) Add the description of the default value of master parameter in the spark-shell

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21553: - Assignee: Donghui Xu > Add the description of the default value of master parameter in the > sp

[jira] [Resolved] (SPARK-21541) Spark Logs show incorrect job status for a job that does not create SparkContext

2017-07-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21541. --- Resolution: Fixed Assignee: Parth Gandhi Fix Version/s: 2.3.0 > Spark Logs sh

[jira] [Commented] (SPARK-21541) Spark Logs show incorrect job status for a job that does not create SparkContext

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105049#comment-16105049 ] Sean Owen commented on SPARK-21541: --- Was this change merged? I don't think it was http

[jira] [Comment Edited] (SPARK-21190) SPIP: Vectorized UDFs in Python

2017-07-28 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105072#comment-16105072 ] Li Jin edited comment on SPARK-21190 at 7/28/17 3:06 PM: - [~cloud

[jira] [Commented] (SPARK-21190) SPIP: Vectorized UDFs in Python

2017-07-28 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105072#comment-16105072 ] Li Jin commented on SPARK-21190: [~cloud_fan], thanks for pointing out `ArrowColumnVector

[jira] [Commented] (SPARK-21555) GROUP BY don't work with expressions with NVL and nested objects

2017-07-28 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105092#comment-16105092 ] Liang-Chi Hsieh commented on SPARK-21555: - The sync between PR and JIRA seems bro

[jira] [Commented] (SPARK-21190) SPIP: Vectorized UDFs in Python

2017-07-28 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105095#comment-16105095 ] Li Jin commented on SPARK-21190: I think the use case 2 of what [~rxin] proposed original

[jira] [Updated] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

2017-07-28 Thread Sergey Zhemzhitsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Zhemzhitsky updated SPARK-21549: --- Priority: Blocker (was: Critical) > Spark fails to complete job correctly in case of

[jira] [Updated] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21549: -- Priority: Major (was: Blocker) > Spark fails to complete job correctly in case of OutputFormat which d

[jira] [Commented] (SPARK-21541) Spark Logs show incorrect job status for a job that does not create SparkContext

2017-07-28 Thread Parth Gandhi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105116#comment-16105116 ] Parth Gandhi commented on SPARK-21541: -- The change has been merged. Thank you. > Sp

[jira] [Commented] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2017-07-28 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105127#comment-16105127 ] Ruslan Dautkhanov commented on SPARK-21274: --- [~viirya], yes it returns {noforma

[jira] [Commented] (SPARK-21541) Spark Logs show incorrect job status for a job that does not create SparkContext

2017-07-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105133#comment-16105133 ] Thomas Graves commented on SPARK-21541: --- it was merged https://github.com/apache/s

[jira] [Comment Edited] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2017-07-28 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105127#comment-16105127 ] Ruslan Dautkhanov edited comment on SPARK-21274 at 7/28/17 3:46 PM: ---

[jira] [Comment Edited] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2017-07-28 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105127#comment-16105127 ] Ruslan Dautkhanov edited comment on SPARK-21274 at 7/28/17 3:47 PM: ---

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2017-07-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105164#comment-16105164 ] Thomas Graves commented on SPARK-17321: --- Can you clarify? as stated above you sho

[jira] [Created] (SPARK-21561) spark-streaming-kafka-010 DSteam is not pulling anything from Kafka

2017-07-28 Thread Vlad Badelita (JIRA)
Vlad Badelita created SPARK-21561: - Summary: spark-streaming-kafka-010 DSteam is not pulling anything from Kafka Key: SPARK-21561 URL: https://issues.apache.org/jira/browse/SPARK-21561 Project: Spark

[jira] [Created] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
Wei Chen created SPARK-21562: Summary: Spark may request extra containers if the rpc between YARN and spark is too fast Key: SPARK-21562 URL: https://issues.apache.org/jira/browse/SPARK-21562 Project: Spa

[jira] [Updated] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Chen updated SPARK-21562: - Description: hi huys, I find an interesting problem when spark tries to request containers from YARN. H

[jira] [Updated] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Chen updated SPARK-21562: - Description: hi huys, I find an interesting problem when spark tries to request containers from YARN. H

[jira] [Updated] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Chen updated SPARK-21562: - Description: hi huys, I find an interesting problem when spark tries to request containers from YARN. H

[jira] [Updated] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Chen updated SPARK-21562: - Description: hi huys, I find an interesting problem when spark tries to request containers from YARN. H

[jira] [Updated] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Chen updated SPARK-21562: - Description: hi huys, I find an interesting problem when spark tries to request containers from YARN. H

[jira] [Updated] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Chen updated SPARK-21562: - Description: hi huys, I find an interesting problem when spark tries to request containers from YARN. H

[jira] [Updated] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Chen updated SPARK-21562: - Description: hi huys, I find an interesting problem when spark tries to request containers from YARN. H

[jira] [Updated] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Chen updated SPARK-21562: - Description: hi huys, I find an interesting problem when spark tries to request containers from YARN. H

[jira] [Updated] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Chen updated SPARK-21562: - Description: hi huys, I find an interesting problem when spark tries to request containers from YARN. H

[jira] [Resolved] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21562. Resolution: Duplicate > Spark may request extra containers if the rpc between YARN and spar

[jira] [Comment Edited] (SPARK-17614) sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra does not support

2017-07-28 Thread Paul Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105402#comment-16105402 ] Paul Wu edited comment on SPARK-17614 at 7/28/17 6:08 PM: -- The f

[jira] [Reopened] (SPARK-17614) sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra does not support

2017-07-28 Thread Paul Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Wu reopened SPARK-17614: - The fix does not support the syntax on the syntax like this: {{.jdbc(JDBC_URL, "(select * from emp)", connec

[jira] [Comment Edited] (SPARK-17614) sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra does not support

2017-07-28 Thread Paul Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105402#comment-16105402 ] Paul Wu edited comment on SPARK-17614 at 7/28/17 6:09 PM: -- The f

[jira] [Resolved] (SPARK-17614) sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra does not support

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17614. --- Resolution: Fixed [~zwu@gmail.com] please don't reopen this > sparkSession.read() .jdbc(***) use

[jira] [Closed] (SPARK-17614) sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra does not support

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen closed SPARK-17614. - > sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra > does not support >

[jira] [Resolved] (SPARK-21561) spark-streaming-kafka-010 DSteam is not pulling anything from Kafka

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21561. --- Resolution: Invalid This isn't a place to ask for input on your code -- you'd have to show a reprodu

[jira] [Comment Edited] (SPARK-17614) sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra does not support

2017-07-28 Thread Paul Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105402#comment-16105402 ] Paul Wu edited comment on SPARK-17614 at 7/28/17 6:14 PM: -- The f

[jira] [Commented] (SPARK-17614) sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra does not support

2017-07-28 Thread Paul Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105425#comment-16105425 ] Paul Wu commented on SPARK-17614: - So create a new issue? Or this is not an issue to you?

[jira] [Updated] (SPARK-21523) Fix bug of strong wolfe linesearch `init` parameter lose effectiveness

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21523: -- Target Version/s: 2.2.1 Priority: Critical (was: Minor) I think this is fairly critical ac

[jira] [Commented] (SPARK-21546) dropDuplicates with watermark yields RuntimeException due to binding failure

2017-07-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105426#comment-16105426 ] Shixiong Zhu commented on SPARK-21546: -- Yeah, good catch. The watermark column shoul

[jira] [Commented] (SPARK-19720) Redact sensitive information from SparkSubmit console output

2017-07-28 Thread Mark Grover (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105427#comment-16105427 ] Mark Grover commented on SPARK-19720: - I wasn't planning on. One could argue the case

[jira] [Commented] (SPARK-17614) sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra does not support

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105431#comment-16105431 ] Sean Owen commented on SPARK-17614: --- Well, it's unrelated to this issue, so this isn't

[jira] [Commented] (SPARK-17614) sparkSession.read() .jdbc(***) use the sql syntax "where 1=0" that Cassandra does not support

2017-07-28 Thread Paul Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105439#comment-16105439 ] Paul Wu commented on SPARK-17614: - Oh, sorry. I thought I could use a query hereas I

[jira] [Commented] (SPARK-19938) java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field

2017-07-28 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105449#comment-16105449 ] Ryan Blue commented on SPARK-19938: --- [~snavatski], I just hit this problem also and fou

[jira] [Commented] (SPARK-21562) Spark may request extra containers if the rpc between YARN and spark is too fast

2017-07-28 Thread Wei Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105505#comment-16105505 ] Wei Chen commented on SPARK-21562: -- Thanks for comments, I will close this one. On Fri,

[jira] [Commented] (SPARK-19720) Redact sensitive information from SparkSubmit console output

2017-07-28 Thread Diogo Munaro Vieira (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105514#comment-16105514 ] Diogo Munaro Vieira commented on SPARK-19720: - Yes, but it's a major security

[jira] [Commented] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2017-07-28 Thread James Conner (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105528#comment-16105528 ] James Conner commented on SPARK-18016: -- Thank you for letting me know, Kazuaki! Ple

[jira] [Created] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

2017-07-28 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21563: -- Summary: Race condition when serializing TaskDescriptions and adding jars Key: SPARK-21563 URL: https://issues.apache.org/jira/browse/SPARK-21563 Project: Spark

[jira] [Created] (SPARK-21564) TaskDescription decoding failure should fail the task

2017-07-28 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-21564: -- Summary: TaskDescription decoding failure should fail the task Key: SPARK-21564 URL: https://issues.apache.org/jira/browse/SPARK-21564 Project: Spark Issue Type:

[jira] [Updated] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-21563: --- Description: cc [~robert3005] I was seeing this exception during some running Spark jobs: {noformat

[jira] [Updated] (SPARK-21564) TaskDescription decoding failure should fail the task

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-21564: --- Description: cc [~robert3005] I was seeing an issue where Spark was throwing this exception: {nofor

[jira] [Created] (SPARK-21565) aggregate query fails with watermark on eventTime but works with watermark on timestamp column generated by current_timestamp

2017-07-28 Thread Amit Assudani (JIRA)
Amit Assudani created SPARK-21565: - Summary: aggregate query fails with watermark on eventTime but works with watermark on timestamp column generated by current_timestamp Key: SPARK-21565 URL: https://issues.apach

[jira] [Updated] (SPARK-21565) aggregate query fails with watermark on eventTime but works with watermark on timestamp column generated by current_timestamp

2017-07-28 Thread Amit Assudani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Assudani updated SPARK-21565: -- Description: *Short Description: * Aggregation query fails with eventTime as watermark column

[jira] [Commented] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

2017-07-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105630#comment-16105630 ] Mridul Muralidharan commented on SPARK-21549: - This affects both mapred ("ma

[jira] [Created] (SPARK-21566) Python method for summary

2017-07-28 Thread Andrew Ray (JIRA)
Andrew Ray created SPARK-21566: -- Summary: Python method for summary Key: SPARK-21566 URL: https://issues.apache.org/jira/browse/SPARK-21566 Project: Spark Issue Type: Improvement Compo

[jira] [Created] (SPARK-21567) Dataset with Tuple of type alias throws error

2017-07-28 Thread Tomasz Bartczak (JIRA)
Tomasz Bartczak created SPARK-21567: --- Summary: Dataset with Tuple of type alias throws error Key: SPARK-21567 URL: https://issues.apache.org/jira/browse/SPARK-21567 Project: Spark Issue Typ

[jira] [Commented] (SPARK-19490) Hive partition columns are case-sensitive

2017-07-28 Thread Taklon Stephen Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105734#comment-16105734 ] Taklon Stephen Wu commented on SPARK-19490: --- https://github.com/apache/spark/pu

[jira] [Created] (SPARK-21568) ConsoleProgressBar should only be enabled in shells

2017-07-28 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-21568: -- Summary: ConsoleProgressBar should only be enabled in shells Key: SPARK-21568 URL: https://issues.apache.org/jira/browse/SPARK-21568 Project: Spark Issue

[jira] [Commented] (SPARK-20433) Security issue with jackson-databind

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105751#comment-16105751 ] Andrew Ash commented on SPARK-20433: Here's the patch I put in my fork of Spark: htt

[jira] [Commented] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105772#comment-16105772 ] Andrew Ash commented on SPARK-21563: And for reference, I added this additional loggi

[jira] [Comment Edited] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

2017-07-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105630#comment-16105630 ] Mridul Muralidharan edited comment on SPARK-21549 at 7/28/17 10:14 PM:

[jira] [Comment Edited] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

2017-07-28 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105630#comment-16105630 ] Mridul Muralidharan edited comment on SPARK-21549 at 7/28/17 10:16 PM:

[jira] [Commented] (SPARK-20433) Security issue with jackson-databind

2017-07-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105792#comment-16105792 ] Sean Owen commented on SPARK-20433: --- You updated to 2.6.7 but indicated above that's st

[jira] [Created] (SPARK-21569) Internal Spark class needs to be kryo-registered

2017-07-28 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-21569: - Summary: Internal Spark class needs to be kryo-registered Key: SPARK-21569 URL: https://issues.apache.org/jira/browse/SPARK-21569 Project: Spark Issue Type

[jira] [Created] (SPARK-21570) File __spark_libs__XXX.zip does not exist on networked file system w/ yarn

2017-07-28 Thread Albert Chu (JIRA)
Albert Chu created SPARK-21570: -- Summary: File __spark_libs__XXX.zip does not exist on networked file system w/ yarn Key: SPARK-21570 URL: https://issues.apache.org/jira/browse/SPARK-21570 Project: Spark

[jira] [Commented] (SPARK-21570) File __spark_libs__XXX.zip does not exist on networked file system w/ yarn

2017-07-28 Thread Albert Chu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105900#comment-16105900 ] Albert Chu commented on SPARK-21570: Oh, and because it will likely be asked and may

[jira] [Commented] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2017-07-28 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105919#comment-16105919 ] Liang-Chi Hsieh commented on SPARK-21274: - [~Tagar] I've tried the query on Postg

[jira] [Commented] (SPARK-20433) Security issue with jackson-databind

2017-07-28 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105951#comment-16105951 ] Andrew Ash commented on SPARK-20433: As I wrote in that PR, it's 2.6.7.1 of jackson-d

[jira] [Commented] (SPARK-19720) Redact sensitive information from SparkSubmit console output

2017-07-28 Thread Diogo Munaro Vieira (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105952#comment-16105952 ] Diogo Munaro Vieira commented on SPARK-19720: - I did a merge request for this

[jira] [Commented] (SPARK-18535) Redact sensitive information from Spark logs and UI

2017-07-28 Thread Diogo Munaro Vieira (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105955#comment-16105955 ] Diogo Munaro Vieira commented on SPARK-18535: - I did a merge request for vers

[jira] [Created] (SPARK-21571) Spark history server leaves incomplete or unreadable history files around forever.

2017-07-28 Thread Eric Vandenberg (JIRA)
Eric Vandenberg created SPARK-21571: --- Summary: Spark history server leaves incomplete or unreadable history files around forever. Key: SPARK-21571 URL: https://issues.apache.org/jira/browse/SPARK-21571

[jira] [Commented] (SPARK-19490) Hive partition columns are case-sensitive

2017-07-28 Thread cen yuhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105979#comment-16105979 ] cen yuhai commented on SPARK-19490: --- I forgot the `fixed PR`, maybe that pr will not by

[jira] [Created] (SPARK-21572) Add description on how to exit the spark-shell in the welcome message

2017-07-28 Thread Donghui Xu (JIRA)
Donghui Xu created SPARK-21572: -- Summary: Add description on how to exit the spark-shell in the welcome message Key: SPARK-21572 URL: https://issues.apache.org/jira/browse/SPARK-21572 Project: Spark

[jira] [Commented] (SPARK-21533) "configure(...)" method not called when using Hive Generic UDFs

2017-07-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16106000#comment-16106000 ] Takeshi Yamamuro commented on SPARK-21533: -- I feel supporting this function make

[jira] [Resolved] (SPARK-20090) Add StructType.fieldNames to Python API

2017-07-28 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-20090. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18618 [https://github.com/ap

[jira] [Created] (SPARK-21573) run-tests script fails with Python 2.6

2017-07-28 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-21573: Summary: run-tests script fails with Python 2.6 Key: SPARK-21573 URL: https://issues.apache.org/jira/browse/SPARK-21573 Project: Spark Issue Type: Test

  1   2   >