[jira] [Commented] (SPARK-22192) An RDD of nested POJO objects cannot be converted into a DataFrame using SQLContext.createDataFrame API

2017-10-03 Thread Asif Hussain Shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190852#comment-16190852 ] Asif Hussain Shahid commented on SPARK-22192: - Not at all. I have added a bug test in

[jira] [Commented] (SPARK-22192) An RDD of nested POJO objects cannot be converted into a DataFrame using SQLContext.createDataFrame API

2017-10-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190838#comment-16190838 ] Hyukjin Kwon commented on SPARK-22192: -- Would you mind if I ask a reproducer? > An RDD of nested

[jira] [Resolved] (SPARK-22136) Implement stream-stream outer joins in append mode

2017-10-03 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-22136. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 19327

[jira] [Updated] (SPARK-21951) Unable to add the new column and writing into the Hive using spark

2017-10-03 Thread jalendhar Baddam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jalendhar Baddam updated SPARK-21951: - Issue Type: Question (was: Bug) > Unable to add the new column and writing into the

[jira] [Updated] (SPARK-21952) Unable to load the csv file into Dataset using Spark with java

2017-10-03 Thread jalendhar Baddam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jalendhar Baddam updated SPARK-21952: - Issue Type: Question (was: Bug) > Unable to load the csv file into Dataset using Spark

[jira] [Resolved] (SPARK-22171) Describe Table Extended Failed when Table Owner is Empty

2017-10-03 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22171. - Resolution: Fixed Fix Version/s: 2.3.0 > Describe Table Extended Failed when Table Owner is Empty

[jira] [Commented] (SPARK-20557) JdbcUtils doesn't support java.sql.Types.TIMESTAMP_WITH_TIMEZONE

2017-10-03 Thread Dan Stine (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190706#comment-16190706 ] Dan Stine commented on SPARK-20557: --- [~JannikArndt] [~smilegator] Can you help me understand the status

[jira] [Created] (SPARK-22195) Add cosine similarity to org.apache.spark.ml.linalg.Vectors

2017-10-03 Thread yuhao yang (JIRA)
yuhao yang created SPARK-22195: -- Summary: Add cosine similarity to org.apache.spark.ml.linalg.Vectors Key: SPARK-22195 URL: https://issues.apache.org/jira/browse/SPARK-22195 Project: Spark

[jira] [Commented] (SPARK-22193) SortMergeJoinExec: typo correction

2017-10-03 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190580#comment-16190580 ] Takeshi Yamamuro commented on SPARK-22193: -- You probably don't file a jira for trivial fixes. >

[jira] [Closed] (SPARK-19426) Add support for custom coalescers on Data

2017-10-03 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro closed SPARK-19426. Resolution: Later > Add support for custom coalescers on Data >

[jira] [Commented] (SPARK-19426) Add support for custom coalescers on Data

2017-10-03 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190578#comment-16190578 ] Takeshi Yamamuro commented on SPARK-19426: -- I'll close for now cuz the priority is not much

[jira] [Resolved] (SPARK-20466) HadoopRDD#addLocalConfiguration throws NPE

2017-10-03 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-20466. Resolution: Fixed Assignee: Sahil Takiar Fix Version/s: 2.1.3

[jira] [Created] (SPARK-22194) Allow namespacing of configs in spark.internal.config

2017-10-03 Thread Gregory Owen (JIRA)
Gregory Owen created SPARK-22194: Summary: Allow namespacing of configs in spark.internal.config Key: SPARK-22194 URL: https://issues.apache.org/jira/browse/SPARK-22194 Project: Spark Issue

[jira] [Assigned] (SPARK-22193) SortMergeJoinExec: typo correction

2017-10-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22193: Assignee: Apache Spark > SortMergeJoinExec: typo correction >

[jira] [Commented] (SPARK-22193) SortMergeJoinExec: typo correction

2017-10-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190365#comment-16190365 ] Apache Spark commented on SPARK-22193: -- User 'rekhajoshm' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22193) SortMergeJoinExec: typo correction

2017-10-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22193: Assignee: (was: Apache Spark) > SortMergeJoinExec: typo correction >

[jira] [Created] (SPARK-22193) SortMergeJoinExec: typo correction

2017-10-03 Thread Rekha Joshi (JIRA)
Rekha Joshi created SPARK-22193: --- Summary: SortMergeJoinExec: typo correction Key: SPARK-22193 URL: https://issues.apache.org/jira/browse/SPARK-22193 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-21866) SPIP: Image support in Spark

2017-10-03 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190239#comment-16190239 ] yuhao yang commented on SPARK-21866: My two cents, 1. In most scenarios, deep learning applications

[jira] [Resolved] (SPARK-21644) LocalLimit.maxRows is defined incorrectly

2017-10-03 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21644. - Resolution: Fixed Fix Version/s: 2.3.0 > LocalLimit.maxRows is defined incorrectly >

[jira] [Created] (SPARK-22192) An RDD of nested POJO objects cannot be converted into a DataFrame using SQLContext.createDataFrame API

2017-10-03 Thread Asif Hussain Shahid (JIRA)
Asif Hussain Shahid created SPARK-22192: --- Summary: An RDD of nested POJO objects cannot be converted into a DataFrame using SQLContext.createDataFrame API Key: SPARK-22192 URL:

[jira] [Resolved] (SPARK-22158) convertMetastore should not ignore table properties

2017-10-03 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22158. - Resolution: Fixed Assignee: Dongjoon Hyun Fix Version/s: 2.3.0 2.2.1

[jira] [Commented] (SPARK-19984) ERROR codegen.CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java'

2017-10-03 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190050#comment-16190050 ] Kazuaki Ishizaki commented on SPARK-19984: -- [~JohnSteidley] Thank you for providing valuable

[jira] [Commented] (SPARK-22191) Add hive serde example with serde properties

2017-10-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190047#comment-16190047 ] Apache Spark commented on SPARK-22191: -- User 'crlalam' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22191) Add hive serde example with serde properties

2017-10-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22191: Assignee: Apache Spark > Add hive serde example with serde properties >

[jira] [Assigned] (SPARK-22191) Add hive serde example with serde properties

2017-10-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22191: Assignee: (was: Apache Spark) > Add hive serde example with serde properties >

[jira] [Created] (SPARK-22191) Add hive serde example with serde properties

2017-10-03 Thread Chinna Rao Lalam (JIRA)
Chinna Rao Lalam created SPARK-22191: Summary: Add hive serde example with serde properties Key: SPARK-22191 URL: https://issues.apache.org/jira/browse/SPARK-22191 Project: Spark Issue

[jira] [Assigned] (SPARK-22184) GraphX fails in case of insufficient memory and checkpoints enabled

2017-10-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22184: Assignee: (was: Apache Spark) > GraphX fails in case of insufficient memory and

[jira] [Assigned] (SPARK-22184) GraphX fails in case of insufficient memory and checkpoints enabled

2017-10-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22184: Assignee: Apache Spark > GraphX fails in case of insufficient memory and checkpoints

[jira] [Commented] (SPARK-22167) Spark Packaging w/R distro issues

2017-10-03 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189954#comment-16189954 ] Felix Cheung commented on SPARK-22167: -- There are likely 2 stages to this. More pressing might be

[jira] [Comment Edited] (SPARK-14172) Hive table partition predicate not passed down correctly

2017-10-03 Thread Saktheesh Balaraj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189481#comment-16189481 ] Saktheesh Balaraj edited comment on SPARK-14172 at 10/3/17 3:21 PM:

[jira] [Updated] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

2017-10-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21549: -- Fix Version/s: (was: 2.2.1) > Spark fails to complete job correctly in case of OutputFormat which

[jira] [Resolved] (SPARK-22189) Number of jobs created while querying partitioned table in hive using spark

2017-10-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22189. --- Resolution: Invalid Questions should go to the mailing list, please. > Number of jobs created while

[jira] [Created] (SPARK-22190) Add Spark executor task metrics to Dropwizard metrics

2017-10-03 Thread Luca Canali (JIRA)
Luca Canali created SPARK-22190: --- Summary: Add Spark executor task metrics to Dropwizard metrics Key: SPARK-22190 URL: https://issues.apache.org/jira/browse/SPARK-22190 Project: Spark Issue

[jira] [Updated] (SPARK-22190) Add Spark executor task metrics to Dropwizard metrics

2017-10-03 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-22190: Attachment: SparkTaskMetrics_Grafana_example.PNG !SparkTaskMetrics_Grafana_example.PNG|thumbnail!

[jira] [Created] (SPARK-22189) Number of jobs created while querying partitioned table in hive using spark

2017-10-03 Thread Astha Arya (JIRA)
Astha Arya created SPARK-22189: -- Summary: Number of jobs created while querying partitioned table in hive using spark Key: SPARK-22189 URL: https://issues.apache.org/jira/browse/SPARK-22189 Project:

[jira] [Commented] (SPARK-22188) Add defense against Cross-Site Scripting, MIME-sniffing and MitM attack

2017-10-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189700#comment-16189700 ] Apache Spark commented on SPARK-22188: -- User 'krishna-pandey' has created a pull request for this

[jira] [Assigned] (SPARK-22188) Add defense against Cross-Site Scripting, MIME-sniffing and MitM attack

2017-10-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22188: Assignee: Apache Spark > Add defense against Cross-Site Scripting, MIME-sniffing and MitM

[jira] [Assigned] (SPARK-22188) Add defense against Cross-Site Scripting, MIME-sniffing and MitM attack

2017-10-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22188: Assignee: (was: Apache Spark) > Add defense against Cross-Site Scripting,

[jira] [Updated] (SPARK-22188) Add defense against Cross-Site Scripting, MIME-sniffing and MitM attack

2017-10-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-22188: -- Shepherd: (was: Sean Owen) Flags: (was: Important) Priority: Minor (was:

[jira] [Created] (SPARK-22188) Add defense against Cross-Site Scripting, MIME-sniffing and MitM attack

2017-10-03 Thread Krishna Pandey (JIRA)
Krishna Pandey created SPARK-22188: -- Summary: Add defense against Cross-Site Scripting, MIME-sniffing and MitM attack Key: SPARK-22188 URL: https://issues.apache.org/jira/browse/SPARK-22188 Project:

[jira] [Closed] (SPARK-16709) Task with commit failed will retry infinite when speculation set to true

2017-10-03 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artur Sukhenko closed SPARK-16709. -- Resolution: Duplicate Fix Version/s: 1.6.2 > Task with commit failed will retry

[jira] [Comment Edited] (SPARK-17885) Spark Streaming deletes checkpointed RDD then tries to load it after restart

2017-10-03 Thread Vishal John (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189483#comment-16189483 ] Vishal John edited comment on SPARK-17885 at 10/3/17 11:27 AM: --- I can see

[jira] [Comment Edited] (SPARK-14172) Hive table partition predicate not passed down correctly

2017-10-03 Thread Saktheesh Balaraj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189481#comment-16189481 ] Saktheesh Balaraj edited comment on SPARK-14172 at 10/3/17 10:06 AM: -

[jira] [Comment Edited] (SPARK-14172) Hive table partition predicate not passed down correctly

2017-10-03 Thread Saktheesh Balaraj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189481#comment-16189481 ] Saktheesh Balaraj edited comment on SPARK-14172 at 10/3/17 10:06 AM: -

[jira] [Comment Edited] (SPARK-14172) Hive table partition predicate not passed down correctly

2017-10-03 Thread Saktheesh Balaraj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189481#comment-16189481 ] Saktheesh Balaraj edited comment on SPARK-14172 at 10/3/17 9:52 AM:

[jira] [Comment Edited] (SPARK-14172) Hive table partition predicate not passed down correctly

2017-10-03 Thread Saktheesh Balaraj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189481#comment-16189481 ] Saktheesh Balaraj edited comment on SPARK-14172 at 10/3/17 9:52 AM:

[jira] [Commented] (SPARK-17885) Spark Streaming deletes checkpointed RDD then tries to load it after restart

2017-10-03 Thread Vishal John (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189483#comment-16189483 ] Vishal John commented on SPARK-17885: - I can see that the checkpointed folder was explicitly deleted

[jira] [Commented] (SPARK-14172) Hive table partition predicate not passed down correctly

2017-10-03 Thread Saktheesh Balaraj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189481#comment-16189481 ] Saktheesh Balaraj commented on SPARK-14172: --- Similar problem is observed while joining 2 hive

[jira] [Commented] (SPARK-17885) Spark Streaming deletes checkpointed RDD then tries to load it after restart

2017-10-03 Thread Vishal John (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189403#comment-16189403 ] Vishal John commented on SPARK-17885: - Hello all, Our application also suffers from the same

[jira] [Resolved] (SPARK-22176) Dataset.show(Int.MaxValue) hits integer overflows

2017-10-03 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22176. - Resolution: Fixed Assignee: Takeshi Yamamuro Fix Version/s: 2.3.0 >

[jira] [Commented] (SPARK-22167) Spark Packaging w/R distro issues

2017-10-03 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189309#comment-16189309 ] holdenk commented on SPARK-22167: - I agree we could improve this, I think though that swapping

[jira] [Updated] (SPARK-22083) When dropping multiple blocks to disk, Spark should release all locks on a failure

2017-10-03 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-22083: Fix Version/s: 2.1.2 > When dropping multiple blocks to disk, Spark should release all locks on a >

[jira] [Updated] (SPARK-18971) Netty issue may cause the shuffle client hang

2017-10-03 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-18971: Fix Version/s: 2.1.2 > Netty issue may cause the shuffle client hang >