[jira] [Commented] (SPARK-20543) R should skip long running or non-essential tests when running on CRAN

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15999262#comment-15999262 ] Apache Spark commented on SPARK-20543: -- User 'felixcheung' has created a pull request for this

[jira] [Resolved] (SPARK-20614) Use the same log4j configuration with Jenkins in AppVeyor

2017-05-05 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-20614. -- Resolution: Fixed Assignee: Hyukjin Kwon Fix Version/s: 2.3.0

[jira] [Commented] (SPARK-20520) R streaming tests failed on Windows

2017-05-05 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15999252#comment-15999252 ] Felix Cheung commented on SPARK-20520: -- waiting for the next RC to try with fix for SPARK-20571 > R

[jira] [Updated] (SPARK-20617) pyspark.sql, filtering with ~isin missing rows

2017-05-05 Thread Ed Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ed Lee updated SPARK-20617: --- Description: Hello encountered a filtering bug using 'isin' in pyspark sql on version 2.2.0, Ubuntu 16.04.

[jira] [Updated] (SPARK-20617) pyspark.sql, filtering with ~isin missing rows

2017-05-05 Thread Ed Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ed Lee updated SPARK-20617: --- Description: Hello encountered a filtering bug using 'isin' in pyspark sql on version 2.2.0, Ubuntu 16.04.

[jira] [Updated] (SPARK-20617) pyspark.sql, filtering with ~isin missing rows

2017-05-05 Thread Ed Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ed Lee updated SPARK-20617: --- Description: Hello encountered a filtering bug using 'isin' in pyspark sql on version 2.2.0, Ubuntu 16.04.

[jira] [Updated] (SPARK-20617) pyspark.sql, filtering with ~isin missing rows

2017-05-05 Thread Ed Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ed Lee updated SPARK-20617: --- Description: Hello encountered a filtering bug using 'isin' in pyspark sql on version 2.2.0, Ubuntu 16.04.

[jira] [Updated] (SPARK-20617) pyspark.sql, filtering with ~isin missing rows

2017-05-05 Thread Ed Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ed Lee updated SPARK-20617: --- Description: Hello encountered a filtering bug using 'isin' in pyspark sql on version 2.2.0, Ubuntu 16.04.

[jira] [Updated] (SPARK-20617) pyspark.sql, filtering with ~isin missing rows

2017-05-05 Thread Ed Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ed Lee updated SPARK-20617: --- Summary: pyspark.sql, filtering with ~isin missing rows (was: pyspark.sql, ~isin when columns contain

[jira] [Updated] (SPARK-20617) pyspark.sql, ~isin when columns contain null (missing rows)

2017-05-05 Thread Ed Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ed Lee updated SPARK-20617: --- Summary: pyspark.sql, ~isin when columns contain null (missing rows) (was: pyspark.sql, isin when columns

[jira] [Updated] (SPARK-20617) pyspark.sql, isin when columns contain null

2017-05-05 Thread Ed Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ed Lee updated SPARK-20617: --- Description: Hello encountered a filtering bug using 'isin' in pyspark sql on version 2.2.0, Ubuntu 16.04.

[jira] [Updated] (SPARK-20617) pyspark.sql, isin when columns contain null

2017-05-05 Thread Ed Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ed Lee updated SPARK-20617: --- Description: Hello encountered a filtering bug using 'isin' in pyspark sql on version 2.2.0, Ubuntu 16.04.

[jira] [Updated] (SPARK-20617) pyspark.sql, isin when columns contain null

2017-05-05 Thread Ed Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ed Lee updated SPARK-20617: --- Description: Hello encountered a filtering bug using 'isin' in pyspark sql on version 2.2.0, Ubuntu 16.04.

[jira] [Updated] (SPARK-20617) pyspark.sql, isin when columns contain null

2017-05-05 Thread Ed Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ed Lee updated SPARK-20617: --- Description: Hello encountered a filtering bug using 'isin' in pyspark sql on version 2.2.0, Ubuntu 16.04.

[jira] [Created] (SPARK-20617) pyspark.sql, isin when columns contain null

2017-05-05 Thread Ed Lee (JIRA)
Ed Lee created SPARK-20617: -- Summary: pyspark.sql, isin when columns contain null Key: SPARK-20617 URL: https://issues.apache.org/jira/browse/SPARK-20617 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-20616) RuleExecutor logDebug of batch results should show diff to start of batch

2017-05-05 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-20616. - Resolution: Fixed Assignee: Juliusz Sompolski Fix Version/s: 2.2.0

[jira] [Commented] (SPARK-19532) [Core]`DataStreamer for file` threads of DFSOutputStream leak if set `spark.speculation` to true

2017-05-05 Thread Abhishek Madav (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15999016#comment-15999016 ] Abhishek Madav commented on SPARK-19532: I am running into this issue wherein codepath similar to

[jira] [Assigned] (SPARK-20615) SparseVector.argmax throws IndexOutOfBoundsException when the sparse vector has a size greater than zero but no elements defined.

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20615: Assignee: (was: Apache Spark) > SparseVector.argmax throws IndexOutOfBoundsException

[jira] [Assigned] (SPARK-20615) SparseVector.argmax throws IndexOutOfBoundsException when the sparse vector has a size greater than zero but no elements defined.

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20615: Assignee: Apache Spark > SparseVector.argmax throws IndexOutOfBoundsException when the

[jira] [Commented] (SPARK-20615) SparseVector.argmax throws IndexOutOfBoundsException when the sparse vector has a size greater than zero but no elements defined.

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998970#comment-15998970 ] Apache Spark commented on SPARK-20615: -- User 'jonmclean' has created a pull request for this issue:

[jira] [Updated] (SPARK-20132) Add documentation for column string functions

2017-05-05 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-20132: --- Fix Version/s: 2.2.0 > Add documentation for column string functions >

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2017-05-05 Thread Rupesh Mane (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998900#comment-15998900 ] Rupesh Mane commented on SPARK-18105: - I'm facing this issue with Spark 2.1.0 but not with Spark

[jira] [Updated] (SPARK-19910) `stack` should not reject NULL values due to type mismatch

2017-05-05 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-19910: -- Affects Version/s: 2.1.1 > `stack` should not reject NULL values due to type mismatch >

[jira] [Commented] (SPARK-19910) `stack` should not reject NULL values due to type mismatch

2017-05-05 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998771#comment-15998771 ] Dongjoon Hyun commented on SPARK-19910: --- Hi, [~cloud_fan] and [~smilegator]. Could you review this

[jira] [Commented] (SPARK-10878) Race condition when resolving Maven coordinates via Ivy

2017-05-05 Thread Jeeyoung Kim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998731#comment-15998731 ] Jeeyoung Kim commented on SPARK-10878: -- [~joshrosen] Yes, I realized what are potential race

[jira] [Updated] (SPARK-20603) Flaky test: o.a.s.sql.kafka010.KafkaSourceSuite deserialization of initial offset with Spark 2.1.0

2017-05-05 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20603: - Affects Version/s: 2.1.1 2.1.0 > Flaky test:

[jira] [Resolved] (SPARK-20603) Flaky test: o.a.s.sql.kafka010.KafkaSourceSuite deserialization of initial offset with Spark 2.1.0

2017-05-05 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20603. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.2 > Flaky test:

[jira] [Assigned] (SPARK-20569) RuntimeReplaceable functions accept invalid third parameter

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20569: Assignee: (was: Apache Spark) > RuntimeReplaceable functions accept invalid third

[jira] [Assigned] (SPARK-20569) RuntimeReplaceable functions accept invalid third parameter

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20569: Assignee: Apache Spark > RuntimeReplaceable functions accept invalid third parameter >

[jira] [Commented] (SPARK-20569) RuntimeReplaceable functions accept invalid third parameter

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998690#comment-15998690 ] Apache Spark commented on SPARK-20569: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-20571) Flaky SparkR StructuredStreaming tests

2017-05-05 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998680#comment-15998680 ] Burak Yavuz commented on SPARK-20571: - Thanks! > Flaky SparkR StructuredStreaming tests >

[jira] [Commented] (SPARK-18971) Netty issue may cause the shuffle client hang

2017-05-05 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998654#comment-15998654 ] Shixiong Zhu commented on SPARK-18971: -- [~tgraves] No, as far as I known. But since Spark 2.2.0 has

[jira] [Comment Edited] (SPARK-18971) Netty issue may cause the shuffle client hang

2017-05-05 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998654#comment-15998654 ] Shixiong Zhu edited comment on SPARK-18971 at 5/5/17 5:49 PM: -- [~tgraves]

[jira] [Assigned] (SPARK-20616) RuleExecutor logDebug of batch results should show diff to start of batch

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20616: Assignee: Apache Spark > RuleExecutor logDebug of batch results should show diff to start

[jira] [Assigned] (SPARK-20616) RuleExecutor logDebug of batch results should show diff to start of batch

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20616: Assignee: (was: Apache Spark) > RuleExecutor logDebug of batch results should show

[jira] [Commented] (SPARK-20616) RuleExecutor logDebug of batch results should show diff to start of batch

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998639#comment-15998639 ] Apache Spark commented on SPARK-20616: -- User 'juliuszsompolski' has created a pull request for this

[jira] [Created] (SPARK-20616) RuleExecutor logDebug of batch results should show diff to start of batch

2017-05-05 Thread Juliusz Sompolski (JIRA)
Juliusz Sompolski created SPARK-20616: - Summary: RuleExecutor logDebug of batch results should show diff to start of batch Key: SPARK-20616 URL: https://issues.apache.org/jira/browse/SPARK-20616

[jira] [Updated] (SPARK-20564) a lot of executor failures when the executor number is more than 2000

2017-05-05 Thread Hua Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hua Liu updated SPARK-20564: Priority: Minor (was: Major) > a lot of executor failures when the executor number is more than 2000 >

[jira] [Resolved] (SPARK-20381) ObjectHashAggregateExec is missing numOutputRows

2017-05-05 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-20381. - Resolution: Fixed Assignee: yucai Fix Version/s: 2.2.0 > ObjectHashAggregateExec is

[jira] [Commented] (SPARK-20608) Standby namenodes should be allowed to included in yarn.spark.access.namenodes to support HDFS HA

2017-05-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998544#comment-15998544 ] Marcelo Vanzin commented on SPARK-20608: Doesn't it work if you add the namespace (not the NN

[jira] [Commented] (SPARK-18971) Netty issue may cause the shuffle client hang

2017-05-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998531#comment-15998531 ] Thomas Graves commented on SPARK-18971: --- [~zsxwing]have you seen any issues with the new netty

[jira] [Comment Edited] (SPARK-18971) Netty issue may cause the shuffle client hang

2017-05-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998531#comment-15998531 ] Thomas Graves edited comment on SPARK-18971 at 5/5/17 4:31 PM: ---

[jira] [Assigned] (SPARK-20613) Double quotes in Windows batch script

2017-05-05 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman reassigned SPARK-20613: - Assignee: Jarrett Meyer > Double quotes in Windows batch script >

[jira] [Commented] (SPARK-20613) Double quotes in Windows batch script

2017-05-05 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998474#comment-15998474 ] Felix Cheung commented on SPARK-20613: -- [~shivaram]could you add jarretmeyer to contributor list in

[jira] [Resolved] (SPARK-20613) Double quotes in Windows batch script

2017-05-05 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-20613. -- Resolution: Fixed Fix Version/s: 2.3.0 2.2.0

[jira] [Commented] (SPARK-20569) RuntimeReplaceable functions accept invalid third parameter

2017-05-05 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998472#comment-15998472 ] Wenchen Fan commented on SPARK-20569: - yea this is a bug, I'm working on a fix > RuntimeReplaceable

[jira] [Commented] (SPARK-20581) Using AVG or SUM on a INT/BIGINT column with fraction operator will yield BIGINT instead of DOUBLE

2017-05-05 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998465#comment-15998465 ] Wenchen Fan commented on SPARK-20581: - [~smilegator] do you remember which PR fixed it? we can

[jira] [Commented] (SPARK-20615) SparseVector.argmax throws IndexOutOfBoundsException when the sparse vector has a size greater than zero but no elements defined.

2017-05-05 Thread Jon McLean (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998460#comment-15998460 ] Jon McLean commented on SPARK-20615: Thank you. I will submit a patch with tests. >

[jira] [Commented] (SPARK-20615) SparseVector.argmax throws IndexOutOfBoundsException when the sparse vector has a size greater than zero but no elements defined.

2017-05-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998459#comment-15998459 ] Sean Owen commented on SPARK-20615: --- Agree, I think you just want to return 0 if numActives == 0 early

[jira] [Created] (SPARK-20615) SparseVector.argmax throws IndexOutOfBoundsException when the sparse vector has a size greater than zero but no elements defined.

2017-05-05 Thread Jon McLean (JIRA)
Jon McLean created SPARK-20615: -- Summary: SparseVector.argmax throws IndexOutOfBoundsException when the sparse vector has a size greater than zero but no elements defined. Key: SPARK-20615 URL:

[jira] [Commented] (SPARK-20495) Add StorageLevel to cacheTable API

2017-05-05 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998442#comment-15998442 ] Wenchen Fan commented on SPARK-20495: - we usually don't backport new API changes, but this one is

[jira] [Comment Edited] (SPARK-20495) Add StorageLevel to cacheTable API

2017-05-05 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998442#comment-15998442 ] Wenchen Fan edited comment on SPARK-20495 at 5/5/17 3:00 PM: - we usually

[jira] [Commented] (SPARK-20495) Add StorageLevel to cacheTable API

2017-05-05 Thread PJ Fanning (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998435#comment-15998435 ] PJ Fanning commented on SPARK-20495: Thanks everyone for working on this change. Is it too late to

[jira] [Resolved] (SPARK-20495) Add StorageLevel to cacheTable API

2017-05-05 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-20495. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 17802

[jira] [Commented] (SPARK-20612) Unresolvable attribute in Filter won't throw analysis exception

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998402#comment-15998402 ] Apache Spark commented on SPARK-20612: -- User 'viirya' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20614) Use the same log4j configuration with Jenkins in AppVeyor

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20614: Assignee: (was: Apache Spark) > Use the same log4j configuration with Jenkins in

[jira] [Assigned] (SPARK-20614) Use the same log4j configuration with Jenkins in AppVeyor

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20614: Assignee: Apache Spark > Use the same log4j configuration with Jenkins in AppVeyor >

[jira] [Commented] (SPARK-20614) Use the same log4j configuration with Jenkins in AppVeyor

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998359#comment-15998359 ] Apache Spark commented on SPARK-20614: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Created] (SPARK-20614) Use the same log4j configuration with Jenkins in AppVeyor

2017-05-05 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-20614: Summary: Use the same log4j configuration with Jenkins in AppVeyor Key: SPARK-20614 URL: https://issues.apache.org/jira/browse/SPARK-20614 Project: Spark

[jira] [Commented] (SPARK-20489) Different results in local mode and yarn mode when working with dates (race condition with SimpleDateFormat?)

2017-05-05 Thread Rick Moritz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998228#comment-15998228 ] Rick Moritz commented on SPARK-20489: - If someone could try and replicate my observations, I think

[jira] [Commented] (SPARK-20608) Standby namenodes should be allowed to included in yarn.spark.access.namenodes to support HDFS HA

2017-05-05 Thread Yuechen Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998197#comment-15998197 ] Yuechen Chen commented on SPARK-20608: -- [~ste...@apache.org] Your worry is reasonable. In our tests,

[jira] [Commented] (SPARK-20608) Standby namenodes should be allowed to included in yarn.spark.access.namenodes to support HDFS HA

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998189#comment-15998189 ] Apache Spark commented on SPARK-20608: -- User 'morenn520' has created a pull request for this issue:

[jira] [Updated] (SPARK-20613) Double quotes in Windows batch script

2017-05-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-20613: -- Priority: Major (was: Blocker) > Double quotes in Windows batch script >

[jira] [Updated] (SPARK-20613) Double quotes in Windows batch script

2017-05-05 Thread Jarrett Meyer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarrett Meyer updated SPARK-20613: -- Description: This is a new issue in version 2.1.1. This problem was not present in 2.1.0. In

[jira] [Assigned] (SPARK-20613) Double quotes in Windows batch script

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20613: Assignee: (was: Apache Spark) > Double quotes in Windows batch script >

[jira] [Assigned] (SPARK-20613) Double quotes in Windows batch script

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20613: Assignee: Apache Spark > Double quotes in Windows batch script >

[jira] [Commented] (SPARK-20613) Double quotes in Windows batch script

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998160#comment-15998160 ] Apache Spark commented on SPARK-20613: -- User 'jarrettmeyer' has created a pull request for this

[jira] [Created] (SPARK-20613) Double quotes in Windows batch script

2017-05-05 Thread Jarrett Meyer (JIRA)
Jarrett Meyer created SPARK-20613: - Summary: Double quotes in Windows batch script Key: SPARK-20613 URL: https://issues.apache.org/jira/browse/SPARK-20613 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-20608) Standby namenodes should be allowed to included in yarn.spark.access.namenodes to support HDFS HA

2017-05-05 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998150#comment-15998150 ] Steve Loughran commented on SPARK-20608: Probably good to pull in someone who understands HDFS

[jira] [Assigned] (SPARK-20546) spark-class gets syntax error in posix mode

2017-05-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-20546: - Assignee: Jessie Yu > spark-class gets syntax error in posix mode >

[jira] [Resolved] (SPARK-20546) spark-class gets syntax error in posix mode

2017-05-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20546. --- Resolution: Fixed Fix Version/s: 2.1.2 2.2.1 Issue resolved by pull

[jira] [Commented] (SPARK-20611) Spark kinesis connector doesnt work with cloudera distribution

2017-05-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998050#comment-15998050 ] Sean Owen commented on SPARK-20611: --- No, there's not necessarily any problem in Spark. The Logging

[jira] [Updated] (SPARK-20608) Standby namenodes should be allowed to included in yarn.spark.access.namenodes to support HDFS HA

2017-05-05 Thread Yuechen Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuechen Chen updated SPARK-20608: - Description: If one Spark Application need to access remote namenodes,

[jira] [Assigned] (SPARK-20612) Unresolvable attribute in Filter won't throw analysis exception

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20612: Assignee: Apache Spark > Unresolvable attribute in Filter won't throw analysis exception

[jira] [Assigned] (SPARK-20612) Unresolvable attribute in Filter won't throw analysis exception

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20612: Assignee: (was: Apache Spark) > Unresolvable attribute in Filter won't throw analysis

[jira] [Commented] (SPARK-20612) Unresolvable attribute in Filter won't throw analysis exception

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997993#comment-15997993 ] Apache Spark commented on SPARK-20612: -- User 'viirya' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-20611) Spark kinesis connector doesnt work with cloudera distribution

2017-05-05 Thread sumit (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997987#comment-15997987 ] sumit edited comment on SPARK-20611 at 5/5/17 9:32 AM: --- Hi [~sowen] does this mean

[jira] [Comment Edited] (SPARK-20611) Spark kinesis connector doesnt work with cloudera distribution

2017-05-05 Thread sumit (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997987#comment-15997987 ] sumit edited comment on SPARK-20611 at 5/5/17 9:29 AM: --- Hi [~sowen] does this mean

[jira] [Commented] (SPARK-20611) Spark kinesis connector doesnt work with cloudera distribution

2017-05-05 Thread sumit (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997987#comment-15997987 ] sumit commented on SPARK-20611: --- Hi [~sowen] does this mean I should log ticket to CDH . I thought as per

[jira] [Created] (SPARK-20612) Unresolvable attribute in Filter won't throw analysis exception

2017-05-05 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-20612: --- Summary: Unresolvable attribute in Filter won't throw analysis exception Key: SPARK-20612 URL: https://issues.apache.org/jira/browse/SPARK-20612 Project: Spark

[jira] [Resolved] (SPARK-20611) Spark kinesis connector doesnt work with cloudera distribution

2017-05-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20611. --- Resolution: Not A Problem If a question is specific to CDH, it doesn't belong here, but rather at

[jira] [Closed] (SPARK-20611) Spark kinesis connector doesnt work with cloudera distribution

2017-05-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen closed SPARK-20611. - > Spark kinesis connector doesnt work with cloudera distribution >

[jira] [Updated] (SPARK-20608) Standby namenodes should be allowed to included in yarn.spark.access.namenodes to support HDFS HA

2017-05-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-20608: -- Priority: Minor (was: Major) Issue Type: Improvement (was: Bug) > Standby namenodes should be

[jira] [Commented] (SPARK-20608) Standby namenodes should be allowed to included in yarn.spark.access.namenodes to support HDFS HA

2017-05-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997948#comment-15997948 ] Sean Owen commented on SPARK-20608: --- CC [~vanzin] [~ste...@apache.org] > Standby namenodes should be

[jira] [Commented] (SPARK-20472) Support for Dynamic Configuration

2017-05-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997946#comment-15997946 ] Sean Owen commented on SPARK-20472: --- JVM config matters. How do you change the driver heap size in

[jira] [Updated] (SPARK-20608) Standby namenodes should be allowed to included in yarn.spark.access.namenodes to support HDFS HA

2017-05-05 Thread Yuechen Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuechen Chen updated SPARK-20608: - Description: If one Spark Application need to access remote namenodes,

[jira] [Assigned] (SPARK-20608) Standby namenodes should be allowed to included in yarn.spark.access.namenodes to support HDFS HA

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20608: Assignee: Apache Spark > Standby namenodes should be allowed to included in >

[jira] [Updated] (SPARK-20608) Standby namenodes should be allowed to included in yarn.spark.access.namenodes to support HDFS HA

2017-05-05 Thread Yuechen Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuechen Chen updated SPARK-20608: - Description: If one Spark Application need to access remote namenodes,

[jira] [Assigned] (SPARK-20608) Standby namenodes should be allowed to included in yarn.spark.access.namenodes to support HDFS HA

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20608: Assignee: (was: Apache Spark) > Standby namenodes should be allowed to included in >

[jira] [Commented] (SPARK-20608) Standby namenodes should be allowed to included in yarn.spark.access.namenodes to support HDFS HA

2017-05-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997900#comment-15997900 ] Apache Spark commented on SPARK-20608: -- User 'morenn520' has created a pull request for this issue:

[jira] [Updated] (SPARK-20611) Spark kinesis connector doesnt work with cloudera distribution

2017-05-05 Thread sumit (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sumit updated SPARK-20611: -- Summary: Spark kinesis connector doesnt work with cloudera distribution (was: Spark kinesis connector doesn

[jira] [Commented] (SPARK-20611) Spark kinesis connector doesn work with cloudera distribution

2017-05-05 Thread sumit (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997889#comment-15997889 ] sumit commented on SPARK-20611: --- please evaluate and review the patch file . If it looks good then I would

[jira] [Updated] (SPARK-20611) Spark kinesis connector doesn work with cloudera distribution

2017-05-05 Thread sumit (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sumit updated SPARK-20611: -- Attachment: spark-kcl.patch attached Patch is doing exactly same what we have done in the past for cassandra

[jira] [Created] (SPARK-20611) Spark kinesis connector doesn work with cloudera distribution

2017-05-05 Thread sumit (JIRA)
sumit created SPARK-20611: - Summary: Spark kinesis connector doesn work with cloudera distribution Key: SPARK-20611 URL: https://issues.apache.org/jira/browse/SPARK-20611 Project: Spark Issue

[jira] [Closed] (SPARK-20610) Support a function get DataFrame/DataSet from Transformer

2017-05-05 Thread darion yaphet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] darion yaphet closed SPARK-20610. - Resolution: Won't Fix > Support a function get DataFrame/DataSet from Transformer >

[jira] [Created] (SPARK-20610) Support a function get DataFrame/DataSet from Transformer

2017-05-05 Thread darion yaphet (JIRA)
darion yaphet created SPARK-20610: - Summary: Support a function get DataFrame/DataSet from Transformer Key: SPARK-20610 URL: https://issues.apache.org/jira/browse/SPARK-20610 Project: Spark

[jira] [Commented] (SPARK-20472) Support for Dynamic Configuration

2017-05-05 Thread Shahbaz Hussain (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997874#comment-15997874 ] Shahbaz Hussain commented on SPARK-20472: - Yes ,the idea is to have a way by which we can persist

[jira] [Closed] (SPARK-20545) union set operator should default to DISTINCT and not ALL semantics

2017-05-05 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-20545. --- Resolution: Cannot Reproduce > union set operator should default to DISTINCT and not ALL semantics >

[jira] [Commented] (SPARK-20545) union set operator should default to DISTINCT and not ALL semantics

2017-05-05 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997845#comment-15997845 ] Xiao Li commented on SPARK-20545: - Please reopen it if you still hit this issue. Thanks! > union set

[jira] [Commented] (SPARK-20545) union set operator should default to DISTINCT and not ALL semantics

2017-05-05 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997843#comment-15997843 ] Xiao Li commented on SPARK-20545: - You can try {noformat} select 3 as `col` union select 3 as `col`

  1   2   >