[jira] [Updated] (SPARK-24195) sc.addFile for local:/ path is broken

2018-05-06 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-24195: - Description: In changing SPARK-6300

[jira] [Updated] (SPARK-24195) sc.addFile for local:/ path is broken

2018-05-06 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-24195: - Affects Version/s: 1.3.1 1.4.1 1.5.2

[jira] [Created] (SPARK-24195) sc.addFile for local:/ path is broken

2018-05-06 Thread Felix Cheung (JIRA)
Felix Cheung created SPARK-24195: Summary: sc.addFile for local:/ path is broken Key: SPARK-24195 URL: https://issues.apache.org/jira/browse/SPARK-24195 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-24194) HadoopFsRelation cannot overwrite a path that is also being read from

2018-05-06 Thread yangz (JIRA)
yangz created SPARK-24194: - Summary: HadoopFsRelation cannot overwrite a path that is also being read from Key: SPARK-24194 URL: https://issues.apache.org/jira/browse/SPARK-24194 Project: Spark

[jira] [Commented] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

2018-05-06 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465433#comment-16465433 ] Felix Cheung commented on SPARK-23291: -- I don't disagree the behavior issue. (ah, so someone did run

[jira] [Resolved] (SPARK-24126) PySpark tests leave a lot of garbage in /tmp

2018-05-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24126. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21198

[jira] [Assigned] (SPARK-24126) PySpark tests leave a lot of garbage in /tmp

2018-05-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24126: Assignee: Marcelo Vanzin > PySpark tests leave a lot of garbage in /tmp >

[jira] [Resolved] (SPARK-24017) Refactor ExternalCatalog to be an interface

2018-05-06 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24017. - Resolution: Fixed Fix Version/s: 2.4.0 > Refactor ExternalCatalog to be an interface >

[jira] [Commented] (SPARK-23094) Json Readers choose wrong encoding when bad records are present and fail

2018-05-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465397#comment-16465397 ] Apache Spark commented on SPARK-23094: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Commented] (SPARK-23723) New charset option for json datasource

2018-05-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465398#comment-16465398 ] Apache Spark commented on SPARK-23723: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Commented] (SPARK-23724) Custom record separator for jsons in charsets different from UTF-8

2018-05-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465399#comment-16465399 ] Apache Spark commented on SPARK-23724: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Commented] (SPARK-24158) Enable no-data micro batches for streaming joins

2018-05-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465392#comment-16465392 ] Apache Spark commented on SPARK-24158: -- User 'tdas' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24158) Enable no-data micro batches for streaming joins

2018-05-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24158: Assignee: Apache Spark (was: Tathagata Das) > Enable no-data micro batches for streaming

[jira] [Assigned] (SPARK-24158) Enable no-data micro batches for streaming joins

2018-05-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24158: Assignee: Tathagata Das (was: Apache Spark) > Enable no-data micro batches for streaming

[jira] [Commented] (SPARK-19512) codegen for compare structs fails

2018-05-06 Thread howie yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465386#comment-16465386 ] howie yu commented on SPARK-19512: -- Hi I still have this issue in 2.3.0

[jira] [Comment Edited] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

2018-05-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465356#comment-16465356 ] Hyukjin Kwon edited comment on SPARK-23291 at 5/7/18 2:04 AM: --

[jira] [Commented] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

2018-05-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465356#comment-16465356 ] Hyukjin Kwon commented on SPARK-23291: -- [~felixcheung], sure, I agree with that in general. However,

[jira] [Commented] (SPARK-23977) Add commit protocol binding to Hadoop 3.1 PathOutputCommitter mechanism

2018-05-06 Thread Wade Jensen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465344#comment-16465344 ] Wade Jensen commented on SPARK-23977: - [~ste...@apache.org] is the intention of this ticket to

[jira] [Comment Edited] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

2018-05-06 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465307#comment-16465307 ] Felix Cheung edited comment on SPARK-23291 at 5/6/18 10:38 PM: --- actually,

[jira] [Comment Edited] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

2018-05-06 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465307#comment-16465307 ] Felix Cheung edited comment on SPARK-23291 at 5/6/18 10:35 PM: --- actually,

[jira] [Commented] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

2018-05-06 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465307#comment-16465307 ] Felix Cheung commented on SPARK-23291: -- actually, I'm not sure we should backport this to a x.x.1

[jira] [Comment Edited] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

2018-05-06 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465307#comment-16465307 ] Felix Cheung edited comment on SPARK-23291 at 5/6/18 10:34 PM: --- actually,

[jira] [Assigned] (SPARK-24193) Sort by disk when number of limit is big in TakeOrderedAndProjectExec

2018-05-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24193: Assignee: (was: Apache Spark) > Sort by disk when number of limit is big in

[jira] [Assigned] (SPARK-24193) Sort by disk when number of limit is big in TakeOrderedAndProjectExec

2018-05-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24193: Assignee: Apache Spark > Sort by disk when number of limit is big in

[jira] [Commented] (SPARK-24193) Sort by disk when number of limit is big in TakeOrderedAndProjectExec

2018-05-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465192#comment-16465192 ] Apache Spark commented on SPARK-24193: -- User 'jinxing64' has created a pull request for this issue:

[jira] [Created] (SPARK-24193) Sort by disk when number of limit is big in TakeOrderedAndProjectExec

2018-05-06 Thread jin xing (JIRA)
jin xing created SPARK-24193: Summary: Sort by disk when number of limit is big in TakeOrderedAndProjectExec Key: SPARK-24193 URL: https://issues.apache.org/jira/browse/SPARK-24193 Project: Spark

[jira] [Created] (SPARK-24192) Invalid Spark URL in local spark session since upgrading from org.apache.spark:spark-sql_2.11:2.2.1 to org.apache.spark:spark-sql_2.11:2.3.0

2018-05-06 Thread Tal Barda (JIRA)
Tal Barda created SPARK-24192: - Summary: Invalid Spark URL in local spark session since upgrading from org.apache.spark:spark-sql_2.11:2.2.1 to org.apache.spark:spark-sql_2.11:2.3.0 Key: SPARK-24192 URL:

[jira] [Commented] (SPARK-10878) Race condition when resolving Maven coordinates via Ivy

2018-05-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465065#comment-16465065 ] Apache Spark commented on SPARK-10878: -- User 'kiszk' has created a pull request for this issue:

[jira] [Commented] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

2018-05-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465030#comment-16465030 ] Apache Spark commented on SPARK-23291: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

2018-05-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465028#comment-16465028 ] Apache Spark commented on SPARK-23291: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Updated] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

2018-05-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-23291: - Target Version/s: 2.3.1, 2.4.0 (was: 2.4.0) > SparkR : substr : In SparkR dataframe , starting

[jira] [Commented] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

2018-05-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465015#comment-16465015 ] Hyukjin Kwon commented on SPARK-23291: -- Sure, will do. > SparkR : substr : In SparkR dataframe ,

[jira] [Commented] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2018-05-06 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465010#comment-16465010 ] Liang-Chi Hsieh commented on SPARK-21274: - [~dkbiswal] No problem. Current EXCEPT ALL rewrite is

[jira] [Commented] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2018-05-06 Thread Dilip Biswal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465008#comment-16465008 ] Dilip Biswal commented on SPARK-21274: -- [~viirya] Thank you. I will study this more tomorrow. From a

[jira] [Comment Edited] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2018-05-06 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465000#comment-16465000 ] Liang-Chi Hsieh edited comment on SPARK-21274 at 5/6/18 7:30 AM: - I read

[jira] [Comment Edited] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2018-05-06 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465000#comment-16465000 ] Liang-Chi Hsieh edited comment on SPARK-21274 at 5/6/18 7:29 AM: - I read

[jira] [Commented] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2018-05-06 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465000#comment-16465000 ] Liang-Chi Hsieh commented on SPARK-21274: - I read the design doc. It looks correct to me. I found