[jira] [Assigned] (SPARK-22349) In on-heap mode, when allocating memory from pool,we should fill memory with `MEMORY_DEBUG_FILL_CLEAN_VALUE`

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22349: Assignee: Apache Spark > In on-heap mode, when allocating memory from pool,we should fill

[jira] [Assigned] (SPARK-22349) In on-heap mode, when allocating memory from pool,we should fill memory with `MEMORY_DEBUG_FILL_CLEAN_VALUE`

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22349: Assignee: (was: Apache Spark) > In on-heap mode, when allocating memory from pool,we s

[jira] [Commented] (SPARK-22349) In on-heap mode, when allocating memory from pool,we should fill memory with `MEMORY_DEBUG_FILL_CLEAN_VALUE`

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16218185#comment-16218185 ] Apache Spark commented on SPARK-22349: -- User '10110346' has created a pull request f

[jira] [Created] (SPARK-22349) In on-heap mode, when allocating memory from pool,we should fill memory with `MEMORY_DEBUG_FILL_CLEAN_VALUE`

2017-10-24 Thread liuxian (JIRA)
liuxian created SPARK-22349: --- Summary: In on-heap mode, when allocating memory from pool,we should fill memory with `MEMORY_DEBUG_FILL_CLEAN_VALUE` Key: SPARK-22349 URL: https://issues.apache.org/jira/browse/SPARK-2234

[jira] [Assigned] (SPARK-13947) The error message from using an invalid table reference is not clear

2017-10-24 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-13947: --- Assignee: Ruben Berenguel > The error message from using an invalid table reference is not clear > -

[jira] [Resolved] (SPARK-13947) The error message from using an invalid table reference is not clear

2017-10-24 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-13947. - Resolution: Fixed Fix Version/s: 2.3.0 > The error message from using an invalid table reference i

[jira] [Resolved] (SPARK-21101) Error running Hive temporary UDTF on latest Spark 2.2

2017-10-24 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21101. - Resolution: Fixed Fix Version/s: 2.3.0 > Error running Hive temporary UDTF on latest Spark 2.2 > -

[jira] [Assigned] (SPARK-21101) Error running Hive temporary UDTF on latest Spark 2.2

2017-10-24 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-21101: --- Assignee: Yuming Wang > Error running Hive temporary UDTF on latest Spark 2.2 >

[jira] [Resolved] (SPARK-22348) The table cache providing ColumnarBatch should also do partition batch pruning

2017-10-24 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22348. - Resolution: Fixed Assignee: Liang-Chi Hsieh Fix Version/s: 2.3.0 > The table cach

[jira] [Updated] (SPARK-22344) Prevent R CMD check from using /tmp

2017-10-24 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-22344: - Affects Version/s: 2.3.0 1.6.3 2.2.0 > Prevent R CM

[jira] [Commented] (SPARK-21616) SparkR 2.3.0 migration guide, release note

2017-10-24 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16218104#comment-16218104 ] Felix Cheung commented on SPARK-21616: -- True, I don't know if we are tracking change

[jira] [Commented] (SPARK-15474) ORC data source fails to write and read back empty dataframe

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16218061#comment-16218061 ] Apache Spark commented on SPARK-15474: -- User 'dongjoon-hyun' has created a pull requ

[jira] [Assigned] (SPARK-22335) Union for DataSet uses column order instead of types for union

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22335: Assignee: (was: Apache Spark) > Union for DataSet uses column order instead of types f

[jira] [Assigned] (SPARK-22335) Union for DataSet uses column order instead of types for union

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22335: Assignee: Apache Spark > Union for DataSet uses column order instead of types for union >

[jira] [Commented] (SPARK-22335) Union for DataSet uses column order instead of types for union

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16218035#comment-16218035 ] Apache Spark commented on SPARK-22335: -- User 'viirya' has created a pull request for

[jira] [Commented] (SPARK-22335) Union for DataSet uses column order instead of types for union

2017-10-24 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16218027#comment-16218027 ] Liang-Chi Hsieh commented on SPARK-22335: - [~CBribiescas] The column position in

[jira] [Assigned] (SPARK-22348) The table cache providing ColumnarBatch should also do partition batch pruning

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22348: Assignee: Apache Spark > The table cache providing ColumnarBatch should also do partition

[jira] [Assigned] (SPARK-22348) The table cache providing ColumnarBatch should also do partition batch pruning

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22348: Assignee: (was: Apache Spark) > The table cache providing ColumnarBatch should also do

[jira] [Commented] (SPARK-22348) The table cache providing ColumnarBatch should also do partition batch pruning

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217992#comment-16217992 ] Apache Spark commented on SPARK-22348: -- User 'viirya' has created a pull request for

[jira] [Created] (SPARK-22348) The table cache providing ColumnarBatch should also do partition batch pruning

2017-10-24 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-22348: --- Summary: The table cache providing ColumnarBatch should also do partition batch pruning Key: SPARK-22348 URL: https://issues.apache.org/jira/browse/SPARK-22348

[jira] [Commented] (SPARK-22277) Chi Square selector garbling Vector content.

2017-10-24 Thread Peng Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217979#comment-16217979 ] Peng Meng commented on SPARK-22277: --- For problem 1 and 2, could you please post the tes

[jira] [Updated] (SPARK-17074) generate equi-height histogram for column

2017-10-24 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-17074: - Affects Version/s: (was: 2.0.0) 2.3.0 > generate equi-height histogram

[jira] [Updated] (SPARK-22347) UDF is evaluated when 'F.when' condition is false

2017-10-24 Thread Nicolas Porter (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Porter updated SPARK-22347: --- Description: Here's a simple example on how to reproduce this: {code} from pyspark.sql impor

[jira] [Updated] (SPARK-22347) UDF is evaluated when 'F.when' condition is false

2017-10-24 Thread Nicolas Porter (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Porter updated SPARK-22347: --- Description: Here's a simple example on how to reproduce this: {code} from pyspark.sql impor

[jira] [Updated] (SPARK-22347) UDF is evaluated when 'F.when' condition is false

2017-10-24 Thread Nicolas Porter (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Porter updated SPARK-22347: --- Description: Here's a simple example on how to reproduce this: {code} from pyspark.sql impor

[jira] [Updated] (SPARK-22347) UDF is evaluated when 'F.when' condition is false

2017-10-24 Thread Nicolas Porter (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Porter updated SPARK-22347: --- Description: Here's a simple example on how to reproduce this: {code} from pyspark.sql impor

[jira] [Updated] (SPARK-22347) UDF is evaluated when 'F.when' condition is false

2017-10-24 Thread Nicolas Porter (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Porter updated SPARK-22347: --- Description: Here's a simple example on how to reproduce this: {code} from pyspark.sql impor

[jira] [Updated] (SPARK-22347) UDF is evaluated when 'F.when' condition is false

2017-10-24 Thread Nicolas Porter (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Porter updated SPARK-22347: --- Description: Here's a simple example on how to reproduce this: {code} from pyspark.sql impor

[jira] [Created] (SPARK-22347) UDF is evaluated when 'F.when' condition is false

2017-10-24 Thread Nicolas Porter (JIRA)
Nicolas Porter created SPARK-22347: -- Summary: UDF is evaluated when 'F.when' condition is false Key: SPARK-22347 URL: https://issues.apache.org/jira/browse/SPARK-22347 Project: Spark Issue T

[jira] [Updated] (SPARK-22346) Update VectorAssembler to work with StreamingDataframes

2017-10-24 Thread Bago Amirbekian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bago Amirbekian updated SPARK-22346: Description: The issue In batch mode, VectorAssembler can take multiple columns of VectorTy

[jira] [Updated] (SPARK-22346) Update VectorAssembler to work with StreamingDataframes

2017-10-24 Thread Bago Amirbekian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bago Amirbekian updated SPARK-22346: Description: The issue In batch mode, VectorAssembler can take multiple columns of VectorTy

[jira] [Updated] (SPARK-22346) Update VectorAssembler to work with StreamingDataframes

2017-10-24 Thread Bago Amirbekian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bago Amirbekian updated SPARK-22346: Description: The issue In batch mode, VectorAssembler can take multiple columns of VectorTy

[jira] [Updated] (SPARK-22346) Update VectorAssembler to work with StreamingDataframes

2017-10-24 Thread Bago Amirbekian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bago Amirbekian updated SPARK-22346: Description: The issue In batch mode, VectorAssembler can take multiple columns of VectorTy

[jira] [Updated] (SPARK-22346) Update VectorAssembler to work with StreamingDataframes

2017-10-24 Thread Bago Amirbekian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bago Amirbekian updated SPARK-22346: Description: The issue In batch mode, VectorAssembler can take multiple columns of VectorTy

[jira] [Created] (SPARK-22346) Update VectorAssembler to work with StreamingDataframes

2017-10-24 Thread Bago Amirbekian (JIRA)
Bago Amirbekian created SPARK-22346: --- Summary: Update VectorAssembler to work with StreamingDataframes Key: SPARK-22346 URL: https://issues.apache.org/jira/browse/SPARK-22346 Project: Spark

[jira] [Commented] (SPARK-22340) pyspark setJobGroup doesn't match java threads

2017-10-24 Thread Leif Walsh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217835#comment-16217835 ] Leif Walsh commented on SPARK-22340: Ok, this is fairly straightforward. The problem

[jira] [Commented] (SPARK-22345) Sort-merge join generates incorrect code for CodegenFallback filter conditions

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217772#comment-16217772 ] Apache Spark commented on SPARK-22345: -- User 'rdblue' has created a pull request for

[jira] [Assigned] (SPARK-22345) Sort-merge join generates incorrect code for CodegenFallback filter conditions

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22345: Assignee: Apache Spark > Sort-merge join generates incorrect code for CodegenFallback filt

[jira] [Assigned] (SPARK-22345) Sort-merge join generates incorrect code for CodegenFallback filter conditions

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22345: Assignee: (was: Apache Spark) > Sort-merge join generates incorrect code for CodegenFa

[jira] [Updated] (SPARK-22345) Sort-merge join generates incorrect code for CodegenFallback filter conditions

2017-10-24 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated SPARK-22345: -- Description: I have a job that is producing incorrect results from a sort-merge join with a filter on

[jira] [Commented] (SPARK-22316) Cannot Select ReducedAggregator Column

2017-10-24 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217666#comment-16217666 ] Russell Spitzer commented on SPARK-22316: - [~hvanhovell] This was the ticket I to

[jira] [Created] (SPARK-22345) Sort-merge join generates incorrect code for CodegenFallback filter conditions

2017-10-24 Thread Ryan Blue (JIRA)
Ryan Blue created SPARK-22345: - Summary: Sort-merge join generates incorrect code for CodegenFallback filter conditions Key: SPARK-22345 URL: https://issues.apache.org/jira/browse/SPARK-22345 Project: Spa

[jira] [Commented] (SPARK-22340) pyspark setJobGroup doesn't match java threads

2017-10-24 Thread Leif Walsh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217535#comment-16217535 ] Leif Walsh commented on SPARK-22340: This is less spooky than I initially thought, I

[jira] [Commented] (SPARK-22335) Union for DataSet uses column order instead of types for union

2017-10-24 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217532#comment-16217532 ] Dongjoon Hyun commented on SPARK-22335: --- To be clear, I have no objection on your i

[jira] [Commented] (SPARK-22335) Union for DataSet uses column order instead of types for union

2017-10-24 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217524#comment-16217524 ] Dongjoon Hyun commented on SPARK-22335: --- Hm. I see your point. What I meant was the

[jira] [Commented] (SPARK-22335) Union for DataSet uses column order instead of types for union

2017-10-24 Thread Carlos Bribiescas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217497#comment-16217497 ] Carlos Bribiescas commented on SPARK-22335: --- I'm not sure I understand what you

[jira] [Assigned] (SPARK-22291) Postgresql UUID[] to Cassandra: Conversion Error

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22291: Assignee: Apache Spark > Postgresql UUID[] to Cassandra: Conversion Error > --

[jira] [Assigned] (SPARK-22291) Postgresql UUID[] to Cassandra: Conversion Error

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22291: Assignee: (was: Apache Spark) > Postgresql UUID[] to Cassandra: Conversion Error > ---

[jira] [Commented] (SPARK-22291) Postgresql UUID[] to Cassandra: Conversion Error

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217462#comment-16217462 ] Apache Spark commented on SPARK-22291: -- User 'jmchung' has created a pull request fo

[jira] [Updated] (SPARK-22324) Upgrade Arrow to version 0.8.0

2017-10-24 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-22324: - Description: Arrow version 0.8.0 is slated for release in early November, but I'd like to start

[jira] [Commented] (SPARK-21043) Add unionByName API to Dataset

2017-10-24 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217415#comment-16217415 ] Reynold Xin commented on SPARK-21043: - Because some people expect union by position t

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-24 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217398#comment-16217398 ] Steve Loughran commented on SPARK-22240: no, spark 2.2 doesn't fix this. I have

[jira] [Comment Edited] (SPARK-12359) Add showString() to DataSet API.

2017-10-24 Thread Alexandre Dupriez (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217366#comment-16217366 ] Alexandre Dupriez edited comment on SPARK-12359 at 10/24/17 6:08 PM: --

[jira] [Comment Edited] (SPARK-12359) Add showString() to DataSet API.

2017-10-24 Thread Alexandre Dupriez (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217366#comment-16217366 ] Alexandre Dupriez edited comment on SPARK-12359 at 10/24/17 6:01 PM: --

[jira] [Assigned] (SPARK-22341) [2.3.0] cannot run Spark on Yarn when Yarn impersonation is turned off

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22341: Assignee: Apache Spark > [2.3.0] cannot run Spark on Yarn when Yarn impersonation is turne

[jira] [Assigned] (SPARK-22341) [2.3.0] cannot run Spark on Yarn when Yarn impersonation is turned off

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22341: Assignee: (was: Apache Spark) > [2.3.0] cannot run Spark on Yarn when Yarn impersonati

[jira] [Commented] (SPARK-12359) Add showString() to DataSet API.

2017-10-24 Thread Alexandre Dupriez (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217366#comment-16217366 ] Alexandre Dupriez commented on SPARK-12359: --- Looking at [this source|https://g

[jira] [Commented] (SPARK-22341) [2.3.0] cannot run Spark on Yarn when Yarn impersonation is turned off

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217368#comment-16217368 ] Apache Spark commented on SPARK-22341: -- User 'vanzin' has created a pull request for

[jira] [Commented] (SPARK-22335) Union for DataSet uses column order instead of types for union

2017-10-24 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217357#comment-16217357 ] Dongjoon Hyun commented on SPARK-22335: --- [~CBribiescas]. Is this issue about types?

[jira] [Commented] (SPARK-16367) Wheelhouse Support for PySpark

2017-10-24 Thread Dan Blanchard (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217354#comment-16217354 ] Dan Blanchard commented on SPARK-16367: --- Right, sorry. I just meant to point out th

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2017-10-24 Thread Ashwin Shankar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217333#comment-16217333 ] Ashwin Shankar commented on SPARK-18105: Hi [~davies] [~cloud_fan] We hit the sam

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-24 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217325#comment-16217325 ] Steve Loughran commented on SPARK-22240: I'm doing some testing with master & rea

[jira] [Comment Edited] (SPARK-16367) Wheelhouse Support for PySpark

2017-10-24 Thread Semet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217322#comment-16217322 ] Semet edited comment on SPARK-16367 at 10/24/17 5:29 PM: - Yes, I

[jira] [Commented] (SPARK-16367) Wheelhouse Support for PySpark

2017-10-24 Thread Semet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217322#comment-16217322 ] Semet commented on SPARK-16367: --- Yes, I don't use it because it is a feature of {{pip}}: {{

[jira] [Created] (SPARK-22344) Prevent R CMD check from using /tmp

2017-10-24 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-22344: - Summary: Prevent R CMD check from using /tmp Key: SPARK-22344 URL: https://issues.apache.org/jira/browse/SPARK-22344 Project: Spark Issue T

[jira] [Commented] (SPARK-15799) Release SparkR on CRAN

2017-10-24 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217310#comment-16217310 ] Shivaram Venkataraman commented on SPARK-15799: --- I created https://issues.a

[jira] [Commented] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

2017-10-24 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217299#comment-16217299 ] Juan Rodríguez Hortalá commented on SPARK-22148: Hi, I've been working

[jira] [Updated] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

2017-10-24 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Rodríguez Hortalá updated SPARK-22148: --- Attachment: SPARK-22148_WIP.diff > TaskSetManager.abortIfCompletelyBlackliste

[jira] [Commented] (SPARK-15799) Release SparkR on CRAN

2017-10-24 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217279#comment-16217279 ] Hossein Falaki commented on SPARK-15799: Is there a ticket to follow up on new po

[jira] [Commented] (SPARK-16367) Wheelhouse Support for PySpark

2017-10-24 Thread Dan Blanchard (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217253#comment-16217253 ] Dan Blanchard commented on SPARK-16367: --- You don't actually appear to use the {{whe

[jira] [Commented] (SPARK-22341) [2.3.0] cannot run Spark on Yarn when Yarn impersonation is turned off

2017-10-24 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217226#comment-16217226 ] Marcelo Vanzin commented on SPARK-22341: I was messing with this area recently, s

[jira] [Comment Edited] (SPARK-13587) Support virtualenv in PySpark

2017-10-24 Thread Semet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217221#comment-16217221 ] Semet edited comment on SPARK-13587 at 10/24/17 4:46 PM: - Hello.

[jira] [Commented] (SPARK-13587) Support virtualenv in PySpark

2017-10-24 Thread Semet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217221#comment-16217221 ] Semet commented on SPARK-13587: --- Hello. For me this solution is equivalent with my "Wheelho

[jira] [Commented] (SPARK-9686) Spark Thrift server doesn't return correct JDBC metadata

2017-10-24 Thread Andriy Kushnir (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217201#comment-16217201 ] Andriy Kushnir commented on SPARK-9686: --- [~rxin], I did a little research for this e

[jira] [Created] (SPARK-22343) Add support for publishing Spark metrics into Prometheus

2017-10-24 Thread Janos Matyas (JIRA)
Janos Matyas created SPARK-22343: Summary: Add support for publishing Spark metrics into Prometheus Key: SPARK-22343 URL: https://issues.apache.org/jira/browse/SPARK-22343 Project: Spark Issu

[jira] [Resolved] (SPARK-22301) Add rule to Optimizer for In with empty list of values

2017-10-24 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22301. - Resolution: Fixed Assignee: Marco Gaido Fix Version/s: 2.3.0 > Add rule to Optimizer for

[jira] [Commented] (SPARK-22277) Chi Square selector garbling Vector content.

2017-10-24 Thread Cheburakshu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217134#comment-16217134 ] Cheburakshu commented on SPARK-22277: - There are 2 problems I faced because of which

[jira] [Commented] (SPARK-13587) Support virtualenv in PySpark

2017-10-24 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217115#comment-16217115 ] Nicholas Chammas commented on SPARK-13587: -- To follow-up on my [earlier comment

[jira] [Comment Edited] (SPARK-22331) Make MLlib string params case-insensitive

2017-10-24 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216224#comment-16216224 ] Weichen Xu edited comment on SPARK-22331 at 10/24/17 2:19 PM: -

[jira] [Issue Comment Deleted] (SPARK-22342) refactor schedulerDriver registration

2017-10-24 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-22342: Comment: was deleted (was: @Arthur Rand fyi) > refactor schedulerDriver registrati

[jira] [Comment Edited] (SPARK-22342) refactor schedulerDriver registration

2017-10-24 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216959#comment-16216959 ] Stavros Kontopoulos edited comment on SPARK-22342 at 10/24/17 2:08 PM:

[jira] [Commented] (SPARK-22342) refactor schedulerDriver registration

2017-10-24 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216959#comment-16216959 ] Stavros Kontopoulos commented on SPARK-22342: - @Arthur Rand > refactor sche

[jira] [Updated] (SPARK-22342) refactor schedulerDriver registration

2017-10-24 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-22342: Description: This is an umbrella issue for working on: https://github.com/apache/sp

[jira] [Created] (SPARK-22342) refactor schedulerDriver registration

2017-10-24 Thread Stavros Kontopoulos (JIRA)
Stavros Kontopoulos created SPARK-22342: --- Summary: refactor schedulerDriver registration Key: SPARK-22342 URL: https://issues.apache.org/jira/browse/SPARK-22342 Project: Spark Issue Typ

[jira] [Commented] (SPARK-21043) Add unionByName API to Dataset

2017-10-24 Thread Carlos Bribiescas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216953#comment-16216953 ] Carlos Bribiescas commented on SPARK-21043: --- I really like this feature. Is th

[jira] [Commented] (SPARK-22335) Union for DataSet uses column order instead of types for union

2017-10-24 Thread Carlos Bribiescas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216950#comment-16216950 ] Carlos Bribiescas commented on SPARK-22335: --- I think if unionByName replaced un

[jira] [Assigned] (SPARK-22111) OnlineLDAOptimizer should filter out empty documents beforehand

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22111: Assignee: (was: Apache Spark) > OnlineLDAOptimizer should filter out empty documents b

[jira] [Assigned] (SPARK-22111) OnlineLDAOptimizer should filter out empty documents beforehand

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22111: Assignee: Apache Spark > OnlineLDAOptimizer should filter out empty documents beforehand

[jira] [Commented] (SPARK-22111) OnlineLDAOptimizer should filter out empty documents beforehand

2017-10-24 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216948#comment-16216948 ] Apache Spark commented on SPARK-22111: -- User 'akopich' has created a pull request fo

[jira] [Commented] (SPARK-22118) Should prevent change epoch in success stage while there is some running stage

2017-10-24 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-22118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216908#comment-16216908 ] Maciej Bryński commented on SPARK-22118: I think this problem is resolved by: SPA

[jira] [Comment Edited] (SPARK-22341) [2.3.0] cannot run Spark on Yarn when Yarn impersonation is turned off

2017-10-24 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-22341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216877#comment-16216877 ] Maciej Bryński edited comment on SPARK-22341 at 10/24/17 1:29 PM: -

[jira] [Updated] (SPARK-22340) pyspark setJobGroup doesn't match java threads

2017-10-24 Thread Leif Walsh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leif Walsh updated SPARK-22340: --- Description: With pyspark, {{sc.setJobGroup}}'s documentation says {quote} Assigns a group ID to all

[jira] [Comment Edited] (SPARK-22341) [2.3.0] cannot run Spark on Yarn when Yarn impersonation is turned off

2017-10-24 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-22341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216877#comment-16216877 ] Maciej Bryński edited comment on SPARK-22341 at 10/24/17 1:20 PM: -

[jira] [Commented] (SPARK-22341) [2.3.0] cannot run Spark on Yarn when Yarn impersonation is turned off

2017-10-24 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-22341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216877#comment-16216877 ] Maciej Bryński commented on SPARK-22341: Because Spark 2.2.0 is working perfectly

[jira] [Updated] (SPARK-22341) [2.3.0] cannot run Spark on Yarn when Yarn impersonation is turned off

2017-10-24 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-22341: -- Priority: Major (was: Blocker) > [2.3.0] cannot run Spark on Yarn when Yarn impersonation is turned of

[jira] [Commented] (SPARK-22341) [2.3.0] cannot run Spark on Yarn when Yarn impersonation is turned off

2017-10-24 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216868#comment-16216868 ] Sean Owen commented on SPARK-22341: --- Yes, but why is that a Spark problem? > [2.3.0] c

[jira] [Commented] (SPARK-16599) java.util.NoSuchElementException: None.get at at org.apache.spark.storage.BlockInfoManager.releaseAllLocksForTask(BlockInfoManager.scala:343)

2017-10-24 Thread Pranav Singhania (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216862#comment-16216862 ] Pranav Singhania commented on SPARK-16599: -- I've been temporarily able to avoid

[jira] [Created] (SPARK-22341) [2.3.0] cannot run Spark on Yarn when Yarn impersonation is turned off

2017-10-24 Thread JIRA
Maciej Bryński created SPARK-22341: -- Summary: [2.3.0] cannot run Spark on Yarn when Yarn impersonation is turned off Key: SPARK-22341 URL: https://issues.apache.org/jira/browse/SPARK-22341 Project: S

[jira] [Comment Edited] (SPARK-22277) Chi Square selector garbling Vector content.

2017-10-24 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216623#comment-16216623 ] Weichen Xu edited comment on SPARK-22277 at 10/24/17 12:45 PM:

[jira] [Updated] (SPARK-22118) Should prevent change epoch in success stage while there is some running stage

2017-10-24 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-22118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Bryński updated SPARK-22118: --- Priority: Critical (was: Major) > Should prevent change epoch in success stage while there i

  1   2   >