[jira] [Resolved] (SPARK-20035) Spark 2.0.2 writes empty file if no record is in the dataset

[ https://issues.apache.org/jira/browse/SPARK-20035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-20035. -- Resolution: Duplicate I am pretty sure that this is a duplicate of SPARK-15473. Let me resolve

[jira] [Commented] (SPARK-15473) CSV fails to write and read back empty dataframe

[ https://issues.apache.org/jira/browse/SPARK-15473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942609#comment-15942609 ] Hyukjin Kwon commented on SPARK-15473: -- I am not working on this. Please take over this if anyone is

[jira] [Assigned] (SPARK-20105) Add tests for checkType and type string in structField in R

[ https://issues.apache.org/jira/browse/SPARK-20105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20105: Assignee: Apache Spark > Add tests for checkType and type string in structField in R >

[jira] [Assigned] (SPARK-20105) Add tests for checkType and type string in structField in R

[ https://issues.apache.org/jira/browse/SPARK-20105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20105: Assignee: (was: Apache Spark) > Add tests for checkType and type string in

[jira] [Commented] (SPARK-20105) Add tests for checkType and type string in structField in R

[ https://issues.apache.org/jira/browse/SPARK-20105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942587#comment-15942587 ] Apache Spark commented on SPARK-20105: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-20104) Don't estimate IsNull or IsNotNull predicates for non-leaf node

[ https://issues.apache.org/jira/browse/SPARK-20104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20104: Assignee: (was: Apache Spark) > Don't estimate IsNull or IsNotNull predicates for

[jira] [Commented] (SPARK-20104) Don't estimate IsNull or IsNotNull predicates for non-leaf node

[ https://issues.apache.org/jira/browse/SPARK-20104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942583#comment-15942583 ] Apache Spark commented on SPARK-20104: -- User 'wzhfy' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20104) Don't estimate IsNull or IsNotNull predicates for non-leaf node

[ https://issues.apache.org/jira/browse/SPARK-20104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20104: Assignee: Apache Spark > Don't estimate IsNull or IsNotNull predicates for non-leaf node

[jira] [Created] (SPARK-20105) Add tests for checkType and type string in structField in R

Hyukjin Kwon created SPARK-20105: Summary: Add tests for checkType and type string in structField in R Key: SPARK-20105 URL: https://issues.apache.org/jira/browse/SPARK-20105 Project: Spark

[jira] [Created] (SPARK-20104) Don't estimate IsNull or IsNotNull predicates for non-leaf node

Zhenhua Wang created SPARK-20104: Summary: Don't estimate IsNull or IsNotNull predicates for non-leaf node Key: SPARK-20104 URL: https://issues.apache.org/jira/browse/SPARK-20104 Project: Spark

[jira] [Created] (SPARK-20103) Spark structured steaming from kafka - last message processed again after resume from checkpoint

Rajesh Mutha created SPARK-20103: Summary: Spark structured steaming from kafka - last message processed again after resume from checkpoint Key: SPARK-20103 URL: https://issues.apache.org/jira/browse/SPARK-20103

[jira] [Commented] (SPARK-6567) Large linear model parallelism via a join and reduceByKey

[ https://issues.apache.org/jira/browse/SPARK-6567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942498#comment-15942498 ] Joseph K. Bradley commented on SPARK-6567: -- Linking [SPARK-10078], which tracks adding

[jira] [Resolved] (SPARK-19281) spark.ml Python API for FPGrowth

[ https://issues.apache.org/jira/browse/SPARK-19281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-19281. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17218

[jira] [Assigned] (SPARK-19281) spark.ml Python API for FPGrowth

[ https://issues.apache.org/jira/browse/SPARK-19281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-19281: - Assignee: Maciej Szymkiewicz > spark.ml Python API for FPGrowth >

[jira] [Updated] (SPARK-19281) spark.ml Python API for FPGrowth

[ https://issues.apache.org/jira/browse/SPARK-19281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-19281: -- Shepherd: Joseph K. Bradley (was: Nick Pentreath) > spark.ml Python API for FPGrowth

[jira] [Commented] (SPARK-20072) Clarify ALS-WR documentation

[ https://issues.apache.org/jira/browse/SPARK-20072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942473#comment-15942473 ] chris snow commented on SPARK-20072: Will do. Thanks Sean. > Clarify ALS-WR documentation >

[jira] [Commented] (SPARK-20102) Fix two minor build script issues blocking 2.1.1 RC + master snapshot builds

[ https://issues.apache.org/jira/browse/SPARK-20102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942454#comment-15942454 ] Apache Spark commented on SPARK-20102: -- User 'JoshRosen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20102) Fix two minor build script issues blocking 2.1.1 RC + master snapshot builds

[ https://issues.apache.org/jira/browse/SPARK-20102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20102: Assignee: Apache Spark (was: Josh Rosen) > Fix two minor build script issues blocking

[jira] [Assigned] (SPARK-20102) Fix two minor build script issues blocking 2.1.1 RC + master snapshot builds

[ https://issues.apache.org/jira/browse/SPARK-20102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20102: Assignee: Josh Rosen (was: Apache Spark) > Fix two minor build script issues blocking

[jira] [Created] (SPARK-20102) Fix two minor build script issues blocking 2.1.1 RC + master snapshot builds

Josh Rosen created SPARK-20102: -- Summary: Fix two minor build script issues blocking 2.1.1 RC + master snapshot builds Key: SPARK-20102 URL: https://issues.apache.org/jira/browse/SPARK-20102 Project:

[jira] [Assigned] (SPARK-20086) issue with pyspark 2.1.0 window function

[ https://issues.apache.org/jira/browse/SPARK-20086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell reassigned SPARK-20086: - Assignee: Herman van Hovell > issue with pyspark 2.1.0 window function >

[jira] [Resolved] (SPARK-20086) issue with pyspark 2.1.0 window function

[ https://issues.apache.org/jira/browse/SPARK-20086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-20086. --- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > issue with

[jira] [Commented] (SPARK-19476) Running threads in Spark DataFrame foreachPartition() causes NullPointerException

[ https://issues.apache.org/jira/browse/SPARK-19476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942431#comment-15942431 ] Gal Topper commented on SPARK-19476: The example in the description does indeed fully materialize the

[jira] [Updated] (SPARK-19476) Running threads in Spark DataFrame foreachPartition() causes NullPointerException

[ https://issues.apache.org/jira/browse/SPARK-19476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-19476: -- Priority: Minor (was: Major) Issue Type: Improvement (was: Bug) You certainly access the

[jira] [Commented] (SPARK-19476) Running threads in Spark DataFrame foreachPartition() causes NullPointerException

[ https://issues.apache.org/jira/browse/SPARK-19476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942419#comment-15942419 ] Gal Topper commented on SPARK-19476: I'm pretty sure we're talking about different things. My code

[jira] [Comment Edited] (SPARK-14922) Alter Table Drop Partition Using Predicate-based Partition Spec

[ https://issues.apache.org/jira/browse/SPARK-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942413#comment-15942413 ] Dongjoon Hyun edited comment on SPARK-14922 at 3/26/17 7:16 PM:

[jira] [Closed] (SPARK-17732) ALTER TABLE DROP PARTITION should support comparators

[ https://issues.apache.org/jira/browse/SPARK-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-17732. - Resolution: Duplicate > ALTER TABLE DROP PARTITION should support comparators >

[jira] [Reopened] (SPARK-14922) Alter Table Drop Partition Using Predicate-based Partition Spec

[ https://issues.apache.org/jira/browse/SPARK-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-14922: --- [~smilegator], [~hyukjin.kwon]. I will reopen this issue instead of SPARK-17732. This issue

[jira] [Commented] (SPARK-19476) Running threads in Spark DataFrame foreachPartition() causes NullPointerException

[ https://issues.apache.org/jira/browse/SPARK-19476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942405#comment-15942405 ] Sean Owen commented on SPARK-19476: --- You don't need an executor per I/O call, you need one slot. I

[jira] [Resolved] (SPARK-19941) Spark should not schedule tasks on executors on decommissioning YARN nodes

[ https://issues.apache.org/jira/browse/SPARK-19941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19941. --- Resolution: Won't Fix > Spark should not schedule tasks on executors on decommissioning YARN nodes >

[jira] [Resolved] (SPARK-19977) Scheduler Delay (in UI Advanced Metrics) for a task gradually increases from 5 ms to 30 seconds in Spark Streaming application

[ https://issues.apache.org/jira/browse/SPARK-19977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19977. --- Resolution: Not A Problem > Scheduler Delay (in UI Advanced Metrics) for a task gradually increases

[jira] [Commented] (SPARK-20037) impossible to set kafka offsets using kafka 0.10 and spark 2.0.0

[ https://issues.apache.org/jira/browse/SPARK-20037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942399#comment-15942399 ] Sean Owen commented on SPARK-20037: --- I'm not sure this coherently describes the problem in a way that's

[jira] [Commented] (SPARK-20101) Use OffHeapColumnVector when "spark.memory.offHeap.enabled" is set to "true"

[ https://issues.apache.org/jira/browse/SPARK-20101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942384#comment-15942384 ] Apache Spark commented on SPARK-20101: -- User 'kiszk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20101) Use OffHeapColumnVector when "spark.memory.offHeap.enabled" is set to "true"

[ https://issues.apache.org/jira/browse/SPARK-20101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20101: Assignee: Apache Spark > Use OffHeapColumnVector when "spark.memory.offHeap.enabled" is

[jira] [Assigned] (SPARK-20101) Use OffHeapColumnVector when "spark.memory.offHeap.enabled" is set to "true"

[ https://issues.apache.org/jira/browse/SPARK-20101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20101: Assignee: (was: Apache Spark) > Use OffHeapColumnVector when

[jira] [Commented] (SPARK-20098) DataType's typeName method returns with 'StructF' in case of StructField

[ https://issues.apache.org/jira/browse/SPARK-20098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942382#comment-15942382 ] Apache Spark commented on SPARK-20098: -- User 'szalai1' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20098) DataType's typeName method returns with 'StructF' in case of StructField

[ https://issues.apache.org/jira/browse/SPARK-20098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20098: Assignee: (was: Apache Spark) > DataType's typeName method returns with 'StructF' in

[jira] [Assigned] (SPARK-20098) DataType's typeName method returns with 'StructF' in case of StructField

[ https://issues.apache.org/jira/browse/SPARK-20098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20098: Assignee: Apache Spark > DataType's typeName method returns with 'StructF' in case of

[jira] [Commented] (SPARK-19476) Running threads in Spark DataFrame foreachPartition() causes NullPointerException

[ https://issues.apache.org/jira/browse/SPARK-19476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942380#comment-15942380 ] Gal Topper commented on SPARK-19476: BTW, my workaround was quite painful: I created a

[jira] [Comment Edited] (SPARK-19476) Running threads in Spark DataFrame foreachPartition() causes NullPointerException

[ https://issues.apache.org/jira/browse/SPARK-19476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942326#comment-15942326 ] Gal Topper edited comment on SPARK-19476 at 3/26/17 6:08 PM: - In our case, we

[jira] [Created] (SPARK-20101) Use OffHeapColumnVector when "spark.memory.offHeap.enabled" is set to "true"

Kazuaki Ishizaki created SPARK-20101: Summary: Use OffHeapColumnVector when "spark.memory.offHeap.enabled" is set to "true" Key: SPARK-20101 URL: https://issues.apache.org/jira/browse/SPARK-20101

[jira] [Commented] (SPARK-19476) Running threads in Spark DataFrame foreachPartition() causes NullPointerException

[ https://issues.apache.org/jira/browse/SPARK-19476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942326#comment-15942326 ] Gal Topper commented on SPARK-19476: In our case, we use Akka Streams to make many parallel, async

[jira] [Comment Edited] (SPARK-14726) Support for sampling when inferring schema in CSV data source

[ https://issues.apache.org/jira/browse/SPARK-14726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247378#comment-15247378 ] Hyukjin Kwon edited comment on SPARK-14726 at 3/26/17 2:09 PM: --- This is

[jira] [Commented] (SPARK-14057) sql time stamps do not respect time zones

[ https://issues.apache.org/jira/browse/SPARK-14057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942285#comment-15942285 ] Hyukjin Kwon commented on SPARK-14057: -- [~vsparmar], would {{spark.sql.session.timeZone}} solve this

[jira] [Resolved] (SPARK-13169) CROSS JOIN slow or fails on tiny table

[ https://issues.apache.org/jira/browse/SPARK-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-13169. -- Resolution: Cannot Reproduce I am resolving this. I can't reproduce this against the current

[jira] [Comment Edited] (SPARK-11784) Support Timestamp filter pushdown in Parquet datasource

[ https://issues.apache.org/jira/browse/SPARK-11784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557940#comment-15557940 ] Hyukjin Kwon edited comment on SPARK-11784 at 3/26/17 1:42 PM: --- Could you

[jira] [Updated] (SPARK-20025) Driver fail over will not work, if SPARK_LOCAL* env is set.

[ https://issues.apache.org/jira/browse/SPARK-20025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-20025: -- Target Version/s: 2.2.0 (was: 2.1.1, 2.2.0) > Driver fail over will not work, if SPARK_LOCAL* env is

[jira] [Updated] (SPARK-19570) Allow to disable hive in pyspark shell

[ https://issues.apache.org/jira/browse/SPARK-19570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-19570: -- Target Version/s: (was: 2.1.1) > Allow to disable hive in pyspark shell >

[jira] [Commented] (SPARK-20100) Consolidate SessionState construction

[ https://issues.apache.org/jira/browse/SPARK-20100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1594#comment-1594 ] Apache Spark commented on SPARK-20100: -- User 'hvanhovell' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20100) Consolidate SessionState construction

[ https://issues.apache.org/jira/browse/SPARK-20100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20100: Assignee: Herman van Hovell (was: Apache Spark) > Consolidate SessionState construction

[jira] [Assigned] (SPARK-20100) Consolidate SessionState construction

[ https://issues.apache.org/jira/browse/SPARK-20100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20100: Assignee: Apache Spark (was: Herman van Hovell) > Consolidate SessionState construction

[jira] [Created] (SPARK-20100) Consolidate SessionState construction

Herman van Hovell created SPARK-20100: - Summary: Consolidate SessionState construction Key: SPARK-20100 URL: https://issues.apache.org/jira/browse/SPARK-20100 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-20086) issue with pyspark 2.1.0 window function

[ https://issues.apache.org/jira/browse/SPARK-20086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20086: Assignee: (was: Apache Spark) > issue with pyspark 2.1.0 window function >

[jira] [Assigned] (SPARK-20086) issue with pyspark 2.1.0 window function

[ https://issues.apache.org/jira/browse/SPARK-20086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20086: Assignee: Apache Spark > issue with pyspark 2.1.0 window function >

[jira] [Commented] (SPARK-20086) issue with pyspark 2.1.0 window function

[ https://issues.apache.org/jira/browse/SPARK-20086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942190#comment-15942190 ] Apache Spark commented on SPARK-20086: -- User 'hvanhovell' has created a pull request for this issue:

[jira] [Resolved] (SPARK-20046) Facilitate loop optimizations in a JIT compiler regarding sqlContext.read.parquet()

[ https://issues.apache.org/jira/browse/SPARK-20046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-20046. --- Resolution: Fixed Assignee: Kazuaki Ishizaki Fix Version/s: 2.2.0 >

[jira] [Resolved] (SPARK-20092) Trigger AppVeyor R tests for changes in Scala code related with R API

[ https://issues.apache.org/jira/browse/SPARK-20092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-20092. -- Resolution: Fixed Assignee: Hyukjin Kwon Fix Version/s: 2.2.0