[jira] [Commented] (SPARK-22980) Wrong answer when using pandas_udf

[ https://issues.apache.org/jira/browse/SPARK-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315805#comment-16315805 ] Hyukjin Kwon commented on SPARK-22980: -- [~smilegator], May I ask why this was reopened and what I

[jira] [Updated] (SPARK-22989) sparkstreaming ui show 0 records when spark-streaming-kafka application restore from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-22989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoshijie updated SPARK-22989: --- Description: when a spark-streaming-kafka application restore from checkpoint , I find

[jira] [Created] (SPARK-22989) sparkstreaming ui show 0 records when spark-streaming-kafka application restore from checkpoint

zhaoshijie created SPARK-22989: -- Summary: sparkstreaming ui show 0 records when spark-streaming-kafka application restore from checkpoint Key: SPARK-22989 URL: https://issues.apache.org/jira/browse/SPARK-22989

[jira] [Updated] (SPARK-22988) Why does dataset's unpersist clear all the caches have the same logical plan?

[ https://issues.apache.org/jira/browse/SPARK-22988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang Cheng updated SPARK-22988: --- Description: When I do followings: dataset A = some dataset A.persist dataset B = A.doSomthing

[jira] [Reopened] (SPARK-22980) Wrong answer when using pandas_udf

[ https://issues.apache.org/jira/browse/SPARK-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reopened SPARK-22980: - > Wrong answer when using pandas_udf > -- > > Key:

[jira] [Updated] (SPARK-22988) Why does dataset's unpersist clear all the caches have the same logical plan?

[ https://issues.apache.org/jira/browse/SPARK-22988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang Cheng updated SPARK-22988: --- Description: When I do followings: dataset A = some dataset A.persist dataset B = A.doSomthing

[jira] [Created] (SPARK-22988) Why does dataset's unpersist clear all the caches have the same logical plan?

Wang Cheng created SPARK-22988: -- Summary: Why does dataset's unpersist clear all the caches have the same logical plan? Key: SPARK-22988 URL: https://issues.apache.org/jira/browse/SPARK-22988 Project:

[jira] [Resolved] (SPARK-22979) Avoid per-record type dispatch in Python data conversion (EvaluatePython.fromJava)

[ https://issues.apache.org/jira/browse/SPARK-22979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22979. - Resolution: Fixed Assignee: Hyukjin Kwon Fix Version/s: 2.3.0 > Avoid per-record

[jira] [Resolved] (SPARK-22566) Better error message for `_merge_type` in Pandas to Spark DF conversion

[ https://issues.apache.org/jira/browse/SPARK-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-22566. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19792

[jira] [Assigned] (SPARK-22566) Better error message for `_merge_type` in Pandas to Spark DF conversion

[ https://issues.apache.org/jira/browse/SPARK-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-22566: - Assignee: Guilherme Berger > Better error message for `_merge_type` in Pandas to Spark

[jira] [Commented] (SPARK-22987) UnsafeExternalSorter cases OOM when invoking `getIterator` function.

[ https://issues.apache.org/jira/browse/SPARK-22987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315657#comment-16315657 ] Apache Spark commented on SPARK-22987: -- User 'liutang123' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22987) UnsafeExternalSorter cases OOM when invoking `getIterator` function.

[ https://issues.apache.org/jira/browse/SPARK-22987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22987: Assignee: (was: Apache Spark) > UnsafeExternalSorter cases OOM when invoking

[jira] [Assigned] (SPARK-22987) UnsafeExternalSorter cases OOM when invoking `getIterator` function.

[ https://issues.apache.org/jira/browse/SPARK-22987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22987: Assignee: Apache Spark > UnsafeExternalSorter cases OOM when invoking `getIterator`

[jira] [Created] (SPARK-22987) UnsafeExternalSorter cases OOM when invoking `getIterator` function.

Lijia Liu created SPARK-22987: - Summary: UnsafeExternalSorter cases OOM when invoking `getIterator` function. Key: SPARK-22987 URL: https://issues.apache.org/jira/browse/SPARK-22987 Project: Spark

[jira] [Commented] (SPARK-22711) _pickle.PicklingError: args[0] from __newobj__ args has the wrong class from cloudpickle.py

[ https://issues.apache.org/jira/browse/SPARK-22711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315648#comment-16315648 ] Prateek commented on SPARK-22711: - @ [~hyukjin.kwon] Code updated. Make sure you have downloaded nltk,

[jira] [Updated] (SPARK-22711) _pickle.PicklingError: args[0] from __newobj__ args has the wrong class from cloudpickle.py

[ https://issues.apache.org/jira/browse/SPARK-22711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prateek updated SPARK-22711: Attachment: Jira_Spark_minimized_code.py updated missing code > _pickle.PicklingError: args[0] from

[jira] [Updated] (SPARK-22711) _pickle.PicklingError: args[0] from __newobj__ args has the wrong class from cloudpickle.py

[ https://issues.apache.org/jira/browse/SPARK-22711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prateek updated SPARK-22711: Attachment: (was: Jira_Spark_minimized_code.py) > _pickle.PicklingError: args[0] from __newobj__ args

[jira] [Resolved] (SPARK-22985) Fix argument escaping bug in from_utc_timestamp / to_utc_timestamp codegen

[ https://issues.apache.org/jira/browse/SPARK-22985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22985. - Resolution: Fixed Fix Version/s: 2.3.0 > Fix argument escaping bug in from_utc_timestamp /

[jira] [Commented] (SPARK-22986) Avoid instantiating multiple instances of broadcast variables

[ https://issues.apache.org/jira/browse/SPARK-22986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315615#comment-16315615 ] Apache Spark commented on SPARK-22986: -- User 'ho3rexqj' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22986) Avoid instantiating multiple instances of broadcast variables

[ https://issues.apache.org/jira/browse/SPARK-22986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22986: Assignee: Apache Spark > Avoid instantiating multiple instances of broadcast variables >

[jira] [Assigned] (SPARK-22986) Avoid instantiating multiple instances of broadcast variables

[ https://issues.apache.org/jira/browse/SPARK-22986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22986: Assignee: (was: Apache Spark) > Avoid instantiating multiple instances of broadcast

[jira] [Created] (SPARK-22986) Avoid instantiating multiple instances of broadcast variables

ho3rexqj created SPARK-22986: Summary: Avoid instantiating multiple instances of broadcast variables Key: SPARK-22986 URL: https://issues.apache.org/jira/browse/SPARK-22986 Project: Spark

[jira] [Updated] (SPARK-22968) java.lang.IllegalStateException: No current assignment for partition kssh-2

[ https://issues.apache.org/jira/browse/SPARK-22968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jepson updated SPARK-22968: --- Component/s: (was: Structured Streaming) Spark Core > java.lang.IllegalStateException:

[jira] [Commented] (SPARK-22968) java.lang.IllegalStateException: No current assignment for partition kssh-2

[ https://issues.apache.org/jira/browse/SPARK-22968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315552#comment-16315552 ] Jepson commented on SPARK-22968: [~srowen] Thanks for quick response. I turn up the parameter

[jira] [Commented] (SPARK-22967) VersionSuite failed on Windows caused by unescapeSQLString()

[ https://issues.apache.org/jira/browse/SPARK-22967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315547#comment-16315547 ] Hyukjin Kwon commented on SPARK-22967: -- Ah, I meant to fix the tests to use URI forms instead of

[jira] [Assigned] (SPARK-22985) Fix argument escaping bug in from_utc_timestamp / to_utc_timestamp codegen

[ https://issues.apache.org/jira/browse/SPARK-22985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22985: Assignee: Apache Spark (was: Josh Rosen) > Fix argument escaping bug in

[jira] [Assigned] (SPARK-22985) Fix argument escaping bug in from_utc_timestamp / to_utc_timestamp codegen

[ https://issues.apache.org/jira/browse/SPARK-22985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22985: Assignee: Josh Rosen (was: Apache Spark) > Fix argument escaping bug in

[jira] [Commented] (SPARK-22985) Fix argument escaping bug in from_utc_timestamp / to_utc_timestamp codegen

[ https://issues.apache.org/jira/browse/SPARK-22985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315530#comment-16315530 ] Apache Spark commented on SPARK-22985: -- User 'JoshRosen' has created a pull request for this issue:

[jira] [Created] (SPARK-22985) Fix argument escaping bug in from_utc_timestamp / to_utc_timestamp codegen

Josh Rosen created SPARK-22985: -- Summary: Fix argument escaping bug in from_utc_timestamp / to_utc_timestamp codegen Key: SPARK-22985 URL: https://issues.apache.org/jira/browse/SPARK-22985 Project:

[jira] [Assigned] (SPARK-22984) Fix incorrect bitmap copying and offset shifting in GenerateUnsafeRowJoiner

[ https://issues.apache.org/jira/browse/SPARK-22984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22984: Assignee: Josh Rosen (was: Apache Spark) > Fix incorrect bitmap copying and offset

[jira] [Commented] (SPARK-22984) Fix incorrect bitmap copying and offset shifting in GenerateUnsafeRowJoiner

[ https://issues.apache.org/jira/browse/SPARK-22984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315520#comment-16315520 ] Apache Spark commented on SPARK-22984: -- User 'JoshRosen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22984) Fix incorrect bitmap copying and offset shifting in GenerateUnsafeRowJoiner

[ https://issues.apache.org/jira/browse/SPARK-22984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22984: Assignee: Apache Spark (was: Josh Rosen) > Fix incorrect bitmap copying and offset

[jira] [Created] (SPARK-22984) Fix incorrect bitmap copying and offset shifting in GenerateUnsafeRowJoiner

Josh Rosen created SPARK-22984: -- Summary: Fix incorrect bitmap copying and offset shifting in GenerateUnsafeRowJoiner Key: SPARK-22984 URL: https://issues.apache.org/jira/browse/SPARK-22984 Project:

[jira] [Assigned] (SPARK-22983) Don't push filters beneath aggregates with empty grouping expressions

[ https://issues.apache.org/jira/browse/SPARK-22983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22983: Assignee: Apache Spark (was: Josh Rosen) > Don't push filters beneath aggregates with

[jira] [Assigned] (SPARK-22983) Don't push filters beneath aggregates with empty grouping expressions

[ https://issues.apache.org/jira/browse/SPARK-22983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22983: Assignee: Josh Rosen (was: Apache Spark) > Don't push filters beneath aggregates with

[jira] [Commented] (SPARK-22983) Don't push filters beneath aggregates with empty grouping expressions

[ https://issues.apache.org/jira/browse/SPARK-22983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315517#comment-16315517 ] Apache Spark commented on SPARK-22983: -- User 'JoshRosen' has created a pull request for this issue:

[jira] [Created] (SPARK-22983) Don't push filters beneath aggregates with empty grouping expressions

Josh Rosen created SPARK-22983: -- Summary: Don't push filters beneath aggregates with empty grouping expressions Key: SPARK-22983 URL: https://issues.apache.org/jira/browse/SPARK-22983 Project: Spark

[jira] [Assigned] (SPARK-22982) Remove unsafe asynchronous close() call from FileDownloadChannel

[ https://issues.apache.org/jira/browse/SPARK-22982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22982: Assignee: Apache Spark (was: Josh Rosen) > Remove unsafe asynchronous close() call from

[jira] [Assigned] (SPARK-22982) Remove unsafe asynchronous close() call from FileDownloadChannel

[ https://issues.apache.org/jira/browse/SPARK-22982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22982: Assignee: Josh Rosen (was: Apache Spark) > Remove unsafe asynchronous close() call from

[jira] [Commented] (SPARK-22982) Remove unsafe asynchronous close() call from FileDownloadChannel

[ https://issues.apache.org/jira/browse/SPARK-22982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315514#comment-16315514 ] Apache Spark commented on SPARK-22982: -- User 'JoshRosen' has created a pull request for this issue:

[jira] [Created] (SPARK-22982) Remove unsafe asynchronous close() call from FileDownloadChannel

Josh Rosen created SPARK-22982: -- Summary: Remove unsafe asynchronous close() call from FileDownloadChannel Key: SPARK-22982 URL: https://issues.apache.org/jira/browse/SPARK-22982 Project: Spark

[jira] [Resolved] (SPARK-22980) Wrong answer when using pandas_udf

[ https://issues.apache.org/jira/browse/SPARK-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22980. -- Resolution: Not A Problem Please reopen this if I misunderstood. I am taking an action quick

[jira] [Commented] (SPARK-22918) sbt test (spark - local) fail after upgrading to 2.2.1 with: java.security.AccessControlException: access denied org.apache.derby.security.SystemPermission( "engine",

[ https://issues.apache.org/jira/browse/SPARK-22918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315438#comment-16315438 ] Felix Cheung commented on SPARK-22918: -- [~sameerag] we might want to check this for 2.3.0 release >

[jira] [Commented] (SPARK-22632) Fix the behavior of timestamp values for R's DataFrame to respect session timezone

[ https://issues.apache.org/jira/browse/SPARK-22632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315436#comment-16315436 ] Felix Cheung commented on SPARK-22632: -- yes, first I'd agree we should generalize this to R & Python

[jira] [Commented] (SPARK-21727) Operating on an ArrayType in a SparkR DataFrame throws error

[ https://issues.apache.org/jira/browse/SPARK-21727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315434#comment-16315434 ] Felix Cheung commented on SPARK-21727: -- I think we should use is.atomic(object) ? > Operating on

[jira] [Assigned] (SPARK-22952) Deprecate stageAttemptId in favour of stageAttemptNumber

[ https://issues.apache.org/jira/browse/SPARK-22952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22952: Assignee: Apache Spark > Deprecate stageAttemptId in favour of stageAttemptNumber >

[jira] [Assigned] (SPARK-22952) Deprecate stageAttemptId in favour of stageAttemptNumber

[ https://issues.apache.org/jira/browse/SPARK-22952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22952: Assignee: (was: Apache Spark) > Deprecate stageAttemptId in favour of

[jira] [Commented] (SPARK-22952) Deprecate stageAttemptId in favour of stageAttemptNumber

[ https://issues.apache.org/jira/browse/SPARK-22952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315356#comment-16315356 ] Apache Spark commented on SPARK-22952: -- User 'advancedxy' has created a pull request for this issue:

[jira] [Commented] (SPARK-22954) ANALYZE TABLE fails with NoSuchTableException for temporary tables (but should have reported "not supported on views")

[ https://issues.apache.org/jira/browse/SPARK-22954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315336#comment-16315336 ] Suchith J N commented on SPARK-22954: - I have opened a pull request. > ANALYZE TABLE fails with

[jira] [Commented] (SPARK-22954) ANALYZE TABLE fails with NoSuchTableException for temporary tables (but should have reported "not supported on views")

[ https://issues.apache.org/jira/browse/SPARK-22954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315334#comment-16315334 ] Apache Spark commented on SPARK-22954: -- User 'suchithjn225' has created a pull request for this

[jira] [Assigned] (SPARK-22954) ANALYZE TABLE fails with NoSuchTableException for temporary tables (but should have reported "not supported on views")

[ https://issues.apache.org/jira/browse/SPARK-22954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22954: Assignee: (was: Apache Spark) > ANALYZE TABLE fails with NoSuchTableException for

[jira] [Assigned] (SPARK-22954) ANALYZE TABLE fails with NoSuchTableException for temporary tables (but should have reported "not supported on views")

[ https://issues.apache.org/jira/browse/SPARK-22954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22954: Assignee: Apache Spark > ANALYZE TABLE fails with NoSuchTableException for temporary

[jira] [Commented] (SPARK-22980) Wrong answer when using pandas_udf

[ https://issues.apache.org/jira/browse/SPARK-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315217#comment-16315217 ] Hyukjin Kwon commented on SPARK-22980: -- I think that's because we expect Pandas's Series in Scala