[jira] [Commented] (SPARK-24498) Add JDK compiler for runtime codegen

[ https://issues.apache.org/jira/browse/SPARK-24498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514696#comment-16514696 ] Takeshi Yamamuro commented on SPARK-24498: -- I'm also interested in this, so I'll look into

[jira] [Commented] (SPARK-23901) Data Masking Functions

[ https://issues.apache.org/jira/browse/SPARK-23901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514687#comment-16514687 ] Marco Gaido commented on SPARK-23901: - These functions can be used as any other function in Hive,

[jira] [Commented] (SPARK-23901) Data Masking Functions

[ https://issues.apache.org/jira/browse/SPARK-23901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514685#comment-16514685 ] Wenchen Fan commented on SPARK-23901: - According to the Hive JIRA, it's used in the Hive

[jira] [Comment Edited] (SPARK-24540) Support for multiple delimiter in Spark CSV read

[ https://issues.apache.org/jira/browse/SPARK-24540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514684#comment-16514684 ] Takeshi Yamamuro edited comment on SPARK-24540 at 6/16/18 5:28 AM: ---

[jira] [Comment Edited] (SPARK-24540) Support for multiple delimiter in Spark CSV read

[ https://issues.apache.org/jira/browse/SPARK-24540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514684#comment-16514684 ] Takeshi Yamamuro edited comment on SPARK-24540 at 6/16/18 5:28 AM: ---

[jira] [Commented] (SPARK-24540) Support for multiple delimiter in Spark CSV read

[ https://issues.apache.org/jira/browse/SPARK-24540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514684#comment-16514684 ] Takeshi Yamamuro commented on SPARK-24540: -- Probably, this is a restriction of univocity

[jira] [Assigned] (SPARK-24571) Support literals with values of the Char type

[ https://issues.apache.org/jira/browse/SPARK-24571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24571: Assignee: Apache Spark > Support literals with values of the Char type >

[jira] [Commented] (SPARK-24571) Support literals with values of the Char type

[ https://issues.apache.org/jira/browse/SPARK-24571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514669#comment-16514669 ] Apache Spark commented on SPARK-24571: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24571) Support literals with values of the Char type

[ https://issues.apache.org/jira/browse/SPARK-24571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24571: Assignee: (was: Apache Spark) > Support literals with values of the Char type >

[jira] [Commented] (SPARK-23901) Data Masking Functions

[ https://issues.apache.org/jira/browse/SPARK-23901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514667#comment-16514667 ] Reynold Xin commented on SPARK-23901: - Why are we adding 1200 lines of code for some functions that

[jira] [Updated] (SPARK-24571) Support literals with values of the Char type

[ https://issues.apache.org/jira/browse/SPARK-24571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-24571: --- Description: Currently, Spark doesn't support literals with the Char (java.lang.Character) type.

[jira] [Created] (SPARK-24571) Support literals with values of the Char type

Maxim Gekk created SPARK-24571: -- Summary: Support literals with values of the Char type Key: SPARK-24571 URL: https://issues.apache.org/jira/browse/SPARK-24571 Project: Spark Issue Type:

[jira] [Commented] (SPARK-24571) Support literals with values of the Char type

[ https://issues.apache.org/jira/browse/SPARK-24571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514665#comment-16514665 ] Maxim Gekk commented on SPARK-24571: I am working on the improvement. > Support literals with

[jira] [Updated] (SPARK-24216) Spark TypedAggregateExpression uses getSimpleName that is not safe in scala

[ https://issues.apache.org/jira/browse/SPARK-24216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24216: Fix Version/s: 2.3.2 > Spark TypedAggregateExpression uses getSimpleName that is not safe in

[jira] [Commented] (SPARK-24570) SparkSQL - show schemas/tables in dropdowns of SQL client tools (ie Squirrel SQL, DBVisualizer.etc)

[ https://issues.apache.org/jira/browse/SPARK-24570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514629#comment-16514629 ] Takeshi Yamamuro commented on SPARK-24570: -- Is this a Spark itself issue? > SparkSQL - show

[jira] [Created] (SPARK-24570) SparkSQL - show schemas/tables in dropdowns of SQL client tools (ie Squirrel SQL, DBVisualizer.etc)

t oo created SPARK-24570: Summary: SparkSQL - show schemas/tables in dropdowns of SQL client tools (ie Squirrel SQL, DBVisualizer.etc) Key: SPARK-24570 URL: https://issues.apache.org/jira/browse/SPARK-24570

[jira] [Updated] (SPARK-24570) SparkSQL - show schemas/tables in dropdowns of SQL client tools (ie Squirrel SQL, DBVisualizer.etc)

[ https://issues.apache.org/jira/browse/SPARK-24570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-24570: - Attachment: connect-to-sql-db-ssms-locate-table.png > SparkSQL - show schemas/tables in dropdowns of SQL client

[jira] [Comment Edited] (SPARK-21569) Internal Spark class needs to be kryo-registered

[ https://issues.apache.org/jira/browse/SPARK-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514529#comment-16514529 ] YongGang Cao edited comment on SPARK-21569 at 6/15/18 11:47 PM: seems

[jira] [Comment Edited] (SPARK-21569) Internal Spark class needs to be kryo-registered

[ https://issues.apache.org/jira/browse/SPARK-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514529#comment-16514529 ] YongGang Cao edited comment on SPARK-21569 at 6/15/18 11:45 PM: seems

[jira] [Commented] (SPARK-21569) Internal Spark class needs to be kryo-registered

[ https://issues.apache.org/jira/browse/SPARK-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514529#comment-16514529 ] YongGang Cao commented on SPARK-21569: -- seems this is not workaround-able, at least from java side. 

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514526#comment-16514526 ] Apache Spark commented on SPARK-24552: -- User 'vanzin' has created a pull request for this issue:

[jira] [Created] (SPARK-24569) Spark Aggregator with output type Option[Boolean] creates column of type Row

John Conwell created SPARK-24569: Summary: Spark Aggregator with output type Option[Boolean] creates column of type Row Key: SPARK-24569 URL: https://issues.apache.org/jira/browse/SPARK-24569

[jira] [Resolved] (SPARK-24452) long = int*int or long = int+int may cause overflow.

[ https://issues.apache.org/jira/browse/SPARK-24452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24452. - Resolution: Fixed Fix Version/s: 2.3.2 2.4.0 Issue resolved by pull

[jira] [Assigned] (SPARK-24452) long = int*int or long = int+int may cause overflow.

[ https://issues.apache.org/jira/browse/SPARK-24452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-24452: --- Assignee: Kazuaki Ishizaki > long = int*int or long = int+int may cause overflow. >

[jira] [Updated] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24552: -- Affects Version/s: 2.2.0 2.2.1 2.3.0

[jira] [Updated] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24552: -- Priority: Blocker (was: Critical) > Task attempt numbers are reused when stages are retried

[jira] [Resolved] (SPARK-24525) Provide an option to limit MemorySink memory usage

[ https://issues.apache.org/jira/browse/SPARK-24525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-24525. - Resolution: Fixed Fix Version/s: 2.4.0 Resolved by 

[jira] [Assigned] (SPARK-24525) Provide an option to limit MemorySink memory usage

[ https://issues.apache.org/jira/browse/SPARK-24525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-24525: --- Assignee: Mukul Murthy > Provide an option to limit MemorySink memory usage >

[jira] [Resolved] (SPARK-24396) Add Structured Streaming ForeachWriter for python

[ https://issues.apache.org/jira/browse/SPARK-24396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24396. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21477

[jira] [Created] (SPARK-24568) Code refactoring for DataType equalsXXX methods

Maryann Xue created SPARK-24568: --- Summary: Code refactoring for DataType equalsXXX methods Key: SPARK-24568 URL: https://issues.apache.org/jira/browse/SPARK-24568 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-23435) R tests should support latest testthat

[ https://issues.apache.org/jira/browse/SPARK-23435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514111#comment-16514111 ] Weiqiang Zhuang edited comment on SPARK-23435 at 6/15/18 5:14 PM: -- yes,

[jira] [Commented] (SPARK-23435) R tests should support latest testthat

[ https://issues.apache.org/jira/browse/SPARK-23435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514111#comment-16514111 ] Weiqiang Zhuang commented on SPARK-23435: - yes, quite similar   ``` sp <-

[jira] [Resolved] (SPARK-24490) Use WebUI.addStaticHandler in web UIs

[ https://issues.apache.org/jira/browse/SPARK-24490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24490. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21510

[jira] [Assigned] (SPARK-24490) Use WebUI.addStaticHandler in web UIs

[ https://issues.apache.org/jira/browse/SPARK-24490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24490: -- Assignee: Jacek Laskowski > Use WebUI.addStaticHandler in web UIs >

[jira] [Updated] (SPARK-24531) HiveExternalCatalogVersionsSuite failing due to missing 2.2.0 version

[ https://issues.apache.org/jira/browse/SPARK-24531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24531: --- Fix Version/s: 2.3.2 > HiveExternalCatalogVersionsSuite failing due to missing 2.2.0

[jira] [Updated] (SPARK-24531) HiveExternalCatalogVersionsSuite failing due to missing 2.2.0 version

[ https://issues.apache.org/jira/browse/SPARK-24531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24531: --- Fix Version/s: 2.2.2 > HiveExternalCatalogVersionsSuite failing due to missing 2.2.0

[jira] [Commented] (SPARK-24535) Fix java version parsing in SparkR

[ https://issues.apache.org/jira/browse/SPARK-24535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514031#comment-16514031 ] Hyukjin Kwon commented on SPARK-24535: -- When I tried it before, I remember I faced some issues ..

[jira] [Commented] (SPARK-24535) Fix java version parsing in SparkR

[ https://issues.apache.org/jira/browse/SPARK-24535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514020#comment-16514020 ] Shivaram Venkataraman commented on SPARK-24535: --- I was going to do it manually. If we do

[jira] [Commented] (SPARK-24535) Fix java version parsing in SparkR

[ https://issues.apache.org/jira/browse/SPARK-24535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514016#comment-16514016 ] Hyukjin Kwon commented on SPARK-24535: -- [~shivaram], how did you run this on Windows? I think we

[jira] [Commented] (SPARK-24535) Fix java version parsing in SparkR

[ https://issues.apache.org/jira/browse/SPARK-24535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514012#comment-16514012 ] Shivaram Venkataraman commented on SPARK-24535: --- I'm not sure if its only failing on

[jira] [Resolved] (SPARK-24476) java.net.SocketTimeoutException: Read timed out under jets3t while running the Spark Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-24476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bharath kumar avusherla resolved SPARK-24476. - Resolution: Fixed > java.net.SocketTimeoutException: Read timed out

[jira] [Assigned] (SPARK-24534) Add a way to bypass entrypoint.sh script if no spark cmd is passed

[ https://issues.apache.org/jira/browse/SPARK-24534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24534: Assignee: Apache Spark > Add a way to bypass entrypoint.sh script if no spark cmd is

[jira] [Assigned] (SPARK-24534) Add a way to bypass entrypoint.sh script if no spark cmd is passed

[ https://issues.apache.org/jira/browse/SPARK-24534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24534: Assignee: (was: Apache Spark) > Add a way to bypass entrypoint.sh script if no spark

[jira] [Commented] (SPARK-24534) Add a way to bypass entrypoint.sh script if no spark cmd is passed

[ https://issues.apache.org/jira/browse/SPARK-24534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513945#comment-16513945 ] Apache Spark commented on SPARK-24534: -- User 'rimolive' has created a pull request for this issue:

[jira] [Commented] (SPARK-24476) java.net.SocketTimeoutException: Read timed out under jets3t while running the Spark Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-24476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513933#comment-16513933 ] Steve Loughran commented on SPARK-24476: * Use S3A, as S3n is unsupported and deleted from the

[jira] [Commented] (SPARK-22918) sbt test (spark - local) fail after upgrading to 2.2.1 with: java.security.AccessControlException: access denied org.apache.derby.security.SystemPermission( "engine",

[ https://issues.apache.org/jira/browse/SPARK-22918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513897#comment-16513897 ] Mihaly Toth commented on SPARK-22918: - I managed to reproduce the problem in a unit test. When using

[jira] [Assigned] (SPARK-21743) top-most limit should not cause memory leak

[ https://issues.apache.org/jira/browse/SPARK-21743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21743: Assignee: Apache Spark (was: Wenchen Fan) > top-most limit should not cause memory leak

[jira] [Assigned] (SPARK-21743) top-most limit should not cause memory leak

[ https://issues.apache.org/jira/browse/SPARK-21743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21743: Assignee: Wenchen Fan (was: Apache Spark) > top-most limit should not cause memory leak

[jira] [Reopened] (SPARK-21743) top-most limit should not cause memory leak

[ https://issues.apache.org/jira/browse/SPARK-21743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell reopened SPARK-21743: --- Reopening issue, this is causing a regression in the CSV reader. > top-most limit

[jira] [Created] (SPARK-24567) nodeBlacklist does not get updated if a spark executor fails to launch on a mesos node

Igor Berman created SPARK-24567: --- Summary: nodeBlacklist does not get updated if a spark executor fails to launch on a mesos node Key: SPARK-24567 URL: https://issues.apache.org/jira/browse/SPARK-24567

[jira] [Comment Edited] (SPARK-24465) LSHModel should support Structured Streaming for transform

[ https://issues.apache.org/jira/browse/SPARK-24465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513649#comment-16513649 ] Liang-Chi Hsieh edited comment on SPARK-24465 at 6/15/18 10:32 AM: ---

[jira] [Commented] (SPARK-24465) LSHModel should support Structured Streaming for transform

[ https://issues.apache.org/jira/browse/SPARK-24465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513649#comment-16513649 ] Liang-Chi Hsieh commented on SPARK-24465: - I'm not sure if SPARK-12878 is a real issue. Seems

[jira] [Commented] (SPARK-12878) Dataframe fails with nested User Defined Types

[ https://issues.apache.org/jira/browse/SPARK-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513647#comment-16513647 ] Liang-Chi Hsieh commented on SPARK-12878: - Is this a real issue? Seems to me that you can't

[jira] [Commented] (SPARK-19498) Discussion: Making MLlib APIs extensible for 3rd party libraries

[ https://issues.apache.org/jira/browse/SPARK-19498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513642#comment-16513642 ] Lucas Partridge commented on SPARK-19498: - How would you prefer people provide their inputs on

[jira] [Updated] (SPARK-24548) JavaPairRDD to Dataset in SPARK generates ambiguous results

[ https://issues.apache.org/jira/browse/SPARK-24548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-24548: Component/s: (was: Spark Core) > JavaPairRDD to Dataset in SPARK generates ambiguous

[jira] [Updated] (SPARK-24548) JavaPairRDD to Dataset in SPARK generates ambiguous results

[ https://issues.apache.org/jira/browse/SPARK-24548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-24548: Component/s: SQL > JavaPairRDD to Dataset in SPARK generates ambiguous results >

[jira] [Assigned] (SPARK-24548) JavaPairRDD to Dataset in SPARK generates ambiguous results

[ https://issues.apache.org/jira/browse/SPARK-24548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24548: Assignee: Apache Spark > JavaPairRDD to Dataset in SPARK generates ambiguous results >

[jira] [Commented] (SPARK-24548) JavaPairRDD to Dataset in SPARK generates ambiguous results

[ https://issues.apache.org/jira/browse/SPARK-24548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513571#comment-16513571 ] Apache Spark commented on SPARK-24548: -- User 'viirya' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24548) JavaPairRDD to Dataset in SPARK generates ambiguous results

[ https://issues.apache.org/jira/browse/SPARK-24548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24548: Assignee: (was: Apache Spark) > JavaPairRDD to Dataset in SPARK generates ambiguous

[jira] [Commented] (SPARK-23435) R tests should support latest testthat

[ https://issues.apache.org/jira/browse/SPARK-23435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513416#comment-16513416 ] Felix Cheung commented on SPARK-23435: -- sorry, I did try but couldn't get it to work, but something