[jira] [Assigned] (SPARK-17636) Parquet filter push down doesn't handle struct fields

2018-09-25 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-17636: --- Assignee: DB Tsai > Parquet filter push down doesn't handle struct fields >

[jira] [Updated] (SPARK-17636) Parquet filter push down doesn't handle struct fields

2018-09-25 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-17636: Fix Version/s: 2.5.0 > Parquet filter push down doesn't handle struct fields >

[jira] [Resolved] (SPARK-25406) Incorrect usage of withSQLConf method in Parquet schema pruning test suite masks failing tests

2018-09-13 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-25406. - Resolution: Fixed Fix Version/s: 2.4.0 3.0.0 Issue resolved by pull request

[jira] [Assigned] (SPARK-25406) Incorrect usage of withSQLConf method in Parquet schema pruning test suite masks failing tests

2018-09-13 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-25406: --- Assignee: Michael Allman > Incorrect usage of withSQLConf method in Parquet schema pruning test

[jira] [Resolved] (SPARK-25363) Schema pruning doesn't work if nested column is used in where clause

2018-09-12 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-25363. - Resolution: Fixed Fix Version/s: 2.4.0 3.0.0 Issue resolved by pull request

[jira] [Assigned] (SPARK-25363) Schema pruning doesn't work if nested column is used in where clause

2018-09-06 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-25363: --- Assignee: Liang-Chi Hsieh > Schema pruning doesn't work if nested column is used in where clause >

[jira] [Resolved] (SPARK-24411) Adding native Java tests for `isInCollection`

2018-08-30 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-24411. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22253

[jira] [Resolved] (SPARK-25235) Merge the REPL code in Scala 2.11 and 2.12 branches

2018-08-28 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-25235. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22246

[jira] [Assigned] (SPARK-25235) Merge the REPL code in Scala 2.11 and 2.12 branches

2018-08-27 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-25235: --- Assignee: DB Tsai > Merge the REPL code in Scala 2.11 and 2.12 branches >

[jira] [Created] (SPARK-25235) Merge the REPL code in Scala 2.11 and 2.12 branches

2018-08-24 Thread DB Tsai (JIRA)
DB Tsai created SPARK-25235: --- Summary: Merge the REPL code in Scala 2.11 and 2.12 branches Key: SPARK-25235 URL: https://issues.apache.org/jira/browse/SPARK-25235 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-24785) Making sure REPL prints Spark UI info and then Welcome message

2018-08-22 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24785: --- Assignee: DB Tsai > Making sure REPL prints Spark UI info and then Welcome message >

[jira] [Resolved] (SPARK-24785) Making sure REPL prints Spark UI info and then Welcome message

2018-08-22 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-24785. - Resolution: Fixed > Making sure REPL prints Spark UI info and then Welcome message >

[jira] [Commented] (SPARK-25050) Handle more than two types in avro union types when writing avro files

2018-08-21 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588068#comment-16588068 ] DB Tsai commented on SPARK-25050: - We handle more than two types in avro when reading into spark, but

[jira] [Updated] (SPARK-25050) Handle more than two types in avro union types when writing avro files

2018-08-21 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-25050: Summary: Handle more than two types in avro union types when writing avro files (was: Handle more than

[jira] [Assigned] (SPARK-23042) Use OneHotEncoderModel to encode labels in MultilayerPerceptronClassifier

2018-08-17 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-23042: --- Assignee: Liang-Chi Hsieh > Use OneHotEncoderModel to encode labels in

[jira] [Resolved] (SPARK-23042) Use OneHotEncoderModel to encode labels in MultilayerPerceptronClassifier

2018-08-17 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-23042. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20232

[jira] [Commented] (SPARK-25138) Spark Shell should show the Scala prompt after initialization is complete

2018-08-17 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584086#comment-16584086 ] DB Tsai commented on SPARK-25138: - [~smilegator] This regression is introduced by newer version of Scala

[jira] [Resolved] (SPARK-25115) Eliminate extra memory copy done when a ByteBuf is used that is backed by > 1 ByteBuffer.

2018-08-14 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-25115. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22105

[jira] [Assigned] (SPARK-25115) Eliminate extra memory copy done when a ByteBuf is used that is backed by > 1 ByteBuffer.

2018-08-14 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-25115: --- Assignee: Norman Maurer > Eliminate extra memory copy done when a ByteBuf is used that is

[jira] [Assigned] (SPARK-22974) CountVectorModel does not attach attributes to output column

2018-08-13 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-22974: --- Assignee: Liang-Chi Hsieh > CountVectorModel does not attach attributes to output column >

[jira] [Resolved] (SPARK-22974) CountVectorModel does not attach attributes to output column

2018-08-13 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-22974. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20313

[jira] [Resolved] (SPARK-25104) Validate user specified output schema

2018-08-13 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-25104. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22094

[jira] [Assigned] (SPARK-25104) Validate user specified output schema

2018-08-13 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-25104: --- Assignee: Gengliang Wang > Validate user specified output schema >

[jira] [Created] (SPARK-25050) Handle more than two types in avro union types

2018-08-07 Thread DB Tsai (JIRA)
DB Tsai created SPARK-25050: --- Summary: Handle more than two types in avro union types Key: SPARK-25050 URL: https://issues.apache.org/jira/browse/SPARK-25050 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-25049) Support custom schema in `to_avro`

2018-08-07 Thread DB Tsai (JIRA)
DB Tsai created SPARK-25049: --- Summary: Support custom schema in `to_avro` Key: SPARK-25049 URL: https://issues.apache.org/jira/browse/SPARK-25049 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-25018) Use `Co-Authored-By` git trailer in `merge_spark_pr.py`

2018-08-03 Thread DB Tsai (JIRA)
DB Tsai created SPARK-25018: --- Summary: Use `Co-Authored-By` git trailer in `merge_spark_pr.py` Key: SPARK-25018 URL: https://issues.apache.org/jira/browse/SPARK-25018 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-24993) Make Avro fast again

2018-08-03 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-24993. - Resolution: Fixed Fix Version/s: 2.4.0 > Make Avro fast again > > >

[jira] [Updated] (SPARK-24993) Make Avro fast again

2018-08-03 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-24993: Affects Version/s: (was: 2.3.0) 2.4.0 > Make Avro fast again >

[jira] [Resolved] (SPARK-25009) Standalone Cluster mode application submit is not working

2018-08-03 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-25009. - Resolution: Fixed Assignee: Devaraj K Fix Version/s: 2.4.0 > Standalone Cluster mode

[jira] [Created] (SPARK-24953) Prune a branch in `CaseWhen` if previously seen

2018-07-27 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24953: --- Summary: Prune a branch in `CaseWhen` if previously seen Key: SPARK-24953 URL: https://issues.apache.org/jira/browse/SPARK-24953 Project: Spark Issue Type:

[jira] [Created] (SPARK-24913) Make `AssertTrue` and `AssertNotNull` non-deterministic

2018-07-24 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24913: --- Summary: Make `AssertTrue` and `AssertNotNull` non-deterministic Key: SPARK-24913 URL: https://issues.apache.org/jira/browse/SPARK-24913 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-24908) [R] remove spaces to make lintr happy

2018-07-24 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-24908. - Resolution: Fixed Target Version/s: 2.4.0 > [R] remove spaces to make lintr happy >

[jira] [Created] (SPARK-24893) Remove the entire CaseWhen if all the outputs are semantic equivalence

2018-07-23 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24893: --- Summary: Remove the entire CaseWhen if all the outputs are semantic equivalence Key: SPARK-24893 URL: https://issues.apache.org/jira/browse/SPARK-24893 Project: Spark

[jira] [Assigned] (SPARK-24890) Short circuiting the `if` condition when `trueValue` and `falseValue` are the same

2018-07-23 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24890: --- Assignee: DB Tsai > Short circuiting the `if` condition when `trueValue` and `falseValue` are the

[jira] [Assigned] (SPARK-24892) Simplify `CaseWhen` to `If` when there is only one branch

2018-07-23 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24892: --- Assignee: DB Tsai > Simplify `CaseWhen` to `If` when there is only one branch >

[jira] [Created] (SPARK-24892) Simplify `CaseWhen` to `If` when there is only one branch

2018-07-23 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24892: --- Summary: Simplify `CaseWhen` to `If` when there is only one branch Key: SPARK-24892 URL: https://issues.apache.org/jira/browse/SPARK-24892 Project: Spark Issue Type:

[jira] [Created] (SPARK-24890) Short circuiting the `if` condition when `trueValue` and `falseValue` are the same

2018-07-23 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24890: --- Summary: Short circuiting the `if` condition when `trueValue` and `falseValue` are the same Key: SPARK-24890 URL: https://issues.apache.org/jira/browse/SPARK-24890 Project:

[jira] [Assigned] (SPARK-24855) Built-in AVRO support should support specified schema on write

2018-07-18 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24855: --- Assignee: Brian Lindblom > Built-in AVRO support should support specified schema on write >

[jira] [Resolved] (SPARK-24402) Optimize `In` expression when only one element in the collection or collection is empty

2018-07-17 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-24402. - Resolution: Resolved > Optimize `In` expression when only one element in the collection or >

[jira] [Assigned] (SPARK-24764) Add ServiceLoader implementation for SparkHadoopUtil

2018-07-11 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24764: --- Assignee: Shruti Gumma > Add ServiceLoader implementation for SparkHadoopUtil >

[jira] [Created] (SPARK-24785) Making sure REPL prints Spark UI info and then Welcome message

2018-07-11 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24785: --- Summary: Making sure REPL prints Spark UI info and then Welcome message Key: SPARK-24785 URL: https://issues.apache.org/jira/browse/SPARK-24785 Project: Spark Issue

[jira] [Commented] (SPARK-24421) sun.misc.Unsafe in JDK9+

2018-07-03 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531705#comment-16531705 ] DB Tsai commented on SPARK-24421: - In JDK9+, `sun.misc.Unsafe` is private. We can either access it

[jira] [Resolved] (SPARK-24420) Upgrade ASM to 6.x to support JDK9+

2018-07-03 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-24420. - Resolution: Fixed Fix Version/s: 2.4.0 > Upgrade ASM to 6.x to support JDK9+ >

[jira] [Resolved] (SPARK-24412) Adding docs about automagical type casting in `isin` and `isInCollection` APIs

2018-06-08 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-24412. - Resolution: Fixed > Adding docs about automagical type casting in `isin` and `isInCollection` APIs >

[jira] [Assigned] (SPARK-24412) Adding docs about automagical type casting in `isin` and `isInCollection` APIs

2018-06-08 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24412: --- Assignee: thrvskn > Adding docs about automagical type casting in `isin` and `isInCollection` APIs

[jira] [Updated] (SPARK-24418) Upgrade to Scala 2.11.12

2018-06-05 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-24418: Description: Scala 2.11.12+ will support JDK9+. However, this is not going to be a simple version bump.

[jira] [Updated] (SPARK-24417) Build and Run Spark on JDK9+

2018-05-31 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-24417: Fix Version/s: (was: 2.4.0) > Build and Run Spark on JDK9+ > > >

[jira] [Updated] (SPARK-24417) Build and Run Spark on JDK9+

2018-05-30 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-24417: Description: This is an umbrella JIRA for Apache Spark to support Java 9+ As Java 8 is going away soon,

[jira] [Resolved] (SPARK-24419) Upgrade SBT to 0.13.17 with Scala 2.10.7

2018-05-30 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-24419. - Resolution: Fixed > Upgrade SBT to 0.13.17 with Scala 2.10.7 >

[jira] [Updated] (SPARK-24417) Build and Run Spark on JDK9+

2018-05-29 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-24417: Description: This is an umbrella JIRA for Apache Spark to support JDK9+ As Java 8 is going way soon,

[jira] [Created] (SPARK-24422) Add JDK9+ in our Jenkins' build servers

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24422: --- Summary: Add JDK9+ in our Jenkins' build servers Key: SPARK-24422 URL: https://issues.apache.org/jira/browse/SPARK-24422 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-24419) Upgrade SBT to 0.13.17 with Scala 2.10.7

2018-05-29 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24419: --- Assignee: DB Tsai > Upgrade SBT to 0.13.17 with Scala 2.10.7 >

[jira] [Assigned] (SPARK-24420) Upgrade ASM to 6.x to support JDK9+

2018-05-29 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24420: --- Assignee: DB Tsai > Upgrade ASM to 6.x to support JDK9+ > --- > >

[jira] [Assigned] (SPARK-24418) Upgrade to Scala 2.11.12

2018-05-29 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24418: --- Assignee: DB Tsai > Upgrade to Scala 2.11.12 > > > Key:

[jira] [Created] (SPARK-24421) sun.misc.Unsafe in JDK9+

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24421: --- Summary: sun.misc.Unsafe in JDK9+ Key: SPARK-24421 URL: https://issues.apache.org/jira/browse/SPARK-24421 Project: Spark Issue Type: Sub-task Components:

[jira] [Created] (SPARK-24420) Upgrade ASM to 6.x to support JDK9+

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24420: --- Summary: Upgrade ASM to 6.x to support JDK9+ Key: SPARK-24420 URL: https://issues.apache.org/jira/browse/SPARK-24420 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-24419) Upgrade SBT to 0.13.17 with Scala 2.10.7

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24419: --- Summary: Upgrade SBT to 0.13.17 with Scala 2.10.7 Key: SPARK-24419 URL: https://issues.apache.org/jira/browse/SPARK-24419 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-24418) Upgrade to Scala 2.11.12

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24418: --- Summary: Upgrade to Scala 2.11.12 Key: SPARK-24418 URL: https://issues.apache.org/jira/browse/SPARK-24418 Project: Spark Issue Type: Sub-task Components:

[jira] [Created] (SPARK-24417) Build and Run Spark on JDK9+

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24417: --- Summary: Build and Run Spark on JDK9+ Key: SPARK-24417 URL: https://issues.apache.org/jira/browse/SPARK-24417 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-24412) Adding docs about automagical type casting in `isin` and `isInCollection` APIs

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24412: --- Summary: Adding docs about automagical type casting in `isin` and `isInCollection` APIs Key: SPARK-24412 URL: https://issues.apache.org/jira/browse/SPARK-24412 Project: Spark

[jira] [Created] (SPARK-24411) Adding native Java tests for `isInCollection`

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24411: --- Summary: Adding native Java tests for `isInCollection` Key: SPARK-24411 URL: https://issues.apache.org/jira/browse/SPARK-24411 Project: Spark Issue Type: New Feature

[jira] [Resolved] (SPARK-24371) Added isInCollection in DataFrame API for Scala and Java.

2018-05-29 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-24371. - Resolution: Fixed > Added isInCollection in DataFrame API for Scala and Java. >

[jira] [Assigned] (SPARK-24402) Optimize `In` expression when only one element in the collection or collection is empty

2018-05-28 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24402: --- Assignee: DB Tsai > Optimize `In` expression when only one element in the collection or >

[jira] [Updated] (SPARK-24402) Optimize `In` expression when only one element in the collection or collection is empty

2018-05-28 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-24402: Description: Two new rules in the logical plan optimizers are added. # When there is only one element in

[jira] [Created] (SPARK-24402) Optimize `In` expression when only one element in the collection or collection is empty

2018-05-28 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24402: --- Summary: Optimize `In` expression when only one element in the collection or collection is empty Key: SPARK-24402 URL: https://issues.apache.org/jira/browse/SPARK-24402

[jira] [Updated] (SPARK-24371) Added isInCollection in DataFrame API for Scala and Java.

2018-05-28 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-24371: Description: Implemented *{{isInCollection}}* in DataFrame API for both Scala and Java, so users can do

[jira] [Updated] (SPARK-24371) Added isInCollection in DataFrame API for Scala and Java.

2018-05-28 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-24371: Description: Implemented *{{isInCollection}}* in DataFrame API for both Scala and Java, so users can do

[jira] [Updated] (SPARK-24371) Added isinSet in DataFrame API for Scala and Java.

2018-05-28 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-24371: Description: Implemented *{{isinSet}}* in DataFrame API for both Scala and Java, so users can do {code}

[jira] [Updated] (SPARK-24371) Added isInCollection in DataFrame API for Scala and Java.

2018-05-28 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-24371: Summary: Added isInCollection in DataFrame API for Scala and Java. (was: Added isinSet in DataFrame API

[jira] [Created] (SPARK-24371) Added isinSet in DataFrame API for Scala and Java.

2018-05-23 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24371: --- Summary: Added isinSet in DataFrame API for Scala and Java. Key: SPARK-24371 URL: https://issues.apache.org/jira/browse/SPARK-24371 Project: Spark Issue Type: New

[jira] [Updated] (SPARK-24225) Support closing AutoClosable objects in MemoryStore so Broadcast Variables can be released properly

2018-05-10 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-24225: Shepherd: DB Tsai Priority: Major (was: Minor) > Support closing AutoClosable objects in MemoryStore

[jira] [Updated] (SPARK-24225) Support closing AutoClosable objects in MemoryStore so Broadcast Variables can be released properly

2018-05-10 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-24225: Issue Type: New Feature (was: Improvement) > Support closing AutoClosable objects in MemoryStore so

[jira] [Assigned] (SPARK-24225) Support closing AutoClosable objects in MemoryStore so Broadcast Variables can be released properly

2018-05-10 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24225: --- Assignee: Doug Rohrer > Support closing AutoClosable objects in MemoryStore so Broadcast Variables

[jira] [Resolved] (SPARK-24181) Better error message for writing sorted data

2018-05-09 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-24181. - Resolution: Fixed Fix Version/s: 2.4.0 > Better error message for writing sorted data >

[jira] [Assigned] (SPARK-24181) Better error message for writing sorted data

2018-05-04 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24181: --- Assignee: DB Tsai > Better error message for writing sorted data >

[jira] [Created] (SPARK-24181) Better error message for writing sorted data

2018-05-04 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24181: --- Summary: Better error message for writing sorted data Key: SPARK-24181 URL: https://issues.apache.org/jira/browse/SPARK-24181 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-24024) Fix deviance calculations in GLM to handle corner cases

2018-04-23 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-24024. - Resolution: Fixed Fix Version/s: 2.4.0 > Fix deviance calculations in GLM to handle corner cases

[jira] [Assigned] (SPARK-24024) Fix deviance calculations in GLM to handle corner cases

2018-04-23 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24024: --- Assignee: Teng Peng > Fix deviance calculations in GLM to handle corner cases >

[jira] [Comment Edited] (SPARK-22231) Support of map, filter, withColumn, dropColumn in nested list of structures

2017-11-14 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252611#comment-16252611 ] DB Tsai edited comment on SPARK-22231 at 11/14/17 10:45 PM: Thanks

[jira] [Commented] (SPARK-22231) Support of map, filter, withColumn, dropColumn in nested list of structures

2017-11-14 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252611#comment-16252611 ] DB Tsai commented on SPARK-22231: - Thanks [~jeremyrsmith] for adding more details. There are couple

[jira] [Assigned] (SPARK-22231) Support of map, filter, withColumn, dropColumn in nested list of structures

2017-10-10 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-22231: --- Assignee: Jeremy Smith > Support of map, filter, withColumn, dropColumn in nested list of

[jira] [Commented] (SPARK-22231) Support of map, filter, withColumn, dropColumn in nested list of structures

2017-10-10 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198286#comment-16198286 ] DB Tsai commented on SPARK-22231: - [~viirya] Thanks. I fixed the typo :) Yes, {{mapItems}} will work on

[jira] [Commented] (SPARK-22231) Support of map, filter, withColumn, dropColumn in nested list of structures

2017-10-10 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198305#comment-16198305 ] DB Tsai commented on SPARK-22231: - [~rxin] {{df.drop("items.b")}} works too. We also want to have

[jira] [Updated] (SPARK-22231) Support of map, filter, withColumn, dropColumn in nested list of structures

2017-10-10 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-22231: Description: At Netflix's algorithm team, we work on ranking problems to find the great content to

[jira] [Updated] (SPARK-22231) Support of map, filter, withColumn, dropColumn in nested list of structures

2017-10-09 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-22231: Description: At Netflix's algorithm team, we work on ranking problems to find the great content to

[jira] [Updated] (SPARK-22231) Support of map, filter, withColumn, dropColumn in nested list of structures

2017-10-09 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-22231: Description: At Netflix's algorithm team, we work on ranking problems to find the great content to

[jira] [Created] (SPARK-22231) Support of map, filter, withColumn, dropColumn in nested list of structures

2017-10-09 Thread DB Tsai (JIRA)
DB Tsai created SPARK-22231: --- Summary: Support of map, filter, withColumn, dropColumn in nested list of structures Key: SPARK-22231 URL: https://issues.apache.org/jira/browse/SPARK-22231 Project: Spark

[jira] [Resolved] (SPARK-20047) Constrained Logistic Regression

2017-04-27 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-20047. - Resolution: Fixed Fix Version/s: 2.2.1 Target Version/s: 2.2.1 (was: 2.3.0) >

[jira] [Resolved] (SPARK-20483) Mesos Coarse mode may starve other Mesos frameworks if max cores is not a multiple of executor cores

2017-04-27 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-20483. - Resolution: Fixed Fix Version/s: 2.2.0 3.0.0 Issue resolved by pull request

[jira] [Assigned] (SPARK-20483) Mesos Coarse mode may starve other Mesos frameworks if max cores is not a multiple of executor cores

2017-04-26 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-20483: --- Assignee: Davis Shepherd > Mesos Coarse mode may starve other Mesos frameworks if max cores is not

[jira] [Updated] (SPARK-20483) Mesos Coarse mode may starve other Mesos frameworks if max cores is not a multiple of executor cores

2017-04-26 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-20483: Shepherd: DB Tsai > Mesos Coarse mode may starve other Mesos frameworks if max cores is not a > multiple

[jira] [Updated] (SPARK-20483) Mesos Coarse mode may starve other Mesos frameworks if max cores is not a multiple of executor cores

2017-04-26 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-20483: Affects Version/s: 2.1.1 2.0.2 > Mesos Coarse mode may starve other Mesos

[jira] [Resolved] (SPARK-20449) Upgrade breeze version to 0.13.1

2017-04-25 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-20449. - Resolution: Fixed Fix Version/s: 2.2.0 3.0.0 Issue resolved by pull request

[jira] [Assigned] (SPARK-20449) Upgrade breeze version to 0.13.1

2017-04-24 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-20449: --- Assignee: Yanbo Liang > Upgrade breeze version to 0.13.1 > > >

[jira] [Updated] (SPARK-20423) fix MLOR coeffs centering when reg == 0

2017-04-21 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-20423: Fix Version/s: (was: 3.0.0) > fix MLOR coeffs centering when reg == 0 >

[jira] [Resolved] (SPARK-20423) fix MLOR coeffs centering when reg == 0

2017-04-21 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-20423. - Resolution: Fixed Fix Version/s: 2.2.0 3.0.0 Issue resolved by pull request

[jira] [Commented] (SPARK-20399) Can't use same regex pattern between 1.6 and 2.x due to unescaped sql string in parser

2017-04-20 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976192#comment-15976192 ] DB Tsai commented on SPARK-20399: - Thanks [~viirya]. We ran into this bugs while migrating from Spark 1.6

[jira] [Updated] (SPARK-20291) NaNvl(FloatType, NullType) should not be cast to NaNvl(DoubleType, DoubleType)

2017-04-12 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-20291: Fix Version/s: 2.1.1 > NaNvl(FloatType, NullType) should not be cast to NaNvl(DoubleType, > DoubleType)

[jira] [Created] (SPARK-20291) NaNvl(FloatType, NullType) should not be cast to NaNvl(DoubleType, DoubleType)

2017-04-11 Thread DB Tsai (JIRA)
DB Tsai created SPARK-20291: --- Summary: NaNvl(FloatType, NullType) should not be cast to NaNvl(DoubleType, DoubleType) Key: SPARK-20291 URL: https://issues.apache.org/jira/browse/SPARK-20291 Project: Spark

[jira] [Updated] (SPARK-20270) na.fill will change the values in long or integer when the default value is in double

2017-04-10 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-20270: Fix Version/s: 2.1.1 2.0.3 > na.fill will change the values in long or integer when the

<    1   2   3   4   5   6   7   >