[jira] [Comment Edited] (SPARK-22936) providing HttpStreamSource and HttpStreamSink

2018-01-02 Thread bluejoe (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16309220#comment-16309220 ] bluejoe edited comment on SPARK-22936 at 1/3/18 7:25 AM: - The latest

[jira] [Commented] (SPARK-22936) providing HttpStreamSource and HttpStreamSink

2018-01-02 Thread bluejoe (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16309220#comment-16309220 ] bluejoe commented on SPARK-22936: - The latest spark-http-stream artifact has been released to the central

[jira] [Commented] (SPARK-17762) invokeJava fails when serialized argument list is larger than INT_MAX (2,147,483,647) bytes

2018-01-02 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16309053#comment-16309053 ] Hossein Falaki commented on SPARK-17762: I think SPARK-17790 is one place where this limit causes

[jira] [Commented] (SPARK-22942) Spark Sql UDF throwing NullPointer when adding a filter on a columns that uses that UDF

2018-01-02 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16309037#comment-16309037 ] Takeshi Yamamuro commented on SPARK-22942: -- Since spark passes null to udfs in optimizer rules,

[jira] [Resolved] (SPARK-22898) collect_set aggregation on bucketed table causes an exchange stage

2018-01-02 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh resolved SPARK-22898. - Resolution: Duplicate > collect_set aggregation on bucketed table causes an exchange

[jira] [Commented] (SPARK-22898) collect_set aggregation on bucketed table causes an exchange stage

2018-01-02 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308970#comment-16308970 ] Liang-Chi Hsieh commented on SPARK-22898: - If no problem I will resolve this as duplicate. You

[jira] [Created] (SPARK-22943) OneHotEncoder supports manual specification of categorySizes

2018-01-02 Thread yuhao yang (JIRA)
yuhao yang created SPARK-22943: -- Summary: OneHotEncoder supports manual specification of categorySizes Key: SPARK-22943 URL: https://issues.apache.org/jira/browse/SPARK-22943 Project: Spark

[jira] [Comment Edited] (SPARK-22942) Spark Sql UDF throwing NullPointer when adding a filter on a columns that uses that UDF

2018-01-02 Thread Matthew Fishkin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308882#comment-16308882 ] Matthew Fishkin edited comment on SPARK-22942 at 1/2/18 11:27 PM: -- I

[jira] [Comment Edited] (SPARK-22942) Spark Sql UDF throwing NullPointer when adding a filter on a columns that uses that UDF

2018-01-02 Thread Matthew Fishkin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308882#comment-16308882 ] Matthew Fishkin edited comment on SPARK-22942 at 1/2/18 11:25 PM: -- I

[jira] [Commented] (SPARK-22942) Spark Sql UDF throwing NullPointer when adding a filter on a columns that uses that UDF

2018-01-02 Thread Matthew Fishkin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308882#comment-16308882 ] Matthew Fishkin commented on SPARK-22942: - I would expect that to work too. I'm more curious why

[jira] [Commented] (SPARK-22126) Fix model-specific optimization support for ML tuning

2018-01-02 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308868#comment-16308868 ] Bryan Cutler commented on SPARK-22126: -- Thanks for taking a look [~josephkb]! I believe it's

[jira] [Commented] (SPARK-22942) Spark Sql UDF throwing NullPointer when adding a filter on a columns that uses that UDF

2018-01-02 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308830#comment-16308830 ] Takeshi Yamamuro commented on SPARK-22942: -- I think you just need NULL checks; {code} val

[jira] [Comment Edited] (SPARK-21687) Spark SQL should set createTime for Hive partition

2018-01-02 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308153#comment-16308153 ] Takeshi Yamamuro edited comment on SPARK-21687 at 1/2/18 10:26 PM: --- I

[jira] [Commented] (SPARK-22404) Provide an option to use unmanaged AM in yarn-client mode

2018-01-02 Thread Devaraj K (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308802#comment-16308802 ] Devaraj K commented on SPARK-22404: --- Thanks [~irashid] for the comment. bq. can you provide a little

[jira] [Updated] (SPARK-22942) Spark Sql UDF throwing NullPointer when adding a filter on a columns that uses that UDF

2018-01-02 Thread Matthew Fishkin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Fishkin updated SPARK-22942: Description: I ran into an interesting issue when trying to do a `filter` on a dataframe

[jira] [Commented] (SPARK-16693) Remove R deprecated methods

2018-01-02 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308765#comment-16308765 ] Shivaram Venkataraman commented on SPARK-16693: --- Did we have the discussion on dev@ ? I

[jira] [Updated] (SPARK-22942) Spark Sql UDF throwing NullPointer when adding a filter on a columns that uses that UDF

2018-01-02 Thread Matthew Fishkin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Fishkin updated SPARK-22942: Description: I ran into an interesting issue when trying to do a `filter` on a dataframe

[jira] [Created] (SPARK-22942) Spark Sql UDF throwing NullPointer when adding a filter on a columns that uses that UDF

2018-01-02 Thread Matthew Fishkin (JIRA)
Matthew Fishkin created SPARK-22942: --- Summary: Spark Sql UDF throwing NullPointer when adding a filter on a columns that uses that UDF Key: SPARK-22942 URL: https://issues.apache.org/jira/browse/SPARK-22942

[jira] [Updated] (SPARK-22942) Spark Sql UDF throwing NullPointer when adding a filter on a columns that uses that UDF

2018-01-02 Thread Matthew Fishkin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Fishkin updated SPARK-22942: Description: I ran into an interesting issue when trying to do a `filter` on a dataframe

[jira] [Created] (SPARK-22941) Allow SparkSubmit to throw exceptions instead of exiting / printing errors.

2018-01-02 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-22941: -- Summary: Allow SparkSubmit to throw exceptions instead of exiting / printing errors. Key: SPARK-22941 URL: https://issues.apache.org/jira/browse/SPARK-22941

[jira] [Assigned] (SPARK-20664) Remove stale applications from SHS listing

2018-01-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20664: Assignee: (was: Apache Spark) > Remove stale applications from SHS listing >

[jira] [Assigned] (SPARK-20664) Remove stale applications from SHS listing

2018-01-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20664: Assignee: Apache Spark > Remove stale applications from SHS listing >

[jira] [Commented] (SPARK-20664) Remove stale applications from SHS listing

2018-01-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308579#comment-16308579 ] Apache Spark commented on SPARK-20664: -- User 'vanzin' has created a pull request for this issue:

[jira] [Commented] (SPARK-21319) UnsafeExternalRowSorter.RowComparator memory leak

2018-01-02 Thread William Kinney (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308566#comment-16308566 ] William Kinney commented on SPARK-21319: Is there a workaround for this for version 2.2.0? >

[jira] [Commented] (SPARK-22940) Test suite HiveExternalCatalogVersionsSuite fails on platforms that don't have wget installed

2018-01-02 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308491#comment-16308491 ] Sean Owen commented on SPARK-22940: --- Agreed. Many other scripts require wget, though at least one will

[jira] [Updated] (SPARK-22940) Test suite HiveExternalCatalogVersionsSuite fails on platforms that don't have wget installed

2018-01-02 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-22940: -- Description: On platforms that don't have wget installed (e.g., Mac OS X), test suite

[jira] [Created] (SPARK-22940) Test suite HiveExternalCatalogVersionsSuite fails on platforms that don't have wget installed

2018-01-02 Thread Bruce Robbins (JIRA)
Bruce Robbins created SPARK-22940: - Summary: Test suite HiveExternalCatalogVersionsSuite fails on platforms that don't have wget installed Key: SPARK-22940 URL: https://issues.apache.org/jira/browse/SPARK-22940

[jira] [Updated] (SPARK-22940) Test suite HiveExternalCatalogVersionsSuite fails on platforms that don't have wget installed

2018-01-02 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-22940: -- Description: On platforms that don't have wget installed (e.g., Mac OS X), test suite

[jira] [Commented] (SPARK-21687) Spark SQL should set createTime for Hive partition

2018-01-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308378#comment-16308378 ] Dongjoon Hyun commented on SPARK-21687: --- Thank you for ccing me, [~maropu]. I agree with that. >

[jira] [Commented] (SPARK-16693) Remove R deprecated methods

2018-01-02 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308367#comment-16308367 ] Felix Cheung commented on SPARK-16693: -- These are all non public methods, so officially not public

[jira] [Commented] (SPARK-22935) Dataset with Java Beans for java.sql.Date throws CompileException

2018-01-02 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308363#comment-16308363 ] Jacek Laskowski commented on SPARK-22935: - It does not seem to be the case as described in

[jira] [Assigned] (SPARK-22939) Support Spark UDF in registerFunction

2018-01-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22939: Assignee: Apache Spark > Support Spark UDF in registerFunction >

[jira] [Commented] (SPARK-22939) Support Spark UDF in registerFunction

2018-01-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308252#comment-16308252 ] Apache Spark commented on SPARK-22939: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22939) Support Spark UDF in registerFunction

2018-01-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22939: Assignee: (was: Apache Spark) > Support Spark UDF in registerFunction >

[jira] [Updated] (SPARK-22939) Support Spark UDF in registerFunction

2018-01-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22939: Summary: Support Spark UDF in registerFunction (was: registerFunction accepts Spark UDF ) > Support

[jira] [Updated] (SPARK-22939) registerFunction accepts Spark UDF

2018-01-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22939: Summary: registerFunction accepts Spark UDF (was: registerFunction also accepts Spark UDF ) >

[jira] [Created] (SPARK-22939) registerFunction also accepts Spark UDF

2018-01-02 Thread Xiao Li (JIRA)
Xiao Li created SPARK-22939: --- Summary: registerFunction also accepts Spark UDF Key: SPARK-22939 URL: https://issues.apache.org/jira/browse/SPARK-22939 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-22897) Expose stageAttemptId in TaskContext

2018-01-02 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22897. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20082

[jira] [Assigned] (SPARK-22897) Expose stageAttemptId in TaskContext

2018-01-02 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-22897: --- Assignee: Xianjin YE > Expose stageAttemptId in TaskContext >

[jira] [Commented] (SPARK-22935) Dataset with Java Beans for java.sql.Date throws CompileException

2018-01-02 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308194#comment-16308194 ] Kazuaki Ishizaki commented on SPARK-22935: -- [~jlaskowski] When you see the scheme of this

[jira] [Commented] (SPARK-22936) providing HttpStreamSource and HttpStreamSink

2018-01-02 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308177#comment-16308177 ] Sean Owen commented on SPARK-22936: --- I think this can and should start as you have started it, as an

[jira] [Commented] (SPARK-16693) Remove R deprecated methods

2018-01-02 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308161#comment-16308161 ] Sean Owen commented on SPARK-16693: --- Would this be a breaking change though? > Remove R deprecated

[jira] [Commented] (SPARK-21687) Spark SQL should set createTime for Hive partition

2018-01-02 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308153#comment-16308153 ] Takeshi Yamamuro commented on SPARK-21687: -- I feel this make some sense (But, this is a not bug,

[jira] [Commented] (SPARK-22938) Assert that SQLConf.get is accessed only on the driver.

2018-01-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308130#comment-16308130 ] Apache Spark commented on SPARK-22938: -- User 'juliuszsompolski' has created a pull request for this

[jira] [Assigned] (SPARK-22938) Assert that SQLConf.get is accessed only on the driver.

2018-01-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22938: Assignee: (was: Apache Spark) > Assert that SQLConf.get is accessed only on the

[jira] [Assigned] (SPARK-22938) Assert that SQLConf.get is accessed only on the driver.

2018-01-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22938: Assignee: Apache Spark > Assert that SQLConf.get is accessed only on the driver. >

[jira] [Created] (SPARK-22938) Assert that SQLConf.get is accessed only on the driver.

2018-01-02 Thread Juliusz Sompolski (JIRA)
Juliusz Sompolski created SPARK-22938: - Summary: Assert that SQLConf.get is accessed only on the driver. Key: SPARK-22938 URL: https://issues.apache.org/jira/browse/SPARK-22938 Project: Spark

[jira] [Assigned] (SPARK-22937) SQL elt for binary inputs

2018-01-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22937: Assignee: Apache Spark > SQL elt for binary inputs > - > >

[jira] [Assigned] (SPARK-22937) SQL elt for binary inputs

2018-01-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22937: Assignee: (was: Apache Spark) > SQL elt for binary inputs > -

[jira] [Commented] (SPARK-22937) SQL elt for binary inputs

2018-01-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308062#comment-16308062 ] Apache Spark commented on SPARK-22937: -- User 'maropu' has created a pull request for this issue:

[jira] [Commented] (SPARK-22918) sbt test (spark - local) fail after upgrading to 2.2.1 with: java.security.AccessControlException: access denied org.apache.derby.security.SystemPermission( "engine",

2018-01-02 Thread Abhay Pradhan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308059#comment-16308059 ] Abhay Pradhan commented on SPARK-22918: --- confirmed that our team is also affected by this issue.

[jira] [Created] (SPARK-22937) SQL elt for binary inputs

2018-01-02 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-22937: Summary: SQL elt for binary inputs Key: SPARK-22937 URL: https://issues.apache.org/jira/browse/SPARK-22937 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-22613) Make UNCACHE TABLE behaviour consistent with CACHE TABLE

2018-01-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22613: Assignee: (was: Apache Spark) > Make UNCACHE TABLE behaviour consistent with CACHE

[jira] [Commented] (SPARK-22613) Make UNCACHE TABLE behaviour consistent with CACHE TABLE

2018-01-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308019#comment-16308019 ] Apache Spark commented on SPARK-22613: -- User 'vinodkc' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22613) Make UNCACHE TABLE behaviour consistent with CACHE TABLE

2018-01-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22613: Assignee: Apache Spark > Make UNCACHE TABLE behaviour consistent with CACHE TABLE >

[jira] [Commented] (SPARK-21687) Spark SQL should set createTime for Hive partition

2018-01-02 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307955#comment-16307955 ] Gabor Somogyi commented on SPARK-21687: --- [~srowen] I've just seen the Branch 2.3 cut mail. Should

[jira] [Commented] (SPARK-21687) Spark SQL should set createTime for Hive partition

2018-01-02 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307877#comment-16307877 ] Gabor Somogyi commented on SPARK-21687: --- I would like to work on this. Please notify me if somebody