[jira] [Comment Edited] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457893#comment-16457893 ] Hyukjin Kwon edited comment on SPARK-23715 at 4/29/18 3:31 AM: --- I already

[jira] [Resolved] (SPARK-23724) Custom record separator for jsons in charsets different from UTF-8

2018-04-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-23724. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20937

[jira] [Assigned] (SPARK-23723) New charset option for json datasource

2018-04-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-23723: Assignee: Maxim Gekk > New charset option for json datasource >

[jira] [Assigned] (SPARK-23094) Json Readers choose wrong encoding when bad records are present and fail

2018-04-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-23094: Assignee: Maxim Gekk (was: Burak Yavuz) > Json Readers choose wrong encoding when bad

[jira] [Resolved] (SPARK-23723) New charset option for json datasource

2018-04-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-23723. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20937

[jira] [Assigned] (SPARK-23724) Custom record separator for jsons in charsets different from UTF-8

2018-04-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-23724: Assignee: Maxim Gekk > Custom record separator for jsons in charsets different from UTF-8

[jira] [Resolved] (SPARK-23094) Json Readers choose wrong encoding when bad records are present and fail

2018-04-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-23094. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20937

[jira] [Commented] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-28 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457896#comment-16457896 ] Bruce Robbins commented on SPARK-23715: --- [~hyukjin.kwon] Yes, I also built from sources and I could

[jira] [Commented] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457894#comment-16457894 ] Hyukjin Kwon commented on SPARK-23715: -- Right, I agree with the PR approach. I just found a small

[jira] [Commented] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457893#comment-16457893 ] Hyukjin Kwon commented on SPARK-23715: -- I already built Hive from source. I did this against

[jira] [Commented] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-28 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457892#comment-16457892 ] Bruce Robbins commented on SPARK-23715: --- Still, I filed an Jira with Hive so they won't release

[jira] [Commented] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-28 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457888#comment-16457888 ] Wenchen Fan commented on SPARK-23715: - `from_utc_timestamp` is also in some traditional RDBMS like

[jira] [Commented] (SPARK-22938) Assert that SQLConf.get is accessed only on the driver.

2018-04-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457879#comment-16457879 ] Apache Spark commented on SPARK-22938: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-28 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457874#comment-16457874 ] Bruce Robbins commented on SPARK-23715: --- I might understand what's going on with Hive. In the

[jira] [Commented] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-28 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457754#comment-16457754 ] Bruce Robbins commented on SPARK-23715: --- I just downloaded and installed hive-2.3.3 (3 April 2018)

[jira] [Commented] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-28 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457674#comment-16457674 ] Bruce Robbins commented on SPARK-23715: --- Could be this: HIVE-14412 > from_utc_timestamp returns

[jira] [Commented] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457669#comment-16457669 ] Hyukjin Kwon commented on SPARK-23715: -- Yea but was just wondering what behaviour Hive wants. It

[jira] [Commented] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-28 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457665#comment-16457665 ] Bruce Robbins commented on SPARK-23715: --- {quote}Which version did you use?{quote} The jars all say

[jira] [Commented] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457658#comment-16457658 ] Hyukjin Kwon commented on SPARK-23715: -- I used the (almost) latest build one. Will double check and

[jira] [Commented] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-28 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457657#comment-16457657 ] Bruce Robbins commented on SPARK-23715: --- Maybe a configuration setting or difference between

[jira] [Updated] (SPARK-23171) Reduce the time costs of the rule runs that do not change the plans

2018-04-28 Thread Teng Peng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teng Peng updated SPARK-23171: -- Issue Type: Improvement (was: Umbrella) > Reduce the time costs of the rule runs that do not change

[jira] [Updated] (SPARK-23171) Reduce the time costs of the rule runs that do not change the plans

2018-04-28 Thread Teng Peng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teng Peng updated SPARK-23171: -- Issue Type: Umbrella (was: Improvement) > Reduce the time costs of the rule runs that do not change

[jira] [Commented] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457604#comment-16457604 ] Hyukjin Kwon commented on SPARK-23715: -- [~bersprockets], mind if I ask to clarify {quote} In Hive

[jira] [Commented] (SPARK-24117) Unified the getSizePerRow

2018-04-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457549#comment-16457549 ] Apache Spark commented on SPARK-24117: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24117) Unified the getSizePerRow

2018-04-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24117: Assignee: (was: Apache Spark) > Unified the getSizePerRow > -

[jira] [Assigned] (SPARK-24117) Unified the getSizePerRow

2018-04-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24117: Assignee: Apache Spark > Unified the getSizePerRow > - > >

[jira] [Created] (SPARK-24117) Unified the getSizePerRow

2018-04-28 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-24117: --- Summary: Unified the getSizePerRow Key: SPARK-24117 URL: https://issues.apache.org/jira/browse/SPARK-24117 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-24046) Rate Source doesn't gradually increase rate when rampUpTime>=RowsPerSecond

2018-04-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457430#comment-16457430 ] Apache Spark commented on SPARK-24046: -- User 'jerryshao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24046) Rate Source doesn't gradually increase rate when rampUpTime>=RowsPerSecond

2018-04-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24046: Assignee: (was: Apache Spark) > Rate Source doesn't gradually increase rate when

[jira] [Assigned] (SPARK-24046) Rate Source doesn't gradually increase rate when rampUpTime>=RowsPerSecond

2018-04-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24046: Assignee: Apache Spark > Rate Source doesn't gradually increase rate when

[jira] [Commented] (SPARK-23954) Converting spark dataframe containing int64 fields to R dataframes leads to impredictable errors.

2018-04-28 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457390#comment-16457390 ] Felix Cheung commented on SPARK-23954: -- yap, please see discussion in SPARK-12360 in particular >