[jira] [Commented] (SPARK-16670) Partition pruning doesn't work when queries contain disjunctions involving partition columns

2016-07-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388989#comment-15388989 ] Xiao Li commented on SPARK-16670: - https://github.com/apache/spark/pull/13585 A related P

[jira] [Comment Edited] (SPARK-16679) Move `private[sql]` methods in public APIs used for Python/R into a single ‘helper class’

2016-07-21 Thread Narine Kokhlikyan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388942#comment-15388942 ] Narine Kokhlikyan edited comment on SPARK-16679 at 7/22/16 5:34 AM: ---

[jira] [Updated] (SPARK-16679) Move `private[sql]` methods in public APIs used for Python/R into a single ‘helper class’

2016-07-21 Thread Narine Kokhlikyan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narine Kokhlikyan updated SPARK-16679: -- Description: Based on our discussions in: https://github.com/apache/spark/pull/12836#is

[jira] [Updated] (SPARK-16679) Move `private[sql]` methods in public APIs used for Python/R into a single ‘helper class’

2016-07-21 Thread Narine Kokhlikyan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narine Kokhlikyan updated SPARK-16679: -- Description: Based on our discussions in: https://github.com/apache/spark/pull/12836#is

[jira] [Commented] (SPARK-16679) Move `private[sql]` methods in public APIs used for Python/R into a single ‘helper class’

2016-07-21 Thread Narine Kokhlikyan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388942#comment-15388942 ] Narine Kokhlikyan commented on SPARK-16679: --- Two R helper methods on scala side

[jira] [Resolved] (SPARK-16556) Silent Ignore Bucket Specification When Creating Table Using Schema Inference

2016-07-21 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-16556. - Resolution: Fixed Fix Version/s: 2.1.0 > Silent Ignore Bucket Specification When Creating

[jira] [Updated] (SPARK-16556) Silent Ignore Bucket Specification When Creating Table Using Schema Inference

2016-07-21 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-16556: Assignee: Xiao Li > Silent Ignore Bucket Specification When Creating Table Using Schema Inference

[jira] [Commented] (SPARK-16287) Implement str_to_map SQL function

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388940#comment-15388940 ] Apache Spark commented on SPARK-16287: -- User 'jaceklaskowski' has created a pull req

[jira] [Updated] (SPARK-16678) Disallow Creating/Altering a View when the same-name Table Exists

2016-07-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-16678: Description: When we create OR alter a view, we check whether the view already exists. In the current impl

[jira] [Updated] (SPARK-16678) Disallow Creating/Altering a View when the same-name Table Exists

2016-07-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-16678: Description: When we create OR alter a view, we check whether the view already exists. In the current impl

[jira] [Updated] (SPARK-16678) Disallow Creating/Altering a View when the same-name Table Exists

2016-07-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-16678: Summary: Disallow Creating/Altering a View when the same-name Table Exists (was: Disallow Creating a View

[jira] [Commented] (SPARK-4003) Add {Big Decimal, Timestamp, Date} types to Java SqlContext

2016-07-21 Thread Adrian Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388922#comment-15388922 ] Adrian Wang commented on SPARK-4003: DataTypes.TimestampType is not using java.sql.Tim

[jira] [Updated] (SPARK-16679) Move `private[sql]` methods in public APIs used for Python/R into a single ‘helper class’

2016-07-21 Thread Narine Kokhlikyan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narine Kokhlikyan updated SPARK-16679: -- Component/s: SparkR > Move `private[sql]` methods in public APIs used for Python/R int

[jira] [Created] (SPARK-16679) Move `private[sql]` methods in public APIs used for Python/R into a single ‘helper class’

2016-07-21 Thread Narine Kokhlikyan (JIRA)
Narine Kokhlikyan created SPARK-16679: - Summary: Move `private[sql]` methods in public APIs used for Python/R into a single ‘helper class’ Key: SPARK-16679 URL: https://issues.apache.org/jira/browse/SPARK-166

[jira] [Commented] (SPARK-16646) LEAST doesn't accept numeric arguments with different data types

2016-07-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388916#comment-15388916 ] Hyukjin Kwon commented on SPARK-16646: -- I see! please let me leave my thought as wel

[jira] [Commented] (SPARK-16646) LEAST doesn't accept numeric arguments with different data types

2016-07-21 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388897#comment-15388897 ] Wenchen Fan commented on SPARK-16646: - some other thoughts about decimal type in Spar

[jira] [Commented] (SPARK-16670) Partition pruning doesn't work when queries contain disjunctions involving partition columns

2016-07-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388894#comment-15388894 ] Xiao Li commented on SPARK-16670: - welcome > Partition pruning doesn't work when queries

[jira] [Assigned] (SPARK-16677) Strange Error when Issuing Load Table Against A View

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16677: Assignee: Apache Spark > Strange Error when Issuing Load Table Against A View > --

[jira] [Commented] (SPARK-16678) Disallow Creating a View when the same-name Table Exists

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1537#comment-1537 ] Apache Spark commented on SPARK-16678: -- User 'gatorsmile' has created a pull request

[jira] [Commented] (SPARK-16677) Strange Error when Issuing Load Table Against A View

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1538#comment-1538 ] Apache Spark commented on SPARK-16677: -- User 'gatorsmile' has created a pull request

[jira] [Assigned] (SPARK-16678) Disallow Creating a View when the same-name Table Exists

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16678: Assignee: (was: Apache Spark) > Disallow Creating a View when the same-name Table Exis

[jira] [Assigned] (SPARK-16678) Disallow Creating a View when the same-name Table Exists

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16678: Assignee: Apache Spark > Disallow Creating a View when the same-name Table Exists > --

[jira] [Assigned] (SPARK-16677) Strange Error when Issuing Load Table Against A View

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16677: Assignee: (was: Apache Spark) > Strange Error when Issuing Load Table Against A View >

[jira] [Commented] (SPARK-16670) Partition pruning doesn't work when queries contain disjunctions involving partition columns

2016-07-21 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388877#comment-15388877 ] Yin Huai commented on SPARK-16670: -- Thanks! > Partition pruning doesn't work when queri

[jira] [Created] (SPARK-16678) Disallow Creating a View when the same-name Table Exists

2016-07-21 Thread Xiao Li (JIRA)
Xiao Li created SPARK-16678: --- Summary: Disallow Creating a View when the same-name Table Exists Key: SPARK-16678 URL: https://issues.apache.org/jira/browse/SPARK-16678 Project: Spark Issue Type: Bu

[jira] [Created] (SPARK-16677) Strange Error when Issuing Load Table Against A View

2016-07-21 Thread Xiao Li (JIRA)
Xiao Li created SPARK-16677: --- Summary: Strange Error when Issuing Load Table Against A View Key: SPARK-16677 URL: https://issues.apache.org/jira/browse/SPARK-16677 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-16670) Partition pruning doesn't work when queries contain disjunctions involving partition columns

2016-07-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388845#comment-15388845 ] Xiao Li commented on SPARK-16670: - I can take a look at this issue. This sounds familiar

[jira] [Updated] (SPARK-16676) Spark jobs stay in pending

2016-07-21 Thread Joe Chong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Chong updated SPARK-16676: -- Attachment: Spark UI stays @ pending.png > Spark jobs stay in pending > -- > >

[jira] [Created] (SPARK-16676) Spark jobs stay in pending

2016-07-21 Thread Joe Chong (JIRA)
Joe Chong created SPARK-16676: - Summary: Spark jobs stay in pending Key: SPARK-16676 URL: https://issues.apache.org/jira/browse/SPARK-16676 Project: Spark Issue Type: Bug Components: ML

[jira] [Commented] (SPARK-16675) Avoid per-record type dispatch in JDBC when writing

2016-07-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388829#comment-15388829 ] Hyukjin Kwon commented on SPARK-16675: -- I will work on this. > Avoid per-record typ

[jira] [Created] (SPARK-16675) Avoid per-record type dispatch in JDBC when writing

2016-07-21 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-16675: Summary: Avoid per-record type dispatch in JDBC when writing Key: SPARK-16675 URL: https://issues.apache.org/jira/browse/SPARK-16675 Project: Spark Issue Typ

[jira] [Assigned] (SPARK-16674) Avoid per-record type dispatch in JDBC when reading

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16674: Assignee: Apache Spark > Avoid per-record type dispatch in JDBC when reading > ---

[jira] [Assigned] (SPARK-16674) Avoid per-record type dispatch in JDBC when reading

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16674: Assignee: (was: Apache Spark) > Avoid per-record type dispatch in JDBC when reading >

[jira] [Commented] (SPARK-16674) Avoid per-record type dispatch in JDBC when reading

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388825#comment-15388825 ] Apache Spark commented on SPARK-16674: -- User 'HyukjinKwon' has created a pull reques

[jira] [Created] (SPARK-16674) Avoid per-record type dispatch in JDBC when reading

2016-07-21 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-16674: Summary: Avoid per-record type dispatch in JDBC when reading Key: SPARK-16674 URL: https://issues.apache.org/jira/browse/SPARK-16674 Project: Spark Issue Typ

[jira] [Commented] (SPARK-16550) Caching data with replication doesn't replicate data

2016-07-21 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388785#comment-15388785 ] Reynold Xin commented on SPARK-16550: - [~shubhamc] can you test the patch [~ekhliang]

[jira] [Resolved] (SPARK-16287) Implement str_to_map SQL function

2016-07-21 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-16287. - Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 > Implement str_to_map SQL

[jira] [Updated] (SPARK-16287) Implement str_to_map SQL function

2016-07-21 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-16287: Assignee: Sandeep Singh > Implement str_to_map SQL function > - > >

[jira] [Commented] (SPARK-15857) Add Caller Context in Spark

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388727#comment-15388727 ] Apache Spark commented on SPARK-15857: -- User 'Sherry302' has created a pull request

[jira] [Assigned] (SPARK-15857) Add Caller Context in Spark

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15857: Assignee: (was: Apache Spark) > Add Caller Context in Spark >

[jira] [Assigned] (SPARK-15857) Add Caller Context in Spark

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15857: Assignee: Apache Spark > Add Caller Context in Spark > --- > >

[jira] [Commented] (SPARK-16574) Distribute computing to each node based on certain hints

2016-07-21 Thread Norman He (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388657#comment-15388657 ] Norman He commented on SPARK-16574: --- Here is how i solved it: /** * Using Redis to

[jira] [Commented] (SPARK-16550) Caching data with replication doesn't replicate data

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388638#comment-15388638 ] Apache Spark commented on SPARK-16550: -- User 'ericl' has created a pull request for

[jira] [Issue Comment Deleted] (SPARK-16520) Link executors to corresponding worker pages

2016-07-21 Thread Tao Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Lin updated SPARK-16520: Comment: was deleted (was: ## What changes were proposed in this pull request? Added a new column named "W

[jira] [Issue Comment Deleted] (SPARK-16520) Link executors to corresponding worker pages

2016-07-21 Thread Tao Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Lin updated SPARK-16520: Comment: was deleted (was: It looks like this if we use the worker URL as the placeholder (instead of "Wor

[jira] [Commented] (SPARK-16673) New Executor Page displays columns that used to be conditionally hidden

2016-07-21 Thread Tao Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388596#comment-15388596 ] Tao Lin commented on SPARK-16673: - Great! If you are going tackle this AFTER SPARK-16520

[jira] [Resolved] (SPARK-16334) [SQL] SQL query on parquet table java.lang.ArrayIndexOutOfBoundsException

2016-07-21 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-16334. - Resolution: Fixed Assignee: Sameer Agarwal Fix Version/s: 2.1.0

[jira] [Assigned] (SPARK-11977) Support accessing a DataFrame column using its name without backticks if the name contains '.'

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11977: Assignee: (was: Apache Spark) > Support accessing a DataFrame column using its name wi

[jira] [Assigned] (SPARK-11977) Support accessing a DataFrame column using its name without backticks if the name contains '.'

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11977: Assignee: Apache Spark > Support accessing a DataFrame column using its name without backt

[jira] [Assigned] (SPARK-11977) Support accessing a DataFrame column using its name without backticks if the name contains '.'

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11977: Assignee: (was: Apache Spark) > Support accessing a DataFrame column using its name wi

[jira] [Commented] (SPARK-11977) Support accessing a DataFrame column using its name without backticks if the name contains '.'

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388542#comment-15388542 ] Apache Spark commented on SPARK-11977: -- User 'rerngvit' has created a pull request f

[jira] [Assigned] (SPARK-11977) Support accessing a DataFrame column using its name without backticks if the name contains '.'

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11977: Assignee: Apache Spark > Support accessing a DataFrame column using its name without backt

[jira] [Commented] (SPARK-16673) New Executor Page displays columns that used to be conditionally hidden

2016-07-21 Thread Alex Bozarth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388536#comment-15388536 ] Alex Bozarth commented on SPARK-16673: -- Once SPARK-16520 is settled I'm willing to t

[jira] [Issue Comment Deleted] (SPARK-4003) Add {Big Decimal, Timestamp, Date} types to Java SqlContext

2016-07-21 Thread Steven Magana-Zook (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Magana-Zook updated SPARK-4003: -- Comment: was deleted (was: I am still seeing this issue in Spark 1.6.1 when I try using

[jira] [Commented] (SPARK-16260) PySpark ML Example Improvements and Cleanup

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388487#comment-15388487 ] Apache Spark commented on SPARK-16260: -- User 'BryanCutler' has created a pull reques

[jira] [Commented] (SPARK-16673) New Executor Page displays columns that used to be conditionally hidden

2016-07-21 Thread Alex Bozarth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388435#comment-15388435 ] Alex Bozarth commented on SPARK-16673: -- SPARK-16520 currently has a pr submitted tha

[jira] [Created] (SPARK-16673) New Executor Page displays columns that used to be conditionally hidden

2016-07-21 Thread Alex Bozarth (JIRA)
Alex Bozarth created SPARK-16673: Summary: New Executor Page displays columns that used to be conditionally hidden Key: SPARK-16673 URL: https://issues.apache.org/jira/browse/SPARK-16673 Project: Spar

[jira] [Updated] (SPARK-16670) Partition pruning doesn't work when queries contain disjunctions involving partition columns

2016-07-21 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-16670: - Attachment: SPARK-16670.html yea. An example can be found in the attached notebook. > Partition pruning

[jira] [Assigned] (SPARK-16672) SQLBuilder should not raise exceptions on EXISTS queries

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16672: Assignee: Apache Spark > SQLBuilder should not raise exceptions on EXISTS queries > --

[jira] [Commented] (SPARK-16672) SQLBuilder should not raise exceptions on EXISTS queries

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388353#comment-15388353 ] Apache Spark commented on SPARK-16672: -- User 'dongjoon-hyun' has created a pull requ

[jira] [Assigned] (SPARK-16672) SQLBuilder should not raise exceptions on EXISTS queries

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16672: Assignee: (was: Apache Spark) > SQLBuilder should not raise exceptions on EXISTS queri

[jira] [Updated] (SPARK-16672) SQLBuilder should not raise exceptions on EXISTS queries

2016-07-21 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-16672: -- Summary: SQLBuilder should not raise exceptions on EXISTS queries (was: SQLBuilder ) > SQLBui

[jira] [Updated] (SPARK-16672) SQLBuilder

2016-07-21 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-16672: -- Description: Currently, `SQLBuilder` raises `empty.reduceLeft` exceptions on **unoptimized** `

[jira] [Created] (SPARK-16672) SQLBuilder

2016-07-21 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-16672: - Summary: SQLBuilder Key: SPARK-16672 URL: https://issues.apache.org/jira/browse/SPARK-16672 Project: Spark Issue Type: Bug Reporter: Dongjoon

[jira] [Commented] (SPARK-16670) Partition pruning doesn't work when queries contain disjunctions involving partition columns

2016-07-21 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388333#comment-15388333 ] Reynold Xin commented on SPARK-16670: - Would be great to attach an example here. >

[jira] [Created] (SPARK-16671) Merge variable substitution code in core and SQL

2016-07-21 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-16671: -- Summary: Merge variable substitution code in core and SQL Key: SPARK-16671 URL: https://issues.apache.org/jira/browse/SPARK-16671 Project: Spark Issue Ty

[jira] [Commented] (SPARK-15581) MLlib 2.1 Roadmap

2016-07-21 Thread Rajesh Bordawekar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388278#comment-15388278 ] Rajesh Bordawekar commented on SPARK-15581: --- We, at IBM Watson Research, are wo

[jira] [Created] (SPARK-16670) Partition pruning doesn't work when queries contain disjunctions involving partition columns

2016-07-21 Thread Yin Huai (JIRA)
Yin Huai created SPARK-16670: Summary: Partition pruning doesn't work when queries contain disjunctions involving partition columns Key: SPARK-16670 URL: https://issues.apache.org/jira/browse/SPARK-16670

[jira] [Commented] (SPARK-16669) Partition pruning for metastore relation size estimates for better join selection.

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388248#comment-15388248 ] Apache Spark commented on SPARK-16669: -- User 'Parth-Brahmbhatt' has created a pull r

[jira] [Assigned] (SPARK-16669) Partition pruning for metastore relation size estimates for better join selection.

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16669: Assignee: (was: Apache Spark) > Partition pruning for metastore relation size estimate

[jira] [Commented] (SPARK-16669) Partition pruning for metastore relation size estimates for better join selection.

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388247#comment-15388247 ] Apache Spark commented on SPARK-16669: -- User 'Parth-Brahmbhatt' has created a pull r

[jira] [Assigned] (SPARK-16669) Partition pruning for metastore relation size estimates for better join selection.

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16669: Assignee: Apache Spark > Partition pruning for metastore relation size estimates for bette

[jira] [Resolved] (SPARK-16656) CreateTableAsSelectSuite is flaky

2016-07-21 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-16656. -- Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 > CreateTableAsSelectSuite is fl

[jira] [Created] (SPARK-16669) Partition pruning for metastore relation size estimates for better join selection.

2016-07-21 Thread Parth Brahmbhatt (JIRA)
Parth Brahmbhatt created SPARK-16669: Summary: Partition pruning for metastore relation size estimates for better join selection. Key: SPARK-16669 URL: https://issues.apache.org/jira/browse/SPARK-16669

[jira] [Commented] (SPARK-16656) CreateTableAsSelectSuite is flaky

2016-07-21 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388236#comment-15388236 ] Yin Huai commented on SPARK-16656: -- https://github.com/apache/spark/pull/14289 has been

[jira] [Commented] (SPARK-16664) Spark 1.6.2 - Persist call on Data frames with more than 200 columns is wiping out the data.

2016-07-21 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388219#comment-15388219 ] Dongjoon Hyun commented on SPARK-16664: --- Also, FYI, RC5 is the same with the curren

[jira] [Commented] (SPARK-16664) Spark 1.6.2 - Persist call on Data frames with more than 200 columns is wiping out the data.

2016-07-21 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388209#comment-15388209 ] Dongjoon Hyun commented on SPARK-16664: --- It looks like that. FYI, here is the resul

[jira] [Commented] (SPARK-16668) Test parquet reader for row groups containing both dictionary and plain encoded pages

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388189#comment-15388189 ] Apache Spark commented on SPARK-16668: -- User 'sameeragarwal' has created a pull requ

[jira] [Assigned] (SPARK-16668) Test parquet reader for row groups containing both dictionary and plain encoded pages

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16668: Assignee: Apache Spark > Test parquet reader for row groups containing both dictionary and

[jira] [Assigned] (SPARK-16668) Test parquet reader for row groups containing both dictionary and plain encoded pages

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16668: Assignee: (was: Apache Spark) > Test parquet reader for row groups containing both dic

[jira] [Created] (SPARK-16668) Test parquet reader for row groups containing both dictionary and plain encoded pages

2016-07-21 Thread Sameer Agarwal (JIRA)
Sameer Agarwal created SPARK-16668: -- Summary: Test parquet reader for row groups containing both dictionary and plain encoded pages Key: SPARK-16668 URL: https://issues.apache.org/jira/browse/SPARK-16668

[jira] [Resolved] (SPARK-16194) No way to dynamically set env vars on driver in cluster mode

2016-07-21 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-16194. --- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14167 [https://github.co

[jira] [Updated] (SPARK-16194) No way to dynamically set env vars on driver in cluster mode

2016-07-21 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-16194: -- Assignee: Michael Gummelt > No way to dynamically set env vars on driver in cluster mode >

[jira] [Commented] (SPARK-4003) Add {Big Decimal, Timestamp, Date} types to Java SqlContext

2016-07-21 Thread Steven Magana-Zook (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388032#comment-15388032 ] Steven Magana-Zook commented on SPARK-4003: --- I am still seeing this issue in Spa

[jira] [Commented] (SPARK-16667) Spark driver executor dont release unused memory

2016-07-21 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15387958#comment-15387958 ] Sean Owen commented on SPARK-16667: --- I don't think there's any info here that suggests

[jira] [Updated] (SPARK-16667) Spark driver executor dont release unused memory

2016-07-21 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-16667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luis Angel Hernández Acosta updated SPARK-16667: Description: I'm running spark app in standalone cluster. My app cr

[jira] [Updated] (SPARK-16667) Spark driver executor dont release unused memory

2016-07-21 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-16667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luis Angel Hernández Acosta updated SPARK-16667: Description: I'm running spark app in standalone cluster. My app cr

[jira] [Updated] (SPARK-16666) Kryo encoder for custom complex classes

2016-07-21 Thread Sam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam updated SPARK-1: Description: I'm trying to create a dataset with some geo data using spark and esri. If `Foo` only have `Point` fi

[jira] [Updated] (SPARK-16667) Spark driver executor dont release unused memory

2016-07-21 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-16667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luis Angel Hernández Acosta updated SPARK-16667: Component/s: GraphX > Spark driver executor dont release unused mem

[jira] [Created] (SPARK-16667) Spark driver executor dont release unused memory

2016-07-21 Thread JIRA
Luis Angel Hernández Acosta created SPARK-16667: --- Summary: Spark driver executor dont release unused memory Key: SPARK-16667 URL: https://issues.apache.org/jira/browse/SPARK-16667 Project

[jira] [Updated] (SPARK-16666) Kryo encoder for custom complex classes

2016-07-21 Thread Sam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam updated SPARK-1: Description: I'm trying to create a dataset with some geo data using spark and esri. If `Foo` only have `Point` fi

[jira] [Updated] (SPARK-16666) Kryo encoder for custom complex classes

2016-07-21 Thread Sam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam updated SPARK-1: Description: I'm trying to create a dataset with some geo data using spark and esri. If `Foo` only have `Point` fi

[jira] [Created] (SPARK-16666) Kryo encoder for custom complex classes

2016-07-21 Thread Sam (JIRA)
Sam created SPARK-1: --- Summary: Kryo encoder for custom complex classes Key: SPARK-1 URL: https://issues.apache.org/jira/browse/SPARK-1 Project: Spark Issue Type: Question Componen

[jira] [Commented] (SPARK-16665) python import pyspark fails in context.py

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15387909#comment-15387909 ] Apache Spark commented on SPARK-16665: -- User 'eastlondoner' has created a pull reque

[jira] [Assigned] (SPARK-16665) python import pyspark fails in context.py

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16665: Assignee: (was: Apache Spark) > python import pyspark fails in context.py > -

[jira] [Commented] (SPARK-16665) python import pyspark fails in context.py

2016-07-21 Thread Andrew Jefferson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15387908#comment-15387908 ] Andrew Jefferson commented on SPARK-16665: -- Pull Request here: https://github.co

[jira] [Assigned] (SPARK-16665) python import pyspark fails in context.py

2016-07-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16665: Assignee: Apache Spark > python import pyspark fails in context.py >

[jira] [Created] (SPARK-16665) python import pyspark fails in context.py

2016-07-21 Thread Andrew Jefferson (JIRA)
Andrew Jefferson created SPARK-16665: Summary: python import pyspark fails in context.py Key: SPARK-16665 URL: https://issues.apache.org/jira/browse/SPARK-16665 Project: Spark Issue Type

[jira] [Commented] (SPARK-12378) CREATE EXTERNAL TABLE AS SELECT EXPORT AWS S3 ERROR

2016-07-21 Thread Chandana Sapparapu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15387889#comment-15387889 ] Chandana Sapparapu commented on SPARK-12378: Environment: Spark 1.6.2 AWS EMR

[jira] [Commented] (SPARK-16664) Spark 1.6.2 - Persist call on Data frames with more than 200 columns is wiping out the data.

2016-07-21 Thread Satish Kolli (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15387858#comment-15387858 ] Satish Kolli commented on SPARK-16664: -- Here is a demonstration from the spark shell

  1   2   >