[jira] [Commented] (SPARK-19644) Memory leak in Spark Streaming

2017-06-18 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053494#comment-16053494 ] Dongjoon Hyun commented on SPARK-19644: --- Hi, [~deenbandhu]. Could you try this with

[jira] [Commented] (SPARK-19644) Memory leak in Spark Streaming

2017-06-18 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053509#comment-16053509 ] Dongjoon Hyun commented on SPARK-19644: --- I see. Thank you anyway, [~deenbandhu]. >

[jira] [Commented] (SPARK-19809) NullPointerException on empty ORC file

2017-06-19 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16054108#comment-16054108 ] Dongjoon Hyun commented on SPARK-19809: --- Yep. I'm trying to fix this with new ORC d

[jira] [Created] (SPARK-21247) Allow case-insensitive type comparisions in Set operation

2017-06-28 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-21247: - Summary: Allow case-insensitive type comparisions in Set operation Key: SPARK-21247 URL: https://issues.apache.org/jira/browse/SPARK-21247 Project: Spark I

[jira] [Updated] (SPARK-21247) Allow case-insensitive type equality in Set operation

2017-06-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21247: -- Summary: Allow case-insensitive type equality in Set operation (was: Allow case-insensitive ty

[jira] [Commented] (SPARK-20256) Fail to start SparkContext/SparkSession with Hive support enabled when user does not have read/write privilege to Hive metastore warehouse dir

2017-07-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071414#comment-16071414 ] Dongjoon Hyun commented on SPARK-20256: --- Hi, [~xwu0226]. Are you still preparing a

[jira] [Created] (SPARK-21278) Upgrade to Py4J 0.10.5

2017-07-01 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-21278: - Summary: Upgrade to Py4J 0.10.5 Key: SPARK-21278 URL: https://issues.apache.org/jira/browse/SPARK-21278 Project: Spark Issue Type: Bug Components

[jira] [Reopened] (SPARK-20946) simplify the config setting logic in SparkSession.getOrCreate

2017-07-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-20946: --- Hi, All. I'll reopen this issue since this is reverted from master and branch-2.2. > simplify th

[jira] [Updated] (SPARK-20946) simplify the config setting logic in SparkSession.getOrCreate

2017-07-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20946: -- Fix Version/s: (was: 2.2.0) > simplify the config setting logic in SparkSession.getOrCreate

[jira] [Commented] (SPARK-21278) Upgrade to Py4J 0.10.5

2017-07-03 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073097#comment-16073097 ] Dongjoon Hyun commented on SPARK-21278: --- Due to [another Py4J bug|https://github.co

[jira] [Commented] (SPARK-21278) Upgrade to Py4J 0.10.6

2017-07-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073749#comment-16073749 ] Dongjoon Hyun commented on SPARK-21278: --- 0.10.6 will be release at the end of this

[jira] [Updated] (SPARK-21278) Upgrade to Py4J 0.10.6

2017-07-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21278: -- Summary: Upgrade to Py4J 0.10.6 (was: Upgrade to Py4J 0.10.5) > Upgrade to Py4J 0.10.6 > -

[jira] [Commented] (SPARK-20946) simplify the config setting logic in SparkSession.getOrCreate

2017-07-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073907#comment-16073907 ] Dongjoon Hyun commented on SPARK-20946: --- Hi, [~cloud_fan]. How do we handle this is

[jira] [Updated] (SPARK-21303) Web-UI shows some Jobs get stuck randomly and stays like that. Neither able to kill

2017-07-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21303: -- Target Version/s: (was: 2.1.1) > Web-UI shows some Jobs get stuck randomly and stays like tha

[jira] [Updated] (SPARK-21303) Web-UI shows some Jobs get stuck randomly and stays like that. Neither able to kill

2017-07-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21303: -- Fix Version/s: (was: 2.1.1) > Web-UI shows some Jobs get stuck randomly and stays like that

[jira] [Updated] (SPARK-21247) Type comparision should respect case-sensitive SQL conf

2017-07-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21247: -- Description: Spark supports case-sensitivity in columns. Especially, for Struct types, with ca

[jira] [Updated] (SPARK-21247) Type comparision should respect case-sensitive SQL conf

2017-07-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21247: -- Summary: Type comparision should respect case-sensitive SQL conf (was: Allow case-insensitive

[jira] [Commented] (SPARK-21316) Dataset Union output is not consistent with the column sequence

2017-07-05 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16075129#comment-16075129 ] Dongjoon Hyun commented on SPARK-21316: --- Union assumes the schema ordering are the

[jira] [Updated] (SPARK-21316) Dataset Union output is not consistent with the column sequence

2017-07-05 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21316: -- Priority: Major (was: Critical) > Dataset Union output is not consistent with the column seque

[jira] [Commented] (SPARK-21316) Dataset Union output is not consistent with the column sequence

2017-07-05 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16075133#comment-16075133 ] Dongjoon Hyun commented on SPARK-21316: --- For your case, could you try the following

[jira] [Commented] (SPARK-21316) Dataset Union output is not consistent with the column sequence

2017-07-06 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076900#comment-16076900 ] Dongjoon Hyun commented on SPARK-21316: --- [~skp33]. It's not `specified in alphabeti

[jira] [Created] (SPARK-21345) SparkSessionBuilderSuite should clean up stopped sessions

2017-07-07 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-21345: - Summary: SparkSessionBuilderSuite should clean up stopped sessions Key: SPARK-21345 URL: https://issues.apache.org/jira/browse/SPARK-21345 Project: Spark I

[jira] [Updated] (SPARK-21345) SparkSessionBuilderSuite should clean up stopped sessions

2017-07-07 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21345: -- Component/s: Tests > SparkSessionBuilderSuite should clean up stopped sessions > --

[jira] [Updated] (SPARK-21344) BinaryType comparison does signed byte array comparison

2017-07-07 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21344: -- Priority: Major (was: Blocker) > BinaryType comparison does signed byte array comparison > ---

[jira] [Commented] (SPARK-21344) BinaryType comparison does signed byte array comparison

2017-07-07 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078825#comment-16078825 ] Dongjoon Hyun commented on SPARK-21344: --- Hi, [~shubhamc]. Thank you for reporting.

[jira] [Updated] (SPARK-21344) BinaryType comparison does signed byte array comparison

2017-07-07 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21344: -- Affects Version/s: 2.0.0 > BinaryType comparison does signed byte array comparison > --

[jira] [Commented] (SPARK-21344) BinaryType comparison does signed byte array comparison

2017-07-07 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078827#comment-16078827 ] Dongjoon Hyun commented on SPARK-21344: --- Also, this cannot be blocker because Spark

[jira] [Commented] (SPARK-21341) Spark 2.1.1: I want to be able to serialize wordVectors on Word2VecModel

2017-07-07 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078832#comment-16078832 ] Dongjoon Hyun commented on SPARK-21341: --- Hi, [~zsellami]. For questions, you had be

[jira] [Commented] (SPARK-21344) BinaryType comparison does signed byte array comparison

2017-07-08 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079017#comment-16079017 ] Dongjoon Hyun commented on SPARK-21344: --- +1 ! :) > BinaryType comparison does sign

[jira] [Created] (SPARK-21349) Make TASK_SIZE_TO_WARN_KB configurable

2017-07-08 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-21349: - Summary: Make TASK_SIZE_TO_WARN_KB configurable Key: SPARK-21349 URL: https://issues.apache.org/jira/browse/SPARK-21349 Project: Spark Issue Type: Improvem

[jira] [Updated] (SPARK-21349) Make TASK_SIZE_TO_WARN_KB configurable

2017-07-08 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21349: -- Description: Although this is just a warning message, this issue tries to make `TASK_SIZE_TO_WA

[jira] [Updated] (SPARK-21349) Make TASK_SIZE_TO_WARN_KB configurable

2017-07-09 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21349: -- Description: Since Spark 1.1.0, Spark emits warning when task size exceeds a threshold, SPARK-2

[jira] [Commented] (SPARK-21349) Make TASK_SIZE_TO_WARN_KB configurable

2017-07-09 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079728#comment-16079728 ] Dongjoon Hyun commented on SPARK-21349: --- Thank you for advice, [~kayousterhout]! Fo

[jira] [Updated] (SPARK-21349) Make TASK_SIZE_TO_WARN_KB configurable

2017-07-09 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21349: -- Description: Since Spark 1.1.0, Spark emits warning when task size exceeds a threshold, SPARK-

[jira] [Commented] (SPARK-21349) Make TASK_SIZE_TO_WARN_KB configurable

2017-07-09 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079751#comment-16079751 ] Dongjoon Hyun commented on SPARK-21349: --- This issue is not about blindly raising th

[jira] [Comment Edited] (SPARK-21349) Make TASK_SIZE_TO_WARN_KB configurable

2017-07-09 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079751#comment-16079751 ] Dongjoon Hyun edited comment on SPARK-21349 at 7/9/17 11:28 PM: ---

[jira] [Comment Edited] (SPARK-21349) Make TASK_SIZE_TO_WARN_KB configurable

2017-07-09 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079751#comment-16079751 ] Dongjoon Hyun edited comment on SPARK-21349 at 7/9/17 11:29 PM: ---

[jira] [Commented] (SPARK-21349) Make TASK_SIZE_TO_WARN_KB configurable

2017-07-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16080817#comment-16080817 ] Dongjoon Hyun commented on SPARK-21349: --- I usually saw 200K~300K. And, the follow

[jira] [Comment Edited] (SPARK-21349) Make TASK_SIZE_TO_WARN_KB configurable

2017-07-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16080890#comment-16080890 ] Dongjoon Hyun edited comment on SPARK-21349 at 7/10/17 7:04 PM: ---

[jira] [Commented] (SPARK-21349) Make TASK_SIZE_TO_WARN_KB configurable

2017-07-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16080890#comment-16080890 ] Dongjoon Hyun commented on SPARK-21349: --- Thank you, [~shivaram] and @Kay Ousterhout

[jira] [Comment Edited] (SPARK-21367) R older version of Roxygen2 on Jenkins

2017-07-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081173#comment-16081173 ] Dongjoon Hyun edited comment on SPARK-21367 at 7/10/17 9:21 PM: ---

[jira] [Reopened] (SPARK-17044) Add window function test in SQLQueryTestSuite

2017-07-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-17044: --- > Add window function test in SQLQueryTestSuite > - >

[jira] [Closed] (SPARK-17044) Add window function test in SQLQueryTestSuite

2017-07-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-17044. - Resolution: Duplicate > Add window function test in SQLQueryTestSuite > -

[jira] [Commented] (SPARK-21367) R older version of Roxygen2 on Jenkins

2017-07-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081173#comment-16081173 ] Dongjoon Hyun commented on SPARK-21367: --- Hi, All. After this, many builds seems to

[jira] [Commented] (SPARK-21367) R older version of Roxygen2 on Jenkins

2017-07-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081274#comment-16081274 ] Dongjoon Hyun commented on SPARK-21367: --- Thank you, [~shaneknapp]. It seems to be `

[jira] [Commented] (SPARK-21362) Add JDBCDialect for Apache Drill

2017-07-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081377#comment-16081377 ] Dongjoon Hyun commented on SPARK-21362: --- Hi, [~radford1] Usually, in Spark communit

[jira] [Updated] (SPARK-20901) Feature parity for ORC with Parquet

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20901: -- Affects Version/s: 2.2.0 > Feature parity for ORC with Parquet > --

[jira] [Updated] (SPARK-20682) Support a new faster ORC data source based on Apache ORC

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20682: -- Affects Version/s: 2.2.0 > Support a new faster ORC data source based on Apache ORC > -

[jira] [Closed] (SPARK-21379) skip.header.line.count is ignored in HiveContext

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-21379. - Resolution: Won't Fix Hi, [~spurusotha...@uk.imshealth.com]. This is a duplication of the previou

[jira] [Commented] (SPARK-21379) skip.header.line.count is ignored in HiveContext

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083186#comment-16083186 ] Dongjoon Hyun commented on SPARK-21379: --- This is the PR and the discussion in the c

[jira] [Updated] (SPARK-21379) skip.header.line.count is ignored in HiveContext

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21379: -- Component/s: (was: Spark Core) SQL > skip.header.line.count is ignored in

[jira] [Commented] (SPARK-21380) Join with Columns thinks inner join is cross join even when aliased

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083230#comment-16083230 ] Dongjoon Hyun commented on SPARK-21380: --- Hi, [~everett]. Thank you for reporting. T

[jira] [Commented] (SPARK-21380) Join with Columns thinks inner join is cross join even when aliased

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083266#comment-16083266 ] Dongjoon Hyun commented on SPARK-21380: --- Hi, [~everett]. It's the correct result of

[jira] [Closed] (SPARK-21380) Join with Columns thinks inner join is cross join even when aliased

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-21380. - Resolution: Not A Problem I'm closing this issue for now since the pattern works correct. If you

[jira] [Commented] (SPARK-21380) Join with Columns thinks inner join is cross join even when aliased

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083272#comment-16083272 ] Dongjoon Hyun commented on SPARK-21380: --- Your case are too simple, so it's optimize

[jira] [Comment Edited] (SPARK-21380) Join with Columns thinks inner join is cross join even when aliased

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083434#comment-16083434 ] Dongjoon Hyun edited comment on SPARK-21380 at 7/12/17 4:39 AM: ---

[jira] [Commented] (SPARK-21380) Join with Columns thinks inner join is cross join even when aliased

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083434#comment-16083434 ] Dongjoon Hyun commented on SPARK-21380: --- One row in real normal table is okay. Your

[jira] [Commented] (SPARK-21380) Join with Columns thinks inner join is cross join even when aliased

2017-07-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084274#comment-16084274 ] Dongjoon Hyun commented on SPARK-21380: --- I see. I agree your point about that warni

[jira] [Commented] (SPARK-21392) Unable to infer schema when loading Parquet file

2017-07-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084707#comment-16084707 ] Dongjoon Hyun commented on SPARK-21392: --- It seems to be a different issue. SPARK-16

[jira] [Created] (SPARK-21422) Depend on Apache ORC 1.4.0

2017-07-14 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-21422: - Summary: Depend on Apache ORC 1.4.0 Key: SPARK-21422 URL: https://issues.apache.org/jira/browse/SPARK-21422 Project: Spark Issue Type: Bug Compon

[jira] [Updated] (SPARK-21422) Depend on Apache ORC 1.4.0

2017-07-14 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21422: -- Description: Like Parquet, this issue aims to depend on the latest Apache ORC 1.4 for Apache S

[jira] [Updated] (SPARK-21422) Depend on Apache ORC 1.4.0

2017-07-14 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21422: -- Description: Like Parquet, this issue aims to depend on the latest Apache ORC 1.4 for Apache S

[jira] [Updated] (SPARK-13860) TPCDS query 39 returns wrong results compared to TPC official result set

2017-07-14 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-13860: -- Affects Version/s: 2.2.0 2.1.1 > TPCDS query 39 returns wrong results co

[jira] [Commented] (SPARK-13860) TPCDS query 39 returns wrong results compared to TPC official result set

2017-07-14 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088305#comment-16088305 ] Dongjoon Hyun commented on SPARK-13860: --- I confirmed that the behavior of Spark 2.1

[jira] [Comment Edited] (SPARK-13860) TPCDS query 39 returns wrong results compared to TPC official result set

2017-07-14 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088305#comment-16088305 ] Dongjoon Hyun edited comment on SPARK-13860 at 7/14/17 11:52 PM: --

[jira] [Commented] (SPARK-21316) Dataset Union output is not consistent with the column sequence

2017-07-18 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092607#comment-16092607 ] Dongjoon Hyun commented on SPARK-21316: --- +1 > Dataset Union output is not consiste

[jira] [Commented] (SPARK-21573) Tests failing with run-tests.py SyntaxError occasionally in Jenkins

2017-07-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16106029#comment-16106029 ] Dongjoon Hyun commented on SPARK-21573: --- Thank you for pinging me, [~hyukjin.kwon].

[jira] [Closed] (SPARK-21574) set hive.exec.max.dynamic.partitions lose effect

2017-07-30 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-21574. - Resolution: Duplicate Hi, [~q79969786]. Thank you for report, but this is a duplicated issue of S

[jira] [Created] (SPARK-21578) Consolidate redundant SparkContext constructors due to SI-8479

2017-07-30 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-21578: - Summary: Consolidate redundant SparkContext constructors due to SI-8479 Key: SPARK-21578 URL: https://issues.apache.org/jira/browse/SPARK-21578 Project: Spark

[jira] [Commented] (SPARK-21578) Consolidate redundant SparkContext constructors due to SI-8479

2017-07-30 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16106679#comment-16106679 ] Dongjoon Hyun commented on SPARK-21578: --- I made a PR for this. - https://github.com

[jira] [Updated] (SPARK-21578) Add JavaSparkContextSuite

2017-07-31 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21578: -- Description: Due to SI-8479, [SPARK-1093|https://issues.apache.org/jira/browse/SPARK-21578] in

[jira] [Updated] (SPARK-21578) Add JavaSparkContextSuite

2017-07-31 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21578: -- Summary: Add JavaSparkContextSuite (was: Consolidate redundant SparkContext constructors due t

[jira] [Reopened] (SPARK-16208) Add `CollapseEmptyPlan` optimizer

2016-06-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-16208: --- I made a new PR. > Add `CollapseEmptyPlan` optimizer > - > >

[jira] [Commented] (SPARK-16208) Add `CollapseEmptyPlan` optimizer

2016-06-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15349983#comment-15349983 ] Dongjoon Hyun commented on SPARK-16208: --- New PR is in the process of review. > Add

[jira] [Commented] (SPARK-16211) DataFrame filter is buggy when used with "and"

2016-06-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350026#comment-15350026 ] Dongjoon Hyun commented on SPARK-16211: --- Hi, [~renatbek], [~srowen], [~hvanhovell].

[jira] [Commented] (SPARK-16210) DataFrame.drop(colName) fails if another column has a period in its name

2016-06-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350034#comment-15350034 ] Dongjoon Hyun commented on SPARK-16210: --- I also tested this. Unfortunately Spark 1

[jira] [Commented] (SPARK-16172) SQL Context's

2016-06-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350037#comment-15350037 ] Dongjoon Hyun commented on SPARK-16172: --- Oh, it's an issue about 1.2.X. Interesting

[jira] [Commented] (SPARK-16172) SQL Context's

2016-06-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350039#comment-15350039 ] Dongjoon Hyun commented on SPARK-16172: --- Otherwise, you can ask this for latest Spa

[jira] [Commented] (SPARK-10591) False negative in QueryTest.checkAnswer

2016-06-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350065#comment-15350065 ] Dongjoon Hyun commented on SPARK-10591: --- Hi, [~lian cheng]. When I check this today

[jira] [Commented] (SPARK-16111) Hide SparkOrcNewRecordReader in API docs

2016-06-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350092#comment-15350092 ] Dongjoon Hyun commented on SPARK-16111: --- Hi, [~mengxr]. Could you review this PR?

[jira] [Commented] (SPARK-16211) DataFrame filter is buggy when used with "and"

2016-06-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350294#comment-15350294 ] Dongjoon Hyun commented on SPARK-16211: --- Great! :) > DataFrame filter is buggy whe

[jira] [Created] (SPARK-16221) Redirect Parquet JUL logger via SLF4J for WRITE operations

2016-06-26 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-16221: - Summary: Redirect Parquet JUL logger via SLF4J for WRITE operations Key: SPARK-16221 URL: https://issues.apache.org/jira/browse/SPARK-16221 Project: Spark

[jira] [Updated] (SPARK-16052) Improve `CollapseRepartition` optimizer for Repartition/RepartitionBy

2016-06-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-16052: -- Description: This issue improves `CollapseRepartition` to optimize the adjacent combinations o

[jira] [Commented] (SPARK-16228) "Percentile" needs explicit cast to double

2016-06-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351708#comment-15351708 ] Dongjoon Hyun commented on SPARK-16228: --- Hi, [~epahomov] and [~srowen]. The root ca

[jira] [Commented] (SPARK-16233) test_sparkSQL.R is failing

2016-06-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353388#comment-15353388 ] Dongjoon Hyun commented on SPARK-16233: --- Oh, thank you for pinging me. For this iss

[jira] [Comment Edited] (SPARK-16233) test_sparkSQL.R is failing

2016-06-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353405#comment-15353405 ] Dongjoon Hyun edited comment on SPARK-16233 at 6/28/16 5:20 PM: ---

[jira] [Comment Edited] (SPARK-16233) test_sparkSQL.R is failing

2016-06-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353405#comment-15353405 ] Dongjoon Hyun edited comment on SPARK-16233 at 6/28/16 5:20 PM: ---

[jira] [Commented] (SPARK-16233) test_sparkSQL.R is failing

2016-06-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353405#comment-15353405 ] Dongjoon Hyun commented on SPARK-16233: --- Hi, [~shivaram]. Actually, in `DataFrameRe

[jira] [Commented] (SPARK-16252) Full Outer join with literal column results in incorrect result

2016-06-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353431#comment-15353431 ] Dongjoon Hyun commented on SPARK-16252: --- Hi, this is fixed now. {code} >>> df1.join

[jira] [Commented] (SPARK-16252) Full Outer join with literal column results in incorrect result

2016-06-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353433#comment-15353433 ] Dongjoon Hyun commented on SPARK-16252: --- I think we can close this issue as `DUPLIC

[jira] [Commented] (SPARK-16181) Incorrect behavior for isNull filter

2016-06-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353434#comment-15353434 ] Dongjoon Hyun commented on SPARK-16181: --- Hi, [~yhuai]. Could you check and close SP

[jira] [Updated] (SPARK-16174) Improve `OptimizeIn` optimizer to remove deterministic repetitions

2016-06-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-16174: -- Summary: Improve `OptimizeIn` optimizer to remove deterministic repetitions (was: Improve Opti

[jira] [Commented] (SPARK-16006) Attemping to write empty DataFrame with no fields throw non-intuitive exception

2016-06-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353914#comment-15353914 ] Dongjoon Hyun commented on SPARK-16006: --- Hi, [~tdas]. Could you review the PR again

[jira] [Updated] (SPARK-16267) Replace deprecated `CREATE TEMPORARY TABLE` from testsuites

2016-06-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-16267: -- Summary: Replace deprecated `CREATE TEMPORARY TABLE` from testsuites (was: Replace deprecate `

[jira] [Created] (SPARK-16267) Remove deprecate `CREATE TEMPORARY TABLE` from testsuites

2016-06-28 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-16267: - Summary: Remove deprecate `CREATE TEMPORARY TABLE` from testsuites Key: SPARK-16267 URL: https://issues.apache.org/jira/browse/SPARK-16267 Project: Spark I

[jira] [Updated] (SPARK-16267) Replace deprecate `CREATE TEMPORARY TABLE` from testsuites

2016-06-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-16267: -- Summary: Replace deprecate `CREATE TEMPORARY TABLE` from testsuites (was: Remove deprecate `CR

[jira] [Updated] (SPARK-16267) Replace deprecated `CREATE TEMPORARY TABLE` from testsuites

2016-06-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-16267: -- Description: After SPARK-15674, `DDLStrategy` prints out the following deprecation messages in

[jira] [Commented] (SPARK-16275) Implement all the Hive fallback functions

2016-06-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354617#comment-15354617 ] Dongjoon Hyun commented on SPARK-16275: --- Sure. It's my pleasure! > Implement all t

[jira] [Commented] (SPARK-16278) Implement map_keys SQL function

2016-06-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354627#comment-15354627 ] Dongjoon Hyun commented on SPARK-16278: --- I'll work on this issue. > Implement map_

[jira] [Commented] (SPARK-16279) Implement map_values SQL function

2016-06-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354646#comment-15354646 ] Dongjoon Hyun commented on SPARK-16279: --- I'll work on this together with SPARK-1627

  1   2   3   4   5   6   7   8   9   10   >