[jira] [Commented] (SPARK-46934) Unable to create Hive View from certain Spark Dataframe StructType

2024-02-20 Thread Yu-Ting LIN (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819062#comment-17819062 ] Yu-Ting LIN commented on SPARK-46934: - [~dongjoon] As I have mentioned before, we ar

[jira] [Commented] (SPARK-46934) Unable to create Hive View from certain Spark Dataframe StructType

2024-02-19 Thread Yu-Ting LIN (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17818633#comment-17818633 ] Yu-Ting LIN commented on SPARK-46934: - [~dongjoon] What do you mean for regression ?

[jira] [Commented] (SPARK-46934) Unable to create Hive View from certain Spark Dataframe StructType

2024-02-19 Thread Yu-Ting LIN (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17818629#comment-17818629 ] Yu-Ting LIN commented on SPARK-46934: - [~dongjoon] we are mainly using Spark 3.3 and

[jira] [Commented] (SPARK-46934) Unable to create Hive View from certain Spark Dataframe StructType

2024-02-04 Thread Yu-Ting LIN (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814184#comment-17814184 ] Yu-Ting LIN commented on SPARK-46934: - [~yao] Thanks. One more question about this h

[jira] [Created] (SPARK-46934) Unable to create Hive View from certain Spark Dataframe StructType

2024-01-31 Thread Yu-Ting LIN (Jira)
Yu-Ting LIN created SPARK-46934: --- Summary: Unable to create Hive View from certain Spark Dataframe StructType Key: SPARK-46934 URL: https://issues.apache.org/jira/browse/SPARK-46934 Project: Spark

[jira] [Updated] (SPARK-41313) Combine fixes for SPARK-3900 and SPARK-21138

2022-11-29 Thread Xing Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xing Lin updated SPARK-41313: - Description: spark-3900 fixed the illegalStateException in cleanupStagingDir in ApplicationMaster's shu

[jira] [Updated] (SPARK-41313) Combine fixes for SPARK-3900 and SPARK-21138

2022-11-28 Thread Xing Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xing Lin updated SPARK-41313: - Description: spark-3900 fixed the illegalStateException in cleanupStagingDir in ApplicationMaster's shut

[jira] [Updated] (SPARK-41313) Combine fixes for SPARK-3900 and SPARK-21138

2022-11-28 Thread Xing Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xing Lin updated SPARK-41313: - Description: spark-3900 fixed the illegalStateException in cleanupStagingDir in ApplicationMaster's shut

[jira] [Updated] (SPARK-41313) Combine fixes for SPARK-3900 and SPARK-21138

2022-11-28 Thread Xing Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xing Lin updated SPARK-41313: - Summary: Combine fixes for SPARK-3900 and SPARK-21138 (was: Combine fix for SPARK-3900 and SPARK-21138)

[jira] [Created] (SPARK-41313) Combine fix for SPARK-3900 and SPARK-21138

2022-11-28 Thread Xing Lin (Jira)
Xing Lin created SPARK-41313: Summary: Combine fix for SPARK-3900 and SPARK-21138 Key: SPARK-41313 URL: https://issues.apache.org/jira/browse/SPARK-41313 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-35304) [k8s] Though finishing a job, the driver pod is running infinitely

2022-08-08 Thread Emilie Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17576866#comment-17576866 ] Emilie Lin commented on SPARK-35304: Hi [~ocworld] do you have any updates for this

[jira] [Commented] (SPARK-28098) Native ORC reader doesn't support subdirectories with Hive tables

2021-04-22 Thread Yu-Tang Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17329120#comment-17329120 ] Yu-Tang Lin commented on SPARK-28098: - Hi [~ddrinka], I already made a PR to resolve

[jira] [Commented] (SPARK-34707) Code-gen broadcast nested loop join (left outer/right outer)

2021-03-22 Thread Zebing Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17306519#comment-17306519 ] Zebing Lin commented on SPARK-34707: Created PR https://github.com/apache/spark/pull

[jira] [Issue Comment Deleted] (SPARK-28263) Spark-submit can not find class (ClassNotFoundException)

2021-01-14 Thread Jack LIN (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack LIN updated SPARK-28263: - Comment: was deleted (was: Just guessing, this probably relates to the settings of `sbt`) > Spark-submi

[jira] [Commented] (SPARK-28263) Spark-submit can not find class (ClassNotFoundException)

2021-01-14 Thread Jack LIN (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264990#comment-17264990 ] Jack LIN commented on SPARK-28263: -- Just guessing, this probably relates to the setting

[jira] [Commented] (SPARK-28861) Jetty property handling: java.lang.NumberFormatException: For input string: "unknown".

2020-05-31 Thread John Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120669#comment-17120669 ] John Lin commented on SPARK-28861: -- Hi, I can reproduce this problem. The full stack t

[jira] [Updated] (SPARK-30511) Spark marks intentionally killed speculative tasks as pending leads to holding idle executors

2020-01-16 Thread Zebing Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zebing Lin updated SPARK-30511: --- Description: *TL;DR* When speculative tasks fail/get killed, they are still considered as pending

[jira] [Updated] (SPARK-30511) Spark marks intentionally killed speculative tasks as pending leads to holding idle executors

2020-01-16 Thread Zebing Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zebing Lin updated SPARK-30511: --- Summary: Spark marks intentionally killed speculative tasks as pending leads to holding idle executo

[jira] [Comment Edited] (SPARK-28403) Executor Allocation Manager can add an extra executor when speculative tasks

2020-01-15 Thread Zebing Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17016316#comment-17016316 ] Zebing Lin edited comment on SPARK-28403 at 1/15/20 8:53 PM: -

[jira] [Commented] (SPARK-28403) Executor Allocation Manager can add an extra executor when speculative tasks

2020-01-15 Thread Zebing Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17016316#comment-17016316 ] Zebing Lin commented on SPARK-28403: In our production, this just caused a fluctuati

[jira] [Updated] (SPARK-30511) Spark marks ended speculative tasks as pending leads to holding idle executors

2020-01-15 Thread Zebing Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zebing Lin updated SPARK-30511: --- Description: *TL;DR* When speculative tasks finished/failed/got killed, they are still considered

[jira] [Updated] (SPARK-30511) Spark marks ended speculative tasks as pending leads to holding idle executors

2020-01-15 Thread Zebing Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zebing Lin updated SPARK-30511: --- Description: *TL;DR* When speculative tasks finished/failed/got killed, they are still considered

[jira] [Updated] (SPARK-30511) Spark marks ended speculative tasks as pending leads to holding idle executors

2020-01-15 Thread Zebing Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zebing Lin updated SPARK-30511: --- Attachment: Screen Shot 2020-01-15 at 11.13.17.png > Spark marks ended speculative tasks as pending

[jira] [Updated] (SPARK-30511) Spark marks ended speculative tasks as pending leads to holding idle executors

2020-01-15 Thread Zebing Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zebing Lin updated SPARK-30511: --- Description: *TL;DR* When speculative tasks finished/failed/got killed, they are still considered

[jira] [Updated] (SPARK-30511) Spark marks ended speculative tasks as pending leads to holding idle executors

2020-01-15 Thread Zebing Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zebing Lin updated SPARK-30511: --- Description: *TL;DR* When speculative tasks finished/failed/got killed, they are still considered

[jira] [Updated] (SPARK-30511) Spark marks ended speculative tasks as pending leads to holding idle executors

2020-01-15 Thread Zebing Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zebing Lin updated SPARK-30511: --- Description: *TL;DR* When speculative tasks finished/failed/got killed, they are still considered

[jira] [Updated] (SPARK-30511) Spark marks ended speculative tasks as pending leads to holding idle executors

2020-01-14 Thread Zebing Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zebing Lin updated SPARK-30511: --- External issue ID: (was: SPARK-2840) > Spark marks ended speculative tasks as pending leads to hol

[jira] [Updated] (SPARK-30511) Spark marks ended speculative tasks as pending leads to holding idle executors

2020-01-14 Thread Zebing Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zebing Lin updated SPARK-30511: --- Description: *TL;DR* When speculative tasks finished/failed/got killed, they are still considered

[jira] [Updated] (SPARK-30511) Spark marks ended speculative tasks as pending leads to holding idle executors

2020-01-14 Thread Zebing Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zebing Lin updated SPARK-30511: --- External issue ID: SPARK-2840 > Spark marks ended speculative tasks as pending leads to holding idle

[jira] [Created] (SPARK-30511) Spark marks ended speculative tasks as pending leads to holding idle executors

2020-01-14 Thread Zebing Lin (Jira)
Zebing Lin created SPARK-30511: -- Summary: Spark marks ended speculative tasks as pending leads to holding idle executors Key: SPARK-30511 URL: https://issues.apache.org/jira/browse/SPARK-30511 Project: S

[jira] [Updated] (SPARK-29667) implicitly convert mismatched datatypes on right side of "IN" operator

2019-10-30 Thread Jessie Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jessie Lin updated SPARK-29667: --- Priority: Minor (was: Major) > implicitly convert mismatched datatypes on right side of "IN" operat

[jira] [Created] (SPARK-29667) implicitly convert mismatched datatypes on right side of "IN" operator

2019-10-30 Thread Jessie Lin (Jira)
Jessie Lin created SPARK-29667: -- Summary: implicitly convert mismatched datatypes on right side of "IN" operator Key: SPARK-29667 URL: https://issues.apache.org/jira/browse/SPARK-29667 Project: Spark

[jira] [Commented] (SPARK-29548) Redirect system print stream to log4j and improve robustness

2019-10-22 Thread Ching Lin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956885#comment-16956885 ] Ching Lin commented on SPARK-29548: --- how about using checkpoint instead of log4j ? >

[jira] [Comment Edited] (SPARK-27718) incorrect result from pagerank

2019-05-15 Thread De-En Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16840945#comment-16840945 ] De-En Lin edited comment on SPARK-27718 at 5/16/19 2:55 AM:

[jira] [Commented] (SPARK-27718) incorrect result from pagerank

2019-05-15 Thread De-En Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16840945#comment-16840945 ] De-En Lin commented on SPARK-27718: --- In wiki, the equation of PageRank is as follows:

[jira] [Updated] (SPARK-27718) incorrect result from pagerank

2019-05-15 Thread De-En Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] De-En Lin updated SPARK-27718: -- Attachment: 螢幕快照 2019-05-16 上午10.09.45.png > incorrect result from pagerank >

[jira] [Updated] (SPARK-27718) incorrect result from pagerank

2019-05-15 Thread De-En Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] De-En Lin updated SPARK-27718: -- External issue URL: https://github.com/apache/spark/pull/24612 > incorrect result from pagerank >

[jira] [Updated] (SPARK-27718) incorrect result from pagerank

2019-05-14 Thread De-En Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] De-En Lin updated SPARK-27718: -- Description: When I executed /examples/src/main/python/pagerank.py  The result is shown as follows  

[jira] [Created] (SPARK-27718) incorrect result from pagerank

2019-05-14 Thread De-En Lin (JIRA)
De-En Lin created SPARK-27718: - Summary: incorrect result from pagerank Key: SPARK-27718 URL: https://issues.apache.org/jira/browse/SPARK-27718 Project: Spark Issue Type: Bug Components

[jira] [Commented] (SPARK-27375) cache not working after discretizer.fit(df).transform(df)

2019-04-04 Thread Zhenyi Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16809978#comment-16809978 ] Zhenyi Lin commented on SPARK-27375: Good to know that it is good in higher version.

[jira] [Updated] (SPARK-27375) cache not working after discretizer.fit(df).transform(df)

2019-04-03 Thread Zhenyi Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyi Lin updated SPARK-27375: --- Summary: cache not working after discretizer.fit(df).transform(df) (was: cache not working after di

[jira] [Updated] (SPARK-27375) cache not working after discretizer.fit(df).transform operation

2019-04-03 Thread Zhenyi Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyi Lin updated SPARK-27375: --- Description: Below gives an example. If cache works, col(r1) should be equal to col(r2) in the outp

[jira] [Updated] (SPARK-27375) cache not working after discretizer.fit(df).transform operation

2019-04-03 Thread Zhenyi Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyi Lin updated SPARK-27375: --- Description: Below gives an example. If cache works, col(r1) should be equal to col(r2) in the outp

[jira] [Updated] (SPARK-27375) cache not working after discretizer.fit(df).transform operation

2019-04-03 Thread Zhenyi Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyi Lin updated SPARK-27375: --- Description: Below gives an example. If cache works, col(r1) in the output should be equal to col(r

[jira] [Updated] (SPARK-27375) cache not working after discretizer.fit(df).transform operation

2019-04-03 Thread Zhenyi Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyi Lin updated SPARK-27375: --- Description: Below gives an example. col(r1) should be equal to col(r2) if cache operation works. H

[jira] [Updated] (SPARK-27375) cache not working after discretizer.fit(df).transform operation

2019-04-03 Thread Zhenyi Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyi Lin updated SPARK-27375: --- Description: Below gives an example. col(r1) should be equal to col(r2) if cache operation works. H

[jira] [Updated] (SPARK-27375) cache not working after discretizer.fit(df).transform operation

2019-04-03 Thread Zhenyi Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyi Lin updated SPARK-27375: --- Summary: cache not working after discretizer.fit(df).transform operation (was: cache not working af

[jira] [Updated] (SPARK-27375) cache not working after discretizer.fit(df).transform

2019-04-03 Thread Zhenyi Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyi Lin updated SPARK-27375: --- Description: Below gives an example. col(r1) should be equal to col(r2) if cache operation works. H

[jira] [Updated] (SPARK-27375) cache not working after discretizer.fit(df).transform

2019-04-03 Thread Zhenyi Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyi Lin updated SPARK-27375: --- Summary: cache not working after discretizer.fit(df).transform (was: cache not working after call d

[jira] [Created] (SPARK-27375) cache not working after call discretizer.fit(df).transform

2019-04-03 Thread Zhenyi Lin (JIRA)
Zhenyi Lin created SPARK-27375: -- Summary: cache not working after call discretizer.fit(df).transform Key: SPARK-27375 URL: https://issues.apache.org/jira/browse/SPARK-27375 Project: Spark Issue

[jira] [Commented] (SPARK-26228) OOM issue encountered when computing Gramian matrix

2018-12-02 Thread Chen Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16706751#comment-16706751 ] Chen Lin commented on SPARK-26228: -- I have tried to set spark.driver.memory from 8g to

[jira] [Updated] (SPARK-26228) OOM issue encountered when computing Gramian matrix

2018-12-02 Thread Chen Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Lin updated SPARK-26228: - Attachment: 1.jpeg > OOM issue encountered when computing Gramian matrix >

[jira] [Commented] (SPARK-26228) OOM issue encountered when computing Gramian matrix

2018-12-02 Thread Chen Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16706706#comment-16706706 ] Chen Lin commented on SPARK-26228: -- Exception in thread "main" java.lang.OutOfMemoryErr

[jira] [Commented] (SPARK-26228) OOM issue encountered when computing Gramian matrix

2018-12-02 Thread Chen Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16706701#comment-16706701 ] Chen Lin commented on SPARK-26228: -- [~shahid] I have upload the screenshot of log.  

[jira] [Updated] (SPARK-26228) OOM issue encountered when computing Gramian matrix

2018-12-02 Thread Chen Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Lin updated SPARK-26228: - Description: {quote}/**  * Computes the Gramian matrix `A^T A`.  *  * @note This cannot be computed o

[jira] [Created] (SPARK-26228) OOM issue encountered when computing Gramian matrix

2018-11-29 Thread Chen Lin (JIRA)
Chen Lin created SPARK-26228: Summary: OOM issue encountered when computing Gramian matrix Key: SPARK-26228 URL: https://issues.apache.org/jira/browse/SPARK-26228 Project: Spark Issue Type: Impr

[jira] [Updated] (SPARK-26228) OOM issue encountered when computing Gramian matrix

2018-11-29 Thread Chen Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Lin updated SPARK-26228: - Description: /**  * Computes the Gramian matrix `A^T A`.  *  * @note This cannot be computed on matri

[jira] [Updated] (SPARK-26228) OOM issue encountered when computing Gramian matrix

2018-11-29 Thread Chen Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Lin updated SPARK-26228: - Description: {quote}/**  * Computes the Gramian matrix `A^T A`.  *  * @note This cannot be computed o

[jira] [Updated] (SPARK-26228) OOM issue encountered when computing Gramian matrix

2018-11-29 Thread Chen Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Lin updated SPARK-26228: - Description: /**  * Computes the Gramian matrix `A^T A`.  *  * @note This cannot be computed on matri

[jira] [Updated] (SPARK-26228) OOM issue encountered when computing Gramian matrix

2018-11-29 Thread Chen Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Lin updated SPARK-26228: - Description: /**  * Computes the Gramian matrix `A^T A`.  *  *@note This cannot be computed on matrice

[jira] [Closed] (SPARK-11574) Spark should support StatsD sink out of box

2017-09-03 Thread Xiaofeng Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaofeng Lin closed SPARK-11574. > Spark should support StatsD sink out of box > --- > >

[jira] [Commented] (SPARK-11574) Spark should support StatsD sink out of box

2017-08-30 Thread Xiaofeng Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148328#comment-16148328 ] Xiaofeng Lin commented on SPARK-11574: -- [~jerryshao], yes my JIRA username is still

[jira] [Created] (SPARK-21505) A dynamic join operator to improve the join reliability

2017-07-21 Thread Lin (JIRA)
Lin created SPARK-21505: --- Summary: A dynamic join operator to improve the join reliability Key: SPARK-21505 URL: https://issues.apache.org/jira/browse/SPARK-21505 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-21005) VectorIndexerModel does not prepare output column field correctly

2017-06-07 Thread Chen Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Lin updated SPARK-21005: - Description: >From my understanding through reading the documentation, VectorIndexer >decides which fea

[jira] [Created] (SPARK-21005) VectorIndexerModel does not prepare output column field correctly

2017-06-07 Thread Chen Lin (JIRA)
Chen Lin created SPARK-21005: Summary: VectorIndexerModel does not prepare output column field correctly Key: SPARK-21005 URL: https://issues.apache.org/jira/browse/SPARK-21005 Project: Spark Is

[jira] [Created] (SPARK-20441) Within the same streaming query, one StreamingRelation should only be transformed to one StreamingExecutionRelation

2017-04-23 Thread Liwei Lin (JIRA)
Liwei Lin created SPARK-20441: - Summary: Within the same streaming query, one StreamingRelation should only be transformed to one StreamingExecutionRelation Key: SPARK-20441 URL: https://issues.apache.org/jira/browse/

[jira] [Commented] (SPARK-20299) NullPointerException when null and string are in a tuple while encoding Dataset

2017-04-17 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15970811#comment-15970811 ] Liwei Lin commented on SPARK-20299: --- hi [~umesh9...@gmail.com], are you planning to wor

[jira] [Updated] (SPARK-7420) Flaky test: o.a.s.streaming.JobGeneratorSuite "Do not clear received block data too soon"

2017-03-19 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-7420: - Component/s: DStreams > Flaky test: o.a.s.streaming.JobGeneratorSuite "Do not clear received block > data

[jira] [Updated] (SPARK-19989) Flaky Test: org.apache.spark.sql.kafka010.KafkaSourceStressSuite

2017-03-19 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-19989: -- Component/s: Structured Streaming > Flaky Test: org.apache.spark.sql.kafka010.KafkaSourceStressSuite >

[jira] [Updated] (SPARK-20002) Add support for unions between streaming and batch datasets

2017-03-19 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-20002: -- Component/s: Structured Streaming > Add support for unions between streaming and batch datasets > -

[jira] [Updated] (SPARK-19932) Disallow a case that might cause OOM for steaming deduplication

2017-03-16 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-19932: -- Summary: Disallow a case that might cause OOM for steaming deduplication (was: Disallow a case that mi

[jira] [Updated] (SPARK-19932) Disallow a case that might case OOM for steaming deduplication

2017-03-16 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-19932: -- Summary: Disallow a case that might case OOM for steaming deduplication (was: Also save event time int

[jira] [Commented] (SPARK-19965) DataFrame batch reader may fail to infer partitions when reading FileStreamSink's output

2017-03-15 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15927431#comment-15927431 ] Liwei Lin commented on SPARK-19965: --- Hi [~zsxwing], are you working on a patch? Mind if

[jira] [Updated] (SPARK-19817) make it clear that `timeZone` option is a general option in DataFrameReader/Writer, DataStreamReader/Writer

2017-03-14 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-19817: -- Summary: make it clear that `timeZone` option is a general option in DataFrameReader/Writer, DataStream

[jira] [Updated] (SPARK-19817) make it clear that `timeZone` option is a general option in DataFrameReader/Writer

2017-03-14 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-19817: -- Component/s: Structured Streaming > make it clear that `timeZone` option is a general option in > Data

[jira] [Updated] (SPARK-19932) Also save event time into StateStore for certain cases

2017-03-12 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-19932: -- Description: {code} spark .readStream // schema: (word, eventTime), like ("a", 10),

[jira] [Created] (SPARK-19932) Also save event time into StateStore for certain cases

2017-03-12 Thread Liwei Lin (JIRA)
Liwei Lin created SPARK-19932: - Summary: Also save event time into StateStore for certain cases Key: SPARK-19932 URL: https://issues.apache.org/jira/browse/SPARK-19932 Project: Spark Issue Type:

[jira] [Commented] (SPARK-19721) Good error message for version mismatch in log files

2017-02-23 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882034#comment-15882034 ] Liwei Lin commented on SPARK-19721: --- I'd like to work on this too. Thanks. > Good erro

[jira] [Commented] (SPARK-19715) Option to Strip Paths in FileSource

2017-02-23 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881965#comment-15881965 ] Liwei Lin commented on SPARK-19715: --- I'll work on this. Thanks! > Option to Strip Path

[jira] [Commented] (SPARK-19633) FileSource read from FileSink

2017-02-17 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15871876#comment-15871876 ] Liwei Lin commented on SPARK-19633: --- Hi [~marmbrus], I'd like to take this if it's ok b

[jira] [Updated] (SPARK-19564) KafkaOffsetReader's consumers should not be in the same group

2017-02-12 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-19564: -- Description: In `KafkaOffsetReader`, when error occurs, we abort the existing consumer and create a ne

[jira] [Created] (SPARK-19564) KafkaOffsetReader's consumers should not be in the same group

2017-02-12 Thread Liwei Lin (JIRA)
Liwei Lin created SPARK-19564: - Summary: KafkaOffsetReader's consumers should not be in the same group Key: SPARK-19564 URL: https://issues.apache.org/jira/browse/SPARK-19564 Project: Spark Issu

[jira] [Comment Edited] (SPARK-19559) Fix flaky KafkaSourceSuite.subscribing topic by pattern with topic deletions

2017-02-12 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15862721#comment-15862721 ] Liwei Lin edited comment on SPARK-19559 at 2/12/17 10:07 AM: -

[jira] [Commented] (SPARK-19559) Fix flaky KafkaSourceSuite.subscribing topic by pattern with topic deletions

2017-02-12 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15862721#comment-15862721 ] Liwei Lin commented on SPARK-19559: --- I think I found the root cause of this; will submi

[jira] [Commented] (SPARK-18736) CreateMap allows non-unique keys

2017-02-02 Thread Shuai Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15851057#comment-15851057 ] Shuai Lin commented on SPARK-18736: --- [~eyalfa] How is it going on? I can work on this o

[jira] (SPARK-19393) Add `approx_percentile` Dataset/DataFrame API

2017-01-30 Thread Liwei Lin (JIRA)
Title: Message Title Liwei Lin resolved as Won't Fix

[jira] (SPARK-19393) Add `approx_percentile` Dataset/DataFrame API

2017-01-28 Thread Liwei Lin (JIRA)
Title: Message Title Liwei Lin updated an issue

[jira] (SPARK-19393) Add `approx_percentile` Dataframe API

2017-01-28 Thread Liwei Lin (JIRA)
Title: Message Title Liwei Lin created an issue

[jira] [Updated] (SPARK-19364) Stream Blocks in Storage Persists Forever when Kinesis Checkpoints are enabled and an exception is thrown

2017-01-28 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-19364: -- Component/s: (was: Spark Core) DStreams > Stream Blocks in Storage Persists Foreve

[jira] [Commented] (SPARK-14098) Generate code that get a float/double value in each column from CachedBatch when DataFrame.cache() is called

2017-01-28 Thread Shuai Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843996#comment-15843996 ] Shuai Lin commented on SPARK-14098: --- [~kiszk] Seems the title/description of this ticke

[jira] [Created] (SPARK-19385) During canonicalization, `NOT(l, r)` should not expect such cases that l.hashcode > r.hashcode

2017-01-27 Thread Liwei Lin (JIRA)
Liwei Lin created SPARK-19385: - Summary: During canonicalization, `NOT(l, r)` should not expect such cases that l.hashcode > r.hashcode Key: SPARK-19385 URL: https://issues.apache.org/jira/browse/SPARK-19385

[jira] [Created] (SPARK-19330) Also show tooltip for successful batches

2017-01-22 Thread Liwei Lin (JIRA)
Liwei Lin created SPARK-19330: - Summary: Also show tooltip for successful batches Key: SPARK-19330 URL: https://issues.apache.org/jira/browse/SPARK-19330 Project: Spark Issue Type: Improvement

[jira] [Comment Edited] (SPARK-15023) Add support for testing against the `ProcessingTime(intervalMS > 0)` trigger and `ManualClock`

2017-01-18 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15829216#comment-15829216 ] Liwei Lin edited comment on SPARK-15023 at 1/19/17 3:01 AM: H

[jira] [Commented] (SPARK-15023) Add support for testing against the `ProcessingTime(intervalMS > 0)` trigger and `ManualClock`

2017-01-18 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15829216#comment-15829216 ] Liwei Lin commented on SPARK-15023: --- Hi [~hyukjin.kwon], yea this was resolved by the P

[jira] [Updated] (SPARK-19168) StateStore should be aborted upon error

2017-01-17 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-19168: -- Description: We should call `StateStore.abort()` when there should be any error before the store is com

[jira] [Updated] (SPARK-19168) StateStore should be aborted upon error

2017-01-17 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-19168: -- Summary: StateStore should be aborted upon error (was: Improvement: filter late data using watermark f

[jira] [Commented] (SPARK-19153) DataFrameWriter.saveAsTable should work with hive format to create partitioned table

2017-01-16 Thread Shuai Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824935#comment-15824935 ] Shuai Lin commented on SPARK-19153: --- Never mind. I can help review it instead. > DataF

[jira] [Commented] (SPARK-19153) DataFrameWriter.saveAsTable should work with hive format to create partitioned table

2017-01-15 Thread Shuai Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15823552#comment-15823552 ] Shuai Lin commented on SPARK-19153: --- [~windpiger] I planned to sent a PR today, just to

[jira] [Commented] (SPARK-19153) DataFrameWriter.saveAsTable should work with hive format to create partitioned table

2017-01-15 Thread Shuai Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15823141#comment-15823141 ] Shuai Lin commented on SPARK-19153: --- bq. To clarify, we want this feature in DataFrameW

[jira] [Commented] (SPARK-19153) DataFrameWriter.saveAsTable should work with hive format to create partitioned table

2017-01-15 Thread Shuai Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15823103#comment-15823103 ] Shuai Lin commented on SPARK-19153: --- I find it's quite straight forward to remove the r

  1   2   3   4   >