[jira] [Created] (SPARK-46091) [KUBERNETES] Respect the existing kubernetes container SPARK_LOCAL_DIRS env

2023-11-24 Thread Fei Wang (Jira)
Fei Wang created SPARK-46091: Summary: [KUBERNETES] Respect the existing kubernetes container SPARK_LOCAL_DIRS env Key: SPARK-46091 URL: https://issues.apache.org/jira/browse/SPARK-46091 Project: Spark

[jira] [Updated] (SPARK-43540) Add working directory into classpath on the driver in K8S cluster mode

2023-05-17 Thread Fei Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-43540: - Summary: Add working directory into classpath on the driver in K8S cluster mode (was: Add current

[jira] [Updated] (SPARK-43540) Add current working directory into classpath on the driver in K8S cluster mode

2023-05-17 Thread Fei Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-43540: - Description: In Yarn cluster modes, the passed files/jars are able to be accessed in the classloader.

[jira] [Updated] (SPARK-43540) Add current working directory into classpath on the driver in K8S cluster mode

2023-05-17 Thread Fei Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-43540: - Description: In Yarn cluster modes, the passed files/jars are able to be accessed in the classloader.

[jira] [Updated] (SPARK-43540) Add current working directory into classpath on the driver in K8S cluster mode

2023-05-17 Thread Fei Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-43540: - Description: In Yarn cluster modes, the passed files/jars are able to be accessed in the classloader.

[jira] [Created] (SPARK-43540) Add current working directory into classpath on the driver in K8S cluster mode

2023-05-17 Thread Fei Wang (Jira)
Fei Wang created SPARK-43540: Summary: Add current working directory into classpath on the driver in K8S cluster mode Key: SPARK-43540 URL: https://issues.apache.org/jira/browse/SPARK-43540 Project:

[jira] [Updated] (SPARK-43504) [K8S] Mounts the hadoop config map on the executor pod

2023-05-15 Thread Fei Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-43504: - Summary: [K8S] Mounts the hadoop config map on the executor pod (was: [K8S] Mount hadoop config map in

[jira] [Updated] (SPARK-43504) [K8S] Mounts the hadoop config map on the executor pod

2023-05-15 Thread Fei Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-43504: - Description: Since SPARK-25815 [,|https://github.com/apache/spark/pull/22911,] the hadoop config map

[jira] [Updated] (SPARK-43504) [K8S] Mount hadoop config map in executor side

2023-05-15 Thread Fei Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-43504: - Description: Since SPARK-25815 [,|https://github.com/apache/spark/pull/22911,] the hadoop config map

[jira] [Updated] (SPARK-43504) [K8S] Mount hadoop config map in executor side

2023-05-15 Thread Fei Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-43504: - Description: Since SPARK-25815 [,|https://github.com/apache/spark/pull/22911,] the hadoop config map

[jira] [Commented] (SPARK-43504) [K8S] Mount hadoop config map in executor side

2023-05-15 Thread Fei Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722674#comment-17722674 ] Fei Wang commented on SPARK-43504: -- gentle ping [~vanzin]  [~dongjoon] [~ifilonenko]  > [K8S] Mount

[jira] [Updated] (SPARK-43504) [K8S] Mount hadoop config map in executor side

2023-05-15 Thread Fei Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-43504: - Description: Since SPARK-25815 [,|https://github.com/apache/spark/pull/22911,] the hadoop config map

[jira] [Created] (SPARK-43504) [K8S] Mount hadoop config map in executor side

2023-05-15 Thread Fei Wang (Jira)
Fei Wang created SPARK-43504: Summary: [K8S] Mount hadoop config map in executor side Key: SPARK-43504 URL: https://issues.apache.org/jira/browse/SPARK-43504 Project: Spark Issue Type:

[jira] [Updated] (SPARK-43419) [K8S] Make limit.cores be able to be fallen back to request.cores

2023-05-08 Thread Fei Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-43419: - Description: make limit.cores be able to be fallen back to request.cores now without limit.cores, we

[jira] [Updated] (SPARK-43419) [K8S] Make limit.cores be able to be fallen back to request.cores

2023-05-08 Thread Fei Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-43419: - Description: make limit.cores be able to be fallen back to request.cores now without limit.cores, we

[jira] [Updated] (SPARK-43419) [K8S] Make limit.cores be able to be fallen back to request.cores

2023-05-08 Thread Fei Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-43419: - Description: make limit.cores be able to be fallen back to request.cores now without limit.cores, we

[jira] [Created] (SPARK-43419) [K8S] Make limit.cores be able to be fallen back to request.cores

2023-05-08 Thread Fei Wang (Jira)
Fei Wang created SPARK-43419: Summary: [K8S] Make limit.cores be able to be fallen back to request.cores Key: SPARK-43419 URL: https://issues.apache.org/jira/browse/SPARK-43419 Project: Spark

[jira] [Commented] (SPARK-20184) performance regression for complex/long sql when enable whole stage codegen

2017-04-12 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965735#comment-15965735 ] Fei Wang commented on SPARK-20184: -- Also use the master branch to test my test case: 1. Java version

[jira] [Comment Edited] (SPARK-20184) performance regression for complex/long sql when enable whole stage codegen

2017-04-12 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965597#comment-15965597 ] Fei Wang edited comment on SPARK-20184 at 4/12/17 9:21 AM: --- try this : 1.

[jira] [Comment Edited] (SPARK-20184) performance regression for complex/long sql when enable whole stage codegen

2017-04-12 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965597#comment-15965597 ] Fei Wang edited comment on SPARK-20184 at 4/12/17 9:21 AM: --- try this : 1.

[jira] [Commented] (SPARK-20184) performance regression for complex/long sql when enable whole stage codegen

2017-04-12 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965597#comment-15965597 ] Fei Wang commented on SPARK-20184: -- try this : 1. create table [code] val df = (1 to 50).map(x =>

[jira] [Commented] (SPARK-20184) performance regression for complex/long sql when enable whole stage codegen

2017-04-11 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965308#comment-15965308 ] Fei Wang commented on SPARK-20184: -- Tested with a smaller table 100,000 rows. Codegen on: 2.6s Codegen

[jira] [Updated] (SPARK-20184) performance regression for complex/long sql when enable whole stage codegen

2017-04-04 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-20184: - Summary: performance regression for complex/long sql when enable whole stage codegen (was: performance

[jira] [Updated] (SPARK-20184) performance regression for complex/long sql when enable codegen

2017-04-04 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-20184: - Description: The performance of following SQL get much worse in spark 2.x in contrast with codegen

[jira] [Updated] (SPARK-20184) performance regression for complex/long sql when enable codegen

2017-04-04 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-20184: - Description: The performance of following SQL get much worse in spark 2.x in contrast with codegen

[jira] [Updated] (SPARK-20184) performance regression for complex/long sql when enable codegen

2017-04-01 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-20184: - Description: The performance of following SQL get much worse in spark 2.x in contrast with codegen

[jira] [Updated] (SPARK-20184) performance regression for complex/long sql when enable codegen

2017-04-01 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-20184: - Description: The performance of following SQL get much worse in spark 2.x in contrast with codegen

[jira] [Updated] (SPARK-20184) performance regression for complex/long sql when enable codegen

2017-04-01 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-20184: - Description: The performance of following SQL get much worse in spark 2.x in contrast with codegen

[jira] [Updated] (SPARK-20184) performance regression for complex/long sql when enable codegen

2017-04-01 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-20184: - Description: The performance of following SQL get much worse in spark 2.x in contrast with codegen

[jira] [Commented] (SPARK-20184) performance regression for complex/long sql when enable codegen

2017-04-01 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952123#comment-15952123 ] Fei Wang commented on SPARK-20184: -- [~r...@databricks.com] [~davies] May be we need split the huge

[jira] [Updated] (SPARK-20184) performance regression for complex/long sql when enable codegen

2017-04-01 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-20184: - Description: Execute following sql with spark 2.x when codegen enabled, the performance is much worse

[jira] [Updated] (SPARK-20184) performance regression for complex/long sql when enable codegen

2017-04-01 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-20184: - Description: Execute flowing sql with spark 2.x when codegen enabled, the performance is much worse

[jira] [Updated] (SPARK-20184) performance regression for complex/long sql when enable codegen

2017-04-01 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-20184: - Description: Execute flowing sql with spark 2.x when codegen enabled, the performance is much worse

[jira] [Updated] (SPARK-20184) performance regression for complex/long sql when enable codegen

2017-04-01 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-20184: - Summary: performance regression for complex/long sql when enable codegen (was: performance regression

[jira] [Updated] (SPARK-20184) performance regression for complex sql when enable codegen

2017-04-01 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-20184: - Description: Execute flowing sql with spark 2.x when codegen enabled, the performance is much worse

[jira] [Updated] (SPARK-20184) performance regression for complex sql when enable codegen

2017-04-01 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-20184: - Description: Execute flowing sql with spark 2.x when codegen enabled, the performance is much worse

[jira] [Updated] (SPARK-20184) performance regression for complex sql when enable codegen

2017-04-01 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-20184: - Description: Execute flowing sql with spark 2.x when codegen enabled, the performance is much worse

[jira] [Created] (SPARK-20184) performance regression for complex sql when enable codegen

2017-04-01 Thread Fei Wang (JIRA)
Fei Wang created SPARK-20184: Summary: performance regression for complex sql when enable codegen Key: SPARK-20184 URL: https://issues.apache.org/jira/browse/SPARK-20184 Project: Spark Issue

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-28 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-17556: - Attachment: (was: executor broadcast.pdf) > Executor side broadcast for broadcast joins >

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-28 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-17556: - Attachment: executor broadcast.pdf > Executor side broadcast for broadcast joins >

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-26 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-17556: - Attachment: executor broadcast.pdf update design doc > Executor side broadcast for broadcast joins >

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-26 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-17556: - Attachment: (was: executor broadcast.pdf) > Executor side broadcast for broadcast joins >

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516772#comment-15516772 ] Fei Wang commented on SPARK-17556: -- [~viirya] in this case how about notify driver to re-persist the

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516732#comment-15516732 ] Fei Wang commented on SPARK-17556: -- That's a good point! In your solution, the broadcast rdd must

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516639#comment-15516639 ] Fei Wang edited comment on SPARK-17556 at 9/23/16 2:50 PM: --- Yes, the main

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516639#comment-15516639 ] Fei Wang commented on SPARK-17556: -- Yes, the main different is is does not introduce overhead to driver,

[jira] [Issue Comment Deleted] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-17556: - Comment: was deleted (was: Not correct, I just collect the broadcast ref to the driver but not the

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516545#comment-15516545 ] Fei Wang commented on SPARK-17556: -- Not correct, I just collect the broadcast ref to the driver but not

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516546#comment-15516546 ] Fei Wang commented on SPARK-17556: -- Not correct, I just collect the broadcast ref to the driver but not

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516386#comment-15516386 ] Fei Wang edited comment on SPARK-17556 at 9/23/16 1:15 PM: --- [~rxin] attached a

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516386#comment-15516386 ] Fei Wang edited comment on SPARK-17556 at 9/23/16 1:05 PM: --- [~rxin] attached a

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516386#comment-15516386 ] Fei Wang commented on SPARK-17556: -- [~rxin] attached a design doc for the executor based broadcast. Will

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-17556: - Attachment: executor broadcast.pdf > Executor side broadcast for broadcast joins >

[jira] [Created] (SPARK-17644) The failed stage never resubmitted due to abort stage in another thread

2016-09-23 Thread Fei Wang (JIRA)
Fei Wang created SPARK-17644: Summary: The failed stage never resubmitted due to abort stage in another thread Key: SPARK-17644 URL: https://issues.apache.org/jira/browse/SPARK-17644 Project: Spark

[jira] [Updated] (SPARK-12742) org.apache.spark.sql.hive.LogicalPlanToSQLSuite failure due to

2016-01-10 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-12742: - Summary: org.apache.spark.sql.hive.LogicalPlanToSQLSuite failure due to (was:

[jira] [Updated] (SPARK-12742) org.apache.spark.sql.hive.LogicalPlanToSQLSuite failure due to Table already exists

2016-01-10 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-12742: - Due Date: 11/Jan/16 Component/s: SQL > org.apache.spark.sql.hive.LogicalPlanToSQLSuite failure

[jira] [Updated] (SPARK-12742) org.apache.spark.sql.hive.LogicalPlanToSQLSuite failure due to Table already exists

2016-01-10 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-12742: - Summary: org.apache.spark.sql.hive.LogicalPlanToSQLSuite failure due to Table already exists (was:

[jira] [Created] (SPARK-12742) org.apache.spark.sql.hive.LogicalPlanToSQLSuite failuer

2016-01-10 Thread Fei Wang (JIRA)
Fei Wang created SPARK-12742: Summary: org.apache.spark.sql.hive.LogicalPlanToSQLSuite failuer Key: SPARK-12742 URL: https://issues.apache.org/jira/browse/SPARK-12742 Project: Spark Issue Type:

[jira] [Created] (SPARK-12222) deserialize RoaringBitmap using Kryo serializer throw Buffer underflow exception

2015-12-08 Thread Fei Wang (JIRA)
Fei Wang created SPARK-1: Summary: deserialize RoaringBitmap using Kryo serializer throw Buffer underflow exception Key: SPARK-1 URL: https://issues.apache.org/jira/browse/SPARK-1 Project:

[jira] [Commented] (SPARK-4131) Support "Writing data into the filesystem from queries"

2015-09-08 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735995#comment-14735995 ] Fei Wang commented on SPARK-4131: - can you try the example of my test suite to see if it works?

[jira] [Commented] (SPARK-10030) Managed memory leak detected when cache table

2015-08-17 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699075#comment-14699075 ] Fei Wang commented on SPARK-10030: -- [~hyukjin.kwon] this is defenitly a spark sql bug,

[jira] [Commented] (SPARK-10030) Managed memory leak detected when cache table

2015-08-16 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698974#comment-14698974 ] Fei Wang commented on SPARK-10030: -- [~liancheng] can you have a look on this issue?

[jira] [Commented] (SPARK-9725) spark sql query string field return empty/garbled string

2015-08-14 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698124#comment-14698124 ] Fei Wang commented on SPARK-9725: - good job :) spark sql query string field return

[jira] [Commented] (SPARK-9725) spark sql query string field return empty/garbled string

2015-08-13 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14696323#comment-14696323 ] Fei Wang commented on SPARK-9725: - in some cases(the long table name) of ours it also show

[jira] [Commented] (SPARK-9725) spark sql query string field return empty/garbled string

2015-08-13 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14696324#comment-14696324 ] Fei Wang commented on SPARK-9725: - we test on suse and redhat, both has this issue. Now we

[jira] [Commented] (SPARK-9725) spark sql query string field return empty/garbled string

2015-08-13 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14696449#comment-14696449 ] Fei Wang commented on SPARK-9725: - [~davies] please see the comment of peizhongshuai

[jira] [Commented] (SPARK-8890) Reduce memory consumption for dynamic partition insert

2015-08-09 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679447#comment-14679447 ] Fei Wang commented on SPARK-8890: - should this be part of tungsten? Reduce memory

[jira] [Commented] (SPARK-9725) spark sql query string field return empty/garbled string

2015-08-08 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662847#comment-14662847 ] Fei Wang commented on SPARK-9725: - And i not set SPARK_PREPEND_CLASSES spark sql query

[jira] [Commented] (SPARK-9725) spark sql query string field return empty/garbled string

2015-08-07 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661383#comment-14661383 ] Fei Wang commented on SPARK-9725: - how much memory you set to executor? spark sql query

[jira] [Comment Edited] (SPARK-9725) spark sql query string field return empty/garbled string

2015-08-07 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661389#comment-14661389 ] Fei Wang edited comment on SPARK-9725 at 8/7/15 6:21 AM: - i try

[jira] [Comment Edited] (SPARK-9725) spark sql query string field return empty/garbled string

2015-08-07 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661389#comment-14661389 ] Fei Wang edited comment on SPARK-9725 at 8/7/15 6:21 AM: - i try

[jira] [Commented] (SPARK-9725) spark sql query string field return empty/garbled string

2015-08-07 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661389#comment-14661389 ] Fei Wang commented on SPARK-9725: - i try the master again and reproduce this issue:

[jira] [Created] (SPARK-9725) spark sql query string field return empty/garbled string

2015-08-06 Thread Fei Wang (JIRA)
Fei Wang created SPARK-9725: --- Summary: spark sql query string field return empty/garbled string Key: SPARK-9725 URL: https://issues.apache.org/jira/browse/SPARK-9725 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-8968) dynamic partitioning in spark sql performance issue due to the high GC overhead

2015-07-10 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-8968: Summary: dynamic partitioning in spark sql performance issue due to the high GC overhead (was: shuffled by

[jira] [Commented] (SPARK-8968) dynamic partitioning in spark sql performance issue due to the high GC overhead

2015-07-10 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621935#comment-14621935 ] Fei Wang commented on SPARK-8968: - changed, how about this? dynamic partitioning in

[jira] [Created] (SPARK-8968) shuffled by the partition clomns when dynamic partitioning to optimize the memory overhead

2015-07-09 Thread Fei Wang (JIRA)
Fei Wang created SPARK-8968: --- Summary: shuffled by the partition clomns when dynamic partitioning to optimize the memory overhead Key: SPARK-8968 URL: https://issues.apache.org/jira/browse/SPARK-8968

[jira] [Commented] (SPARK-7173) Support YARN node label expressions for the application master

2015-06-07 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576547#comment-14576547 ] Fei Wang commented on SPARK-7173: - is this resolved by spark-6470? Support YARN node

[jira] [Closed] (SPARK-7866) print the format string in dataframe explain

2015-05-26 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang closed SPARK-7866. --- Resolution: Won't Fix Wrong print in intelli idea, this is really ok, not a problem. print the format

[jira] [Created] (SPARK-7866) print the format string in dataframe explain

2015-05-26 Thread Fei Wang (JIRA)
Fei Wang created SPARK-7866: --- Summary: print the format string in dataframe explain Key: SPARK-7866 URL: https://issues.apache.org/jira/browse/SPARK-7866 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-7866) print the format string in dataframe explain

2015-05-26 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14559048#comment-14559048 ] Fei Wang commented on SPARK-7866: - Get it, thanks Owen. print the format string in

[jira] [Resolved] (SPARK-3323) yarn website's Tracking UI links to the Standby RM

2015-05-19 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang resolved SPARK-3323. - Resolution: Fixed yarn website's Tracking UI links to the Standby RM

[jira] [Closed] (SPARK-6997) Convert StringType in LocalTableScan

2015-05-19 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang closed SPARK-6997. --- Resolution: Won't Fix Convert StringType in LocalTableScan -

[jira] [Commented] (SPARK-6997) Convert StringType in LocalTableScan

2015-05-19 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549924#comment-14549924 ] Fei Wang commented on SPARK-6997: - this is not nessarry since we have convert the data

[jira] [Closed] (SPARK-7289) Combine Limit and Sort to avoid total ordering

2015-05-19 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang closed SPARK-7289. --- Resolution: Won't Fix use usually not write sql like this Combine Limit and Sort to avoid total ordering

[jira] [Closed] (SPARK-5240) Adding `createDataSourceTable` interface to Catalog

2015-05-19 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang closed SPARK-5240. --- Resolution: Won't Fix Adding `createDataSourceTable` interface to Catalog

[jira] [Created] (SPARK-7656) use CatalystConf in FunctionRegistry

2015-05-14 Thread Fei Wang (JIRA)
Fei Wang created SPARK-7656: --- Summary: use CatalystConf in FunctionRegistry Key: SPARK-7656 URL: https://issues.apache.org/jira/browse/SPARK-7656 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-7631) treenode argString should not print children

2015-05-14 Thread Fei Wang (JIRA)
Fei Wang created SPARK-7631: --- Summary: treenode argString should not print children Key: SPARK-7631 URL: https://issues.apache.org/jira/browse/SPARK-7631 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-7659) Sort by attributes that are not present in the SELECT clause when there is windowfunction analysis error

2015-05-14 Thread Fei Wang (JIRA)
Fei Wang created SPARK-7659: --- Summary: Sort by attributes that are not present in the SELECT clause when there is windowfunction analysis error Key: SPARK-7659 URL: https://issues.apache.org/jira/browse/SPARK-7659

[jira] [Commented] (SPARK-6929) Alias for more complex expression causes attribute not been able to resolve

2015-05-09 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14536627#comment-14536627 ] Fei Wang commented on SPARK-6929: - SPARK SQL use c_0 as inner alias name, i think you can

[jira] [Created] (SPARK-7303) push down project if possible when the child is sort

2015-05-01 Thread Fei Wang (JIRA)
Fei Wang created SPARK-7303: --- Summary: push down project if possible when the child is sort Key: SPARK-7303 URL: https://issues.apache.org/jira/browse/SPARK-7303 Project: Spark Issue Type:

[jira] [Updated] (SPARK-7289) Combine Limit and Sort to avoid total ordering

2015-05-01 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-7289: Description: Optimize following sql select key from (select * from testData order by key) t limit 5 from

[jira] [Created] (SPARK-7289) push down sort when it's child is Limit

2015-04-30 Thread Fei Wang (JIRA)
Fei Wang created SPARK-7289: --- Summary: push down sort when it's child is Limit Key: SPARK-7289 URL: https://issues.apache.org/jira/browse/SPARK-7289 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-7232) Add a Substitution batch for spark sql analyzer

2015-04-29 Thread Fei Wang (JIRA)
Fei Wang created SPARK-7232: --- Summary: Add a Substitution batch for spark sql analyzer Key: SPARK-7232 URL: https://issues.apache.org/jira/browse/SPARK-7232 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-7163) minor refactory for HiveQl

2015-04-27 Thread Fei Wang (JIRA)
Fei Wang created SPARK-7163: --- Summary: minor refactory for HiveQl Key: SPARK-7163 URL: https://issues.apache.org/jira/browse/SPARK-7163 Project: Spark Issue Type: Improvement Components:

[jira] [Created] (SPARK-7123) support table.star in sqlcontext

2015-04-24 Thread Fei Wang (JIRA)
Fei Wang created SPARK-7123: --- Summary: support table.star in sqlcontext Key: SPARK-7123 URL: https://issues.apache.org/jira/browse/SPARK-7123 Project: Spark Issue Type: Bug Components:

[jira] [Created] (SPARK-7093) Using newPredicate in NestedLoopJoin to enable code generation

2015-04-23 Thread Fei Wang (JIRA)
Fei Wang created SPARK-7093: --- Summary: Using newPredicate in NestedLoopJoin to enable code generation Key: SPARK-7093 URL: https://issues.apache.org/jira/browse/SPARK-7093 Project: Spark Issue

[jira] [Created] (SPARK-7109) Push down left side filter for left semi join

2015-04-23 Thread Fei Wang (JIRA)
Fei Wang created SPARK-7109: --- Summary: Push down left side filter for left semi join Key: SPARK-7109 URL: https://issues.apache.org/jira/browse/SPARK-7109 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-5659) Flaky test: o.a.s.streaming.ReceiverSuite.block

2015-04-22 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508302#comment-14508302 ] Fei Wang commented on SPARK-5659: - my locally test with dev/run-tests also go into this

[jira] [Comment Edited] (SPARK-5659) Flaky test: o.a.s.streaming.ReceiverSuite.block

2015-04-22 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508302#comment-14508302 ] Fei Wang edited comment on SPARK-5659 at 4/23/15 2:02 AM: -- my

[jira] [Comment Edited] (SPARK-5659) Flaky test: o.a.s.streaming.ReceiverSuite.block

2015-04-22 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508302#comment-14508302 ] Fei Wang edited comment on SPARK-5659 at 4/23/15 2:03 AM: -- my

  1   2   >