[jira] [Created] (SPARK-40093) is kubernetes jar required if not using that executor?

2022-08-15 Thread t oo (Jira)
t oo created SPARK-40093: Summary: is kubernetes jar required if not using that executor? Key: SPARK-40093 URL: https://issues.apache.org/jira/browse/SPARK-40093 Project: Spark Issue Type: Question

[jira] [Updated] (SPARK-40093) is kubernetes jar required if not using that executor?

2022-08-15 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-40093: - Description: my docker file is very big with pyspark can i remove dis files below if i don't use 'kubernetes ex

[jira] [Created] (SPARK-40092) is breeze required if not using ML?

2022-08-15 Thread t oo (Jira)
t oo created SPARK-40092: Summary: is breeze required if not using ML? Key: SPARK-40092 URL: https://issues.apache.org/jira/browse/SPARK-40092 Project: Spark Issue Type: Question Components

[jira] [Updated] (SPARK-40092) is breeze required if not using ML?

2022-08-15 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-40092: - Description: my docker file is very big with pyspark can i remove dis files below if i don't use 'ML'? 14M    

[jira] [Created] (SPARK-40091) is rocksdbjni required if not using streaming?

2022-08-15 Thread t oo (Jira)
t oo created SPARK-40091: Summary: is rocksdbjni required if not using streaming? Key: SPARK-40091 URL: https://issues.apache.org/jira/browse/SPARK-40091 Project: Spark Issue Type: Question

[jira] [Created] (SPARK-38883) smaller pyspark install if not using streaming?

2022-04-12 Thread t oo (Jira)
t oo created SPARK-38883: Summary: smaller pyspark install if not using streaming? Key: SPARK-38883 URL: https://issues.apache.org/jira/browse/SPARK-38883 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-37420) Oracle JDBC - java.lang.ArithmeticException: Decimal precision 49 exceeds max precision 38

2021-11-20 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-37420: - Description: reading oracle jdbc is not working as expected, i thought a simple df show should work.   {code:j

[jira] [Updated] (SPARK-37420) Oracle JDBC - java.lang.ArithmeticException: Decimal precision 49 exceeds max precision 38

2021-11-20 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-37420: - Description: reading oracle jdbc is not working as expected, i thought a simple df show should work.   {code:j

[jira] [Created] (SPARK-37420) Oracle JDBC - java.lang.ArithmeticException: Decimal precision 49 exceeds max precision 38

2021-11-20 Thread t oo (Jira)
t oo created SPARK-37420: Summary: Oracle JDBC - java.lang.ArithmeticException: Decimal precision 49 exceeds max precision 38 Key: SPARK-37420 URL: https://issues.apache.org/jira/browse/SPARK-37420 Project: S

[jira] [Comment Edited] (SPARK-35974) Spark submit REST cluster/standalone mode - launching an s3a jar with STS

2021-07-29 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17390198#comment-17390198 ] t oo edited comment on SPARK-35974 at 7/29/21, 11:38 PM: - same i

[jira] [Updated] (SPARK-35974) Spark submit REST cluster/standalone mode - launching an s3a jar with STS

2021-07-29 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-35974: - Affects Version/s: (was: 2.4.8) 3.1.2 > Spark submit REST cluster/standalone mode - l

[jira] [Commented] (SPARK-35974) Spark submit REST cluster/standalone mode - launching an s3a jar with STS

2021-07-29 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17390198#comment-17390198 ] t oo commented on SPARK-35974: -- same issue on spark 3.1.2:   { "action" : "SubmissionSta

[jira] [Reopened] (SPARK-35974) Spark submit REST cluster/standalone mode - launching an s3a jar with STS

2021-07-29 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo reopened SPARK-35974: -- > Spark submit REST cluster/standalone mode - launching an s3a jar with STS >

[jira] [Created] (SPARK-36147) [SQL] - log level should be warning if files not found in BasicWriteStatsTracker

2021-07-14 Thread t oo (Jira)
t oo created SPARK-36147: Summary: [SQL] - log level should be warning if files not found in BasicWriteStatsTracker Key: SPARK-36147 URL: https://issues.apache.org/jira/browse/SPARK-36147 Project: Spark

[jira] [Updated] (SPARK-35974) Spark submit REST cluster/standalone mode - launching an s3a jar with STS

2021-07-02 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-35974: - Affects Version/s: (was: 2.4.6) 2.4.8 Description: {code:java} /var/lib/spa

[jira] [Reopened] (SPARK-35974) Spark submit REST cluster/standalone mode - launching an s3a jar with STS

2021-07-02 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo reopened SPARK-35974: -- v2.4.8 is less than 2 months old > Spark submit REST cluster/standalone mode - launching an s3a jar with STS > --

[jira] [Created] (SPARK-35974) Spark submit REST cluster/standalone mode - launching an s3a jar with STS

2021-07-01 Thread t oo (Jira)
t oo created SPARK-35974: Summary: Spark submit REST cluster/standalone mode - launching an s3a jar with STS Key: SPARK-35974 URL: https://issues.apache.org/jira/browse/SPARK-35974 Project: Spark Is

[jira] [Created] (SPARK-35249) to_timestamp can't parse 6 digit microsecond SSSSSS

2021-04-27 Thread t oo (Jira)
t oo created SPARK-35249: Summary: to_timestamp can't parse 6 digit microsecond SS Key: SPARK-35249 URL: https://issues.apache.org/jira/browse/SPARK-35249 Project: Spark Issue Type: Wish

[jira] [Commented] (SPARK-32924) Web UI sort on duration is wrong

2020-11-21 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17236762#comment-17236762 ] t oo commented on SPARK-32924: -- [~pralabhkumar] yes > Web UI sort on duration is wrong > -

[jira] [Commented] (SPARK-33085) "Master removed our application" error leads to FAILED driver status instead of KILLED driver status

2020-10-14 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17213815#comment-17213815 ] t oo commented on SPARK-33085: -- # start long running spark job in cluster mode # terminate

[jira] [Commented] (SPARK-27733) Upgrade to Avro 1.10.0

2020-10-11 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17211955#comment-17211955 ] t oo commented on SPARK-27733: -- hive 1 gone I think > Upgrade to Avro 1.10.0 > ---

[jira] [Created] (SPARK-33085) "Master removed our application" error leads to FAILED driver status instead of KILLED driver status

2020-10-07 Thread t oo (Jira)
t oo created SPARK-33085: Summary: "Master removed our application" error leads to FAILED driver status instead of KILLED driver status Key: SPARK-33085 URL: https://issues.apache.org/jira/browse/SPARK-33085

[jira] [Created] (SPARK-33053) Document driver states (difference between FAILED/ERROR/KILLED)

2020-10-02 Thread t oo (Jira)
t oo created SPARK-33053: Summary: Document driver states (difference between FAILED/ERROR/KILLED) Key: SPARK-33053 URL: https://issues.apache.org/jira/browse/SPARK-33053 Project: Spark Issue Type:

[jira] [Created] (SPARK-32924) Web UI sort on duration is wrong

2020-09-17 Thread t oo (Jira)
t oo created SPARK-32924: Summary: Web UI sort on duration is wrong Key: SPARK-32924 URL: https://issues.apache.org/jira/browse/SPARK-32924 Project: Spark Issue Type: Bug Components: Web UI

[jira] [Updated] (SPARK-32924) Web UI sort on duration is wrong

2020-09-17 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-32924: - Attachment: ui_sort.png > Web UI sort on duration is wrong > > >

[jira] [Created] (SPARK-32373) Spark Standalone - RetryingBlockFetcher tries to get block from worker even 10mins after it was de-registered from spark cluster

2020-07-20 Thread t oo (Jira)
t oo created SPARK-32373: Summary: Spark Standalone - RetryingBlockFetcher tries to get block from worker even 10mins after it was de-registered from spark cluster Key: SPARK-32373 URL: https://issues.apache.org/jira/brow

[jira] [Updated] (SPARK-32197) 'Spark driver' stays running even though 'spark application' has FAILED

2020-07-06 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-32197: - Description: App failed in 6 minutes, driver has been stuck for > 8 hours. I would expect driver to fail if app

[jira] [Updated] (SPARK-32197) 'Spark driver' stays running even though 'spark application' has FAILED

2020-07-06 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-32197: - Description: App failed in 6 minutes, driver has been stuck for > 8 hours. I would expect driver to fail if app

[jira] [Updated] (SPARK-32197) 'Spark driver' stays running even though 'spark application' has FAILED

2020-07-06 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-32197: - Attachment: app_executors.png failed_stages.png failed1.png stuck

[jira] [Created] (SPARK-32197) 'Spark driver' stays running even though 'spark application' has FAILED

2020-07-06 Thread t oo (Jira)
t oo created SPARK-32197: Summary: 'Spark driver' stays running even though 'spark application' has FAILED Key: SPARK-32197 URL: https://issues.apache.org/jira/browse/SPARK-32197 Project: Spark Issu

[jira] [Updated] (SPARK-32197) 'Spark driver' stays running even though 'spark application' has FAILED

2020-07-06 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-32197: - Attachment: applog.txt j1.out driverlog.txt > 'Spark driver' stays running even

[jira] [Created] (SPARK-32040) Idle cores not being allocated

2020-06-20 Thread t oo (Jira)
t oo created SPARK-32040: Summary: Idle cores not being allocated Key: SPARK-32040 URL: https://issues.apache.org/jira/browse/SPARK-32040 Project: Spark Issue Type: Bug Components: Schedule

[jira] [Commented] (SPARK-29037) [Core] Spark gives duplicate result when an application was killed and rerun

2020-04-29 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17095434#comment-17095434 ] t oo commented on SPARK-29037: -- with spark 2.3.4 and hadoop 2.8.5: i am facing this doing s

[jira] [Commented] (SPARK-24617) Spark driver not requesting another executor once original executor exits due to 'lost worker'

2020-03-01 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048627#comment-17048627 ] t oo commented on SPARK-24617: -- same problem in spark 2.3.4 > Spark driver not requesting

[jira] [Commented] (SPARK-30561) start spark applications without a 30second startup penalty

2020-02-25 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17045015#comment-17045015 ] t oo commented on SPARK-30561: -- [~rxin] looks like you committed val SLEEP_TIME_SECS = 5 in

[jira] [Commented] (SPARK-30561) start spark applications without a 30second startup penalty

2020-02-25 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17045013#comment-17045013 ] t oo commented on SPARK-30561: -- [~pwendell] looks like you added the sleep(5000), you don't

[jira] [Commented] (SPARK-30610) spark worker graceful shutdown

2020-02-25 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044974#comment-17044974 ] t oo commented on SPARK-30610: -- >From [~holdenkarau]: In its present state it could help, y

[jira] [Created] (SPARK-30610) spark worker graceful shutdown

2020-01-22 Thread t oo (Jira)
t oo created SPARK-30610: Summary: spark worker graceful shutdown Key: SPARK-30610 URL: https://issues.apache.org/jira/browse/SPARK-30610 Project: Spark Issue Type: Improvement Components:

[jira] [Created] (SPARK-30561) start spark applications without a 30second startup penalty

2020-01-18 Thread t oo (Jira)
t oo created SPARK-30561: Summary: start spark applications without a 30second startup penalty Key: SPARK-30561 URL: https://issues.apache.org/jira/browse/SPARK-30561 Project: Spark Issue Type: Impr

[jira] [Updated] (SPARK-30560) allow driver to consume a fractional core

2020-01-18 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-30560: - Description: see https://stackoverflow.com/questions/56781927/apache-spark-standalone-scheduler-why-does-driver

[jira] [Created] (SPARK-30560) allow driver to consume a fractional core

2020-01-18 Thread t oo (Jira)
t oo created SPARK-30560: Summary: allow driver to consume a fractional core Key: SPARK-30560 URL: https://issues.apache.org/jira/browse/SPARK-30560 Project: Spark Issue Type: Improvement C

[jira] [Commented] (SPARK-27750) Standalone scheduler - ability to prioritize applications over drivers, many drivers act like Denial of Service

2020-01-18 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17018544#comment-17018544 ] t oo commented on SPARK-27750: -- did some digging, let me know if I'm on right track. https

[jira] [Commented] (SPARK-27750) Standalone scheduler - ability to prioritize applications over drivers, many drivers act like Denial of Service

2020-01-16 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017448#comment-17017448 ] t oo commented on SPARK-27750: -- yes, I hit this. 'spark standalone' is the RM, it has no po

[jira] [Comment Edited] (SPARK-27750) Standalone scheduler - ability to prioritize applications over drivers, many drivers act like Denial of Service

2020-01-15 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989249#comment-16989249 ] t oo edited comment on SPARK-27750 at 1/16/20 6:33 AM: --- WDYT [~Ngo

[jira] [Comment Edited] (SPARK-27750) Standalone scheduler - ability to prioritize applications over drivers, many drivers act like Denial of Service

2020-01-15 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989249#comment-16989249 ] t oo edited comment on SPARK-27750 at 1/15/20 11:36 PM: WDYT [~N

[jira] [Comment Edited] (SPARK-27750) Standalone scheduler - ability to prioritize applications over drivers, many drivers act like Denial of Service

2020-01-11 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989249#comment-16989249 ] t oo edited comment on SPARK-27750 at 1/11/20 12:59 PM: WDYT [~N

[jira] [Reopened] (SPARK-27429) [SQL] to_timestamp function with additional argument flag that will allow exception if value could not be cast

2019-12-25 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo reopened SPARK-27429: -- presto has both functionality, https://prestosql.io/docs/current/functions/conditional.html by default they fail

[jira] [Commented] (SPARK-27491) SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark 2.3.x

2019-12-25 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17003393#comment-17003393 ] t oo commented on SPARK-27491: -- [~skonto] curl works curl http://spark-master.corp.com:6066

[jira] [Updated] (SPARK-27491) SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark 2.3.x

2019-12-25 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-27491: - Affects Version/s: 2.4.4 > SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty > resp

[jira] [Updated] (SPARK-30251) faster way to read csv.gz?

2019-12-13 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-30251: - Description: some data providers give files in csv.gz (ie 1gb compressed which is 25gb uncompressed; or 5gb com

[jira] [Created] (SPARK-30251) faster way to read csv.gz?

2019-12-13 Thread t oo (Jira)
t oo created SPARK-30251: Summary: faster way to read csv.gz? Key: SPARK-30251 URL: https://issues.apache.org/jira/browse/SPARK-30251 Project: Spark Issue Type: New Feature Components: Spar

[jira] [Commented] (SPARK-26346) Upgrade parquet to 1.11.0

2019-12-08 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991165#comment-16991165 ] t oo commented on SPARK-26346: -- https://www.apache.org/dist/parquet/apache-parquet-1.11.0/ 

[jira] [Commented] (SPARK-30162) Filter is not being pushed down for Parquet files

2019-12-07 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16990725#comment-16990725 ] t oo commented on SPARK-30162: -- did u try on scala sparkshell? > Filter is not being pushe

[jira] [Commented] (SPARK-27623) Provider org.apache.spark.sql.avro.AvroFileFormat could not be instantiated

2019-12-07 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16990723#comment-16990723 ] t oo commented on SPARK-27623: -- bump > Provider org.apache.spark.sql.avro.AvroFileFormat c

[jira] [Commented] (SPARK-26346) Upgrade parquet to 1.11.0

2019-12-07 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16990722#comment-16990722 ] t oo commented on SPARK-26346: -- bump > Upgrade parquet to 1.11.0 > ---

[jira] [Commented] (SPARK-26091) Upgrade to 2.3.4 for Hive Metastore Client 2.3

2019-12-06 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989616#comment-16989616 ] t oo commented on SPARK-26091: -- but i dont use 2.3.3 > Upgrade to 2.3.4 for Hive Metastore

[jira] [Commented] (SPARK-26091) Upgrade to 2.3.4 for Hive Metastore Client 2.3

2019-12-05 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989461#comment-16989461 ] t oo commented on SPARK-26091: -- i just want to access hivemetaxtore 2.3.4 > Upgrade to 2.3

[jira] [Commented] (SPARK-26091) Upgrade to 2.3.4 for Hive Metastore Client 2.3

2019-12-05 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989256#comment-16989256 ] t oo commented on SPARK-26091: -- can this go in spark 2.4.5 ?   > Upgrade to 2.3.4 for Hiv

[jira] [Commented] (SPARK-22860) Spark workers log ssl passwords passed to the executors

2019-12-05 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989255#comment-16989255 ] t oo commented on SPARK-22860: -- [~kabhwan] can this go in 2.4.5? > Spark workers log ssl p

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

2019-12-05 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989254#comment-16989254 ] t oo commented on SPARK-23534: -- close? > Spark run on Hadoop 3.0.0 > -

[jira] [Commented] (SPARK-24590) Make Jenkins tests passed with hadoop 3 profile

2019-12-05 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989253#comment-16989253 ] t oo commented on SPARK-24590: -- close?   > Make Jenkins tests passed with hadoop 3 profil

[jira] [Comment Edited] (SPARK-5159) Thrift server does not respect hive.server2.enable.doAs=true

2019-12-05 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989251#comment-16989251 ] t oo edited comment on SPARK-5159 at 12/5/19 11:31 PM: --- [~yumwang] 

[jira] [Commented] (SPARK-5159) Thrift server does not respect hive.server2.enable.doAs=true

2019-12-05 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989251#comment-16989251 ] t oo commented on SPARK-5159: - [~yumwang] does removal of hive fork soove this one?   > Thr

[jira] [Commented] (SPARK-27750) Standalone scheduler - ability to prioritize applications over drivers, many drivers act like Denial of Service

2019-12-05 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989249#comment-16989249 ] t oo commented on SPARK-27750: -- bump   > Standalone scheduler - ability to prioritize app

[jira] [Commented] (SPARK-27821) Spark WebUI - show numbers of drivers/apps in waiting/submitted/killed/running state

2019-12-05 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989248#comment-16989248 ] t oo commented on SPARK-27821: -- duration of running drivers missing too > Spark WebUI - sh

[jira] [Commented] (SPARK-25088) Rest Server default & doc updates

2019-09-13 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929103#comment-16929103 ] t oo commented on SPARK-25088: -- they are on different ports, now i have to maintain a patch

[jira] [Commented] (SPARK-28376) Support to write sorted parquet files in each row group

2019-07-14 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884883#comment-16884883 ] t oo commented on SPARK-28376: -- [~rdblue] can u add some color here? > Support to write so

[jira] [Created] (SPARK-28376) Write sorted parquet files

2019-07-12 Thread t oo (JIRA)
t oo created SPARK-28376: Summary: Write sorted parquet files Key: SPARK-28376 URL: https://issues.apache.org/jira/browse/SPARK-28376 Project: Spark Issue Type: New Feature Components: Inpu

[jira] [Commented] (SPARK-15420) Repartition and sort before Parquet writes

2019-07-12 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884220#comment-16884220 ] t oo commented on SPARK-15420: -- PR was abandoned :( > Repartition and sort before Parquet

[jira] [Updated] (SPARK-27750) Standalone scheduler - ability to prioritize applications over drivers, many drivers act like Denial of Service

2019-06-12 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-27750: - Description: If I submit 1000 spark submit drivers then they consume all the cores on my cluster (essentially i

[jira] [Issue Comment Deleted] (SPARK-20202) Remove references to org.spark-project.hive

2019-06-04 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-20202: - Comment: was deleted (was: gentle ping) > Remove references to org.spark-project.hive > ---

[jira] [Issue Comment Deleted] (SPARK-20202) Remove references to org.spark-project.hive

2019-06-04 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-20202: - Comment: was deleted (was: bump) > Remove references to org.spark-project.hive > --

[jira] [Updated] (SPARK-27822) Spark WebUi - for running applications have a drivername column

2019-05-24 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-27822: - Attachment: spark-ui-waiting-2.png > Spark WebUi - for running applications have a drivername column > -

[jira] [Commented] (SPARK-27822) Spark WebUi - for running applications have a drivername column

2019-05-24 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16847329#comment-16847329 ] t oo commented on SPARK-27822: -- [~hyukjin.kwon] attached > Spark WebUi - for running appli

[jira] [Created] (SPARK-27822) Spark WebUi - for running applications have a drivername column

2019-05-23 Thread t oo (JIRA)
t oo created SPARK-27822: Summary: Spark WebUi - for running applications have a drivername column Key: SPARK-27822 URL: https://issues.apache.org/jira/browse/SPARK-27822 Project: Spark Issue Type:

[jira] [Updated] (SPARK-27821) Spark WebUI - show numbers of drivers/apps in waiting/submitted/killed/running state

2019-05-23 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-27821: - Attachment: webui.png > Spark WebUI - show numbers of drivers/apps in > waiting/submitted/killed/running state

[jira] [Created] (SPARK-27821) Spark WebUI - show numbers of drivers/apps in waiting/submitted/killed/running state

2019-05-23 Thread t oo (JIRA)
t oo created SPARK-27821: Summary: Spark WebUI - show numbers of drivers/apps in waiting/submitted/killed/running state Key: SPARK-27821 URL: https://issues.apache.org/jira/browse/SPARK-27821 Project: Spark

[jira] [Created] (SPARK-27750) Standalone scheduler - ability to prioritize applications over drivers, many drivers act like Denial of Service

2019-05-16 Thread t oo (JIRA)
t oo created SPARK-27750: Summary: Standalone scheduler - ability to prioritize applications over drivers, many drivers act like Denial of Service Key: SPARK-27750 URL: https://issues.apache.org/jira/browse/SPARK-27750

[jira] [Commented] (SPARK-27491) SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark 2.3.x

2019-04-25 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825804#comment-16825804 ] t oo commented on SPARK-27491: -- my current workaround to get airflow to integrate with spar

[jira] [Commented] (SPARK-27491) SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark 2.3.x

2019-04-17 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820542#comment-16820542 ] t oo commented on SPARK-27491: -- cc: [~skonto] [~mpmolek] [~gschiavon] [~scrapco...@gmail.co

[jira] [Created] (SPARK-27491) SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark

2019-04-17 Thread t oo (JIRA)
t oo created SPARK-27491: Summary: SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark Key: SPARK-27491 URL: https://issues.apache.org/jira/

[jira] [Updated] (SPARK-27491) SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark 2.3.x

2019-04-17 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-27491: - Summary: SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airfl

[jira] [Commented] (SPARK-27491) SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark 2.3.x

2019-04-17 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820211#comment-16820211 ] t oo commented on SPARK-27491: -- cc: [~ash] [~bolke] > SPARK REST API - "org.apache.spark.d

[jira] [Commented] (SPARK-25088) Rest Server default & doc updates

2019-04-16 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819764#comment-16819764 ] t oo commented on SPARK-25088: -- why block rest if auth is on? for example i want to be able

[jira] [Created] (SPARK-27429) [SQL] to_timestamp function with additional argument flag that will allow exception if value could not be cast

2019-04-10 Thread t oo (JIRA)
t oo created SPARK-27429: Summary: [SQL] to_timestamp function with additional argument flag that will allow exception if value could not be cast Key: SPARK-27429 URL: https://issues.apache.org/jira/browse/SPARK-27429

[jira] [Commented] (SPARK-27284) Spark Standalone aggregated logs in 1 file per appid (ala yarn -logs -applicationId)

2019-03-31 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806462#comment-16806462 ] t oo commented on SPARK-27284: -- [~srowen] in yarn mode if you want all logs for a single sp

[jira] [Created] (SPARK-27284) Spark Standalone aggregated logs in 1 file per appid (ala yarn -logs -applicationId)

2019-03-26 Thread t oo (JIRA)
t oo created SPARK-27284: Summary: Spark Standalone aggregated logs in 1 file per appid (ala yarn -logs -applicationId) Key: SPARK-27284 URL: https://issues.apache.org/jira/browse/SPARK-27284 Project: Spark

[jira] [Commented] (SPARK-26998) spark.ssl.keyStorePassword in plaintext on 'ps -ef' output of executor processes in Standalone mode

2019-03-05 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784791#comment-16784791 ] t oo commented on SPARK-26998: -- [~gsomogyi] please take it forward. [~kabhwan] truststore

[jira] [Commented] (SPARK-26998) spark.ssl.keyStorePassword in plaintext on 'ps -ef' output of executor processes in Standalone mode

2019-03-02 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16782330#comment-16782330 ] t oo commented on SPARK-26998: -- [https://github.com/apache/spark/pull/23820] is only about

[jira] [Updated] (SPARK-26998) spark.ssl.keyStorePassword in plaintext on 'ps -ef' output of executor processes in Standalone mode

2019-02-26 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-26998: - Labels: SECURITY Security secur security security-issue (was: ) > spark.ssl.keyStorePassword in plaintext on 'p

[jira] [Created] (SPARK-26998) spark.ssl.keyStorePassword in plaintext on 'ps -ef' output of executor processes in Standalone mode

2019-02-26 Thread t oo (JIRA)
t oo created SPARK-26998: Summary: spark.ssl.keyStorePassword in plaintext on 'ps -ef' output of executor processes in Standalone mode Key: SPARK-26998 URL: https://issues.apache.org/jira/browse/SPARK-26998 P

[jira] [Commented] (SPARK-25757) Upgrade netty-all from 4.1.17.Final to 4.1.30.Final

2019-02-23 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16775873#comment-16775873 ] t oo commented on SPARK-25757: -- [~lipzhu] want to upg netty to 4.1.33? > Upgrade netty-all

[jira] [Commented] (SPARK-16859) History Server storage information is missing

2019-02-17 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16770679#comment-16770679 ] t oo commented on SPARK-16859: -- I know how you feel, I submitted a PR to fix a security fla

[jira] [Commented] (SPARK-16859) History Server storage information is missing

2019-02-17 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16770482#comment-16770482 ] t oo commented on SPARK-16859: -- please don't leave us [~Hauer] > History Server storage in

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2019-02-17 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16770481#comment-16770481 ] t oo commented on SPARK-17556: -- please don't leave us [~scwf] > Executor side broadcast fo

[jira] [Commented] (SPARK-22860) Spark workers log ssl passwords passed to the executors

2019-02-17 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16770373#comment-16770373 ] t oo commented on SPARK-22860: -- Not just in worker log, but in 'ps -ef' process list > Spa

[jira] [Commented] (SPARK-20286) dynamicAllocation.executorIdleTimeout is ignored after unpersist

2019-02-17 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16770344#comment-16770344 ] t oo commented on SPARK-20286: -- gentle ping > dynamicAllocation.executorIdleTimeout is ign

[jira] [Commented] (SPARK-25594) OOM in long running applications even with UI disabled

2019-02-16 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16770230#comment-16770230 ] t oo commented on SPARK-25594: -- please merge > OOM in long running applications even with

[jira] [Commented] (SPARK-15046) When running hive-thriftserver with yarn on a secure cluster the workers fail with java.lang.NumberFormatException

2019-02-15 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16769890#comment-16769890 ] t oo commented on SPARK-15046: -- gentle ping > When running hive-thriftserver with yarn on

[jira] [Commented] (SPARK-22374) STS ran into OOM in a secure cluster

2019-02-15 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16769885#comment-16769885 ] t oo commented on SPARK-22374: -- gentle ping > STS ran into OOM in a secure cluster > -

  1   2   3   >