[jira] [Commented] (SPARK-25355) Support --proxy-user for Spark on K8s

2022-05-11 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534889#comment-17534889 ] Gabor Somogyi commented on SPARK-25355: --- I've had a deeper look and HADOOP_TOKEN_FILE_LOCATION is

[jira] [Commented] (SPARK-25355) Support --proxy-user for Spark on K8s

2022-05-05 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17532191#comment-17532191 ] Gabor Somogyi commented on SPARK-25355: --- [~unamesk15] I think the only option for now is to obtain

[jira] [Commented] (SPARK-25355) Support --proxy-user for Spark on K8s

2022-05-05 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17532135#comment-17532135 ] Gabor Somogyi commented on SPARK-25355: --- After some playground work, code digging and your

[jira] [Updated] (SPARK-25355) Support --proxy-user for Spark on K8s

2022-05-05 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-25355: -- Attachment: screenshot-1.png > Support --proxy-user for Spark on K8s >

[jira] [Commented] (SPARK-25355) Support --proxy-user for Spark on K8s

2022-05-04 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531777#comment-17531777 ] Gabor Somogyi commented on SPARK-25355: --- Hmmm, seems like I've already debugged such thing and

[jira] [Commented] (SPARK-25355) Support --proxy-user for Spark on K8s

2022-05-04 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531743#comment-17531743 ] Gabor Somogyi commented on SPARK-25355: --- Now I see where the driver blows up and why no Spark logs

[jira] [Commented] (SPARK-25355) Support --proxy-user for Spark on K8s

2022-05-04 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531646#comment-17531646 ] Gabor Somogyi commented on SPARK-25355: --- > (IllegalArgumentException: Empty cookie header string)

[jira] [Comment Edited] (SPARK-25355) Support --proxy-user for Spark on K8s

2022-05-04 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531523#comment-17531523 ] Gabor Somogyi edited comment on SPARK-25355 at 5/4/22 8:00 AM: --- After the

[jira] [Commented] (SPARK-25355) Support --proxy-user for Spark on K8s

2022-05-04 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531523#comment-17531523 ] Gabor Somogyi commented on SPARK-25355: --- After the attached logs now I see more.

[jira] [Commented] (SPARK-39033) Support --proxy-user for Spark on K8s not working

2022-05-03 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531430#comment-17531430 ] Gabor Somogyi commented on SPARK-39033: --- Simply the logs are trash just like I've mentioned this

[jira] [Commented] (SPARK-25355) Support --proxy-user for Spark on K8s

2022-05-02 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530812#comment-17530812 ] Gabor Somogyi commented on SPARK-25355: --- > Thanks for looking further. Your assumption that 3

[jira] [Commented] (SPARK-25355) Support --proxy-user for Spark on K8s

2022-05-02 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530692#comment-17530692 ] Gabor Somogyi commented on SPARK-25355: --- I've had a further look and found the following:

[jira] [Comment Edited] (SPARK-25355) Support --proxy-user for Spark on K8s

2022-05-02 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530678#comment-17530678 ] Gabor Somogyi edited comment on SPARK-25355 at 5/2/22 11:32 AM: Guys,

[jira] [Commented] (SPARK-25355) Support --proxy-user for Spark on K8s

2022-05-02 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530678#comment-17530678 ] Gabor Somogyi commented on SPARK-25355: --- Guys, when I take a look at the logs and hear what you

[jira] [Commented] (SPARK-25355) Support --proxy-user for Spark on K8s

2022-05-02 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530650#comment-17530650 ] Gabor Somogyi commented on SPARK-25355: --- If the issue still persists then please provide the

[jira] [Commented] (SPARK-25355) Support --proxy-user for Spark on K8s

2022-05-02 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530649#comment-17530649 ] Gabor Somogyi commented on SPARK-25355: --- Then there are 2 issues on your side guys. In the

[jira] [Comment Edited] (SPARK-25355) Support --proxy-user for Spark on K8s

2022-04-30 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530370#comment-17530370 ] Gabor Somogyi edited comment on SPARK-25355 at 4/30/22 9:50 AM: Please

[jira] [Commented] (SPARK-25355) Support --proxy-user for Spark on K8s

2022-04-30 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530370#comment-17530370 ] Gabor Somogyi commented on SPARK-25355: --- Please make sure that namenode host:port is available

[jira] [Commented] (SPARK-25355) Support --proxy-user for Spark on K8s

2022-04-30 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530369#comment-17530369 ] Gabor Somogyi commented on SPARK-25355: --- I've answered this question in the community but I would

[jira] [Commented] (SPARK-28173) Add Kafka delegation token proxy user support

2022-03-07 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17502162#comment-17502162 ] Gabor Somogyi commented on SPARK-28173: --- Looks like there is a willingness to merge the Kafka

[jira] [Commented] (SPARK-37391) SIGNIFICANT bottleneck introduced by fix for SPARK-32001

2021-11-23 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447829#comment-17447829 ] Gabor Somogyi commented on SPARK-37391: --- [~hyukjin.kwon] thanks for pinging me. I've added my

[jira] [Commented] (SPARK-36765) Spark Support for MS Sql JDBC connector with Kerberos/Keytab

2021-09-17 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416519#comment-17416519 ] Gabor Somogyi commented on SPARK-36765: --- It was long time ago when I've done that and AFAIR it

[jira] [Commented] (SPARK-31460) spark-sql-kafka source in spark 2.4.4 causes reading stream failure frequently

2021-08-16 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17399594#comment-17399594 ] Gabor Somogyi commented on SPARK-31460: --- When such thing happens then Kafka connector is super

[jira] [Created] (SPARK-35993) Flaky test: org.apache.spark.sql.execution.streaming.state.RocksDBSuite.ensure that concurrent update and cleanup consistent versions

2021-07-02 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-35993: - Summary: Flaky test: org.apache.spark.sql.execution.streaming.state.RocksDBSuite.ensure that concurrent update and cleanup consistent versions Key: SPARK-35993 URL:

[jira] [Commented] (SPARK-33223) Expose state information on SS UI

2021-05-04 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17338901#comment-17338901 ] Gabor Somogyi commented on SPARK-33223: --- [~smilegator] sure, filed SPARK-35311 and preparing a PR.

[jira] [Created] (SPARK-35311) Add exposed SS UI state information metrics to the documentation

2021-05-04 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-35311: - Summary: Add exposed SS UI state information metrics to the documentation Key: SPARK-35311 URL: https://issues.apache.org/jira/browse/SPARK-35311 Project: Spark

[jira] [Closed] (SPARK-34383) Optimize WAL commit phase on SS

2021-03-22 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed SPARK-34383. - > Optimize WAL commit phase on SS > --- > > Key:

[jira] [Resolved] (SPARK-34383) Optimize WAL commit phase on SS

2021-03-22 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved SPARK-34383. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31495

[jira] [Assigned] (SPARK-34383) Optimize WAL commit phase on SS

2021-03-22 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi reassigned SPARK-34383: - Assignee: Jungtaek Lim > Optimize WAL commit phase on SS >

[jira] [Created] (SPARK-34580) Provide the relationship between batch ID and SQL executions (and/or Jobs) in SS UI page

2021-03-01 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-34580: - Summary: Provide the relationship between batch ID and SQL executions (and/or Jobs) in SS UI page Key: SPARK-34580 URL: https://issues.apache.org/jira/browse/SPARK-34580

[jira] [Commented] (SPARK-34580) Provide the relationship between batch ID and SQL executions (and/or Jobs) in SS UI page

2021-03-01 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17292787#comment-17292787 ] Gabor Somogyi commented on SPARK-34580: --- cc [~kabhwan] [~hyukjin.kwon] [~zsxwing] [~viirya] It

[jira] [Updated] (SPARK-34497) JDBC connection provider is not removing kerberos credentials from JVM security context

2021-02-24 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-34497: -- Description: Some of the built-in JDBC connection providers are changing the JVM security

[jira] [Commented] (SPARK-34497) JDBC connection provider is not removing kerberos credentials from JVM security context

2021-02-24 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289815#comment-17289815 ] Gabor Somogyi commented on SPARK-34497: --- Thanks for the suggestion, filled. > JDBC connection

[jira] [Commented] (SPARK-34497) JDBC connection provider is not removing kerberos credentials from JVM security context

2021-02-22 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288387#comment-17288387 ] Gabor Somogyi commented on SPARK-34497: --- Working on this. > JDBC connection provider is not

[jira] [Created] (SPARK-34497) JDBC connection provider is not removing kerberos credentials from JVM security context

2021-02-22 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-34497: - Summary: JDBC connection provider is not removing kerberos credentials from JVM security context Key: SPARK-34497 URL: https://issues.apache.org/jira/browse/SPARK-34497

[jira] [Closed] (SPARK-12312) Support JDBC Kerberos w/ keytab

2021-02-09 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed SPARK-12312. - > Support JDBC Kerberos w/ keytab > --- > > Key:

[jira] [Resolved] (SPARK-12312) Support JDBC Kerberos w/ keytab

2021-02-09 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved SPARK-12312. --- Fix Version/s: 3.1.0 Resolution: Fixed > Support JDBC Kerberos w/ keytab >

[jira] [Closed] (SPARK-31857) Support Azure SQLDB Kerberos login in JDBC connector

2021-02-09 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed SPARK-31857. - > Support Azure SQLDB Kerberos login in JDBC connector >

[jira] [Resolved] (SPARK-31884) Support MongoDB Kerberos login in JDBC connector

2021-02-09 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved SPARK-31884. --- Resolution: Won't Do > Support MongoDB Kerberos login in JDBC connector >

[jira] [Closed] (SPARK-31884) Support MongoDB Kerberos login in JDBC connector

2021-02-09 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed SPARK-31884. - > Support MongoDB Kerberos login in JDBC connector >

[jira] [Commented] (SPARK-31884) Support MongoDB Kerberos login in JDBC connector

2021-02-09 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282280#comment-17282280 ] Gabor Somogyi commented on SPARK-31884: --- Since the API is implemented I think it's now possible to

[jira] [Resolved] (SPARK-31857) Support Azure SQLDB Kerberos login in JDBC connector

2021-02-09 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved SPARK-31857. --- Resolution: Won't Do > Support Azure SQLDB Kerberos login in JDBC connector >

[jira] [Closed] (SPARK-31815) Support Hive Kerberos login in JDBC connector

2021-02-09 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed SPARK-31815. - > Support Hive Kerberos login in JDBC connector > - > >

[jira] [Commented] (SPARK-31857) Support Azure SQLDB Kerberos login in JDBC connector

2021-02-09 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282279#comment-17282279 ] Gabor Somogyi commented on SPARK-31857: --- Since the API is implemented I think it's now possible to

[jira] [Resolved] (SPARK-31815) Support Hive Kerberos login in JDBC connector

2021-02-09 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved SPARK-31815. --- Resolution: Won't Do > Support Hive Kerberos login in JDBC connector >

[jira] [Commented] (SPARK-31815) Support Hive Kerberos login in JDBC connector

2021-02-09 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282277#comment-17282277 ] Gabor Somogyi commented on SPARK-31815: --- Since the API is implemented I think it's now possible to

[jira] [Commented] (SPARK-34198) Add RocksDB StateStore as external module

2021-01-25 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271308#comment-17271308 ] Gabor Somogyi commented on SPARK-34198: --- +1 on this, I've started to review it back in the days

[jira] [Created] (SPARK-34090) HadoopDelegationTokenManager.isServiceEnabled used in KafkaTokenUtil.needTokenUpdate needs to be cached because it slows down Kafka stream processing in case of delegati

2021-01-12 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-34090: - Summary: HadoopDelegationTokenManager.isServiceEnabled used in KafkaTokenUtil.needTokenUpdate needs to be cached because it slows down Kafka stream processing in case of delegation token Key:

[jira] [Created] (SPARK-34032) Add Kafka delegation token truststore and keystore type confiuration

2021-01-06 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-34032: - Summary: Add Kafka delegation token truststore and keystore type confiuration Key: SPARK-34032 URL: https://issues.apache.org/jira/browse/SPARK-34032 Project:

[jira] [Commented] (SPARK-33635) Performance regression in Kafka read

2020-12-19 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17252153#comment-17252153 ] Gabor Somogyi commented on SPARK-33635: --- {quote}Remember, based on all my testing, and raw kafka

[jira] [Commented] (SPARK-33635) Performance regression in Kafka read

2020-12-13 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248550#comment-17248550 ] Gabor Somogyi commented on SPARK-33635: --- {quote}The collect in this test case is only 13 items of

[jira] [Commented] (SPARK-33635) Performance regression in Kafka read

2020-12-13 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248549#comment-17248549 ] Gabor Somogyi commented on SPARK-33635: --- Mixed up with DStreams, in Strutured Streaming and SQL

[jira] [Updated] (SPARK-28367) Kafka connector infinite wait because metadata never updated

2020-12-11 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-28367: -- Fix Version/s: 3.1.0 > Kafka connector infinite wait because metadata never updated >

[jira] [Closed] (SPARK-28367) Kafka connector infinite wait because metadata never updated

2020-12-11 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed SPARK-28367. - > Kafka connector infinite wait because metadata never updated >

[jira] [Resolved] (SPARK-28367) Kafka connector infinite wait because metadata never updated

2020-12-11 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved SPARK-28367. --- Resolution: Fixed > Kafka connector infinite wait because metadata never updated >

[jira] [Commented] (SPARK-28367) Kafka connector infinite wait because metadata never updated

2020-12-11 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247813#comment-17247813 ] Gabor Somogyi commented on SPARK-28367: --- The issue solved in subtasks so closing this. > Kafka

[jira] [Comment Edited] (SPARK-33635) Performance regression in Kafka read

2020-12-09 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246643#comment-17246643 ] Gabor Somogyi edited comment on SPARK-33635 at 12/9/20, 4:39 PM: -

[jira] [Commented] (SPARK-33635) Performance regression in Kafka read

2020-12-09 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246643#comment-17246643 ] Gabor Somogyi commented on SPARK-33635: --- {quote}I no longer believe this is a true regression in

[jira] [Commented] (SPARK-33635) Performance regression in Kafka read

2020-12-09 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246640#comment-17246640 ] Gabor Somogyi commented on SPARK-33635: --- I've changed to SQL because you're not executing a

[jira] [Updated] (SPARK-33635) Performance regression in Kafka read

2020-12-09 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-33635: -- Component/s: (was: Structured Streaming) SQL > Performance regression in

[jira] [Commented] (SPARK-33635) Performance regression in Kafka read

2020-12-09 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246633#comment-17246633 ] Gabor Somogyi commented on SPARK-33635: --- Since you're measuring speed I've ported the Kafka source

[jira] [Commented] (SPARK-33635) Performance regression in Kafka read

2020-12-09 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246631#comment-17246631 ] Gabor Somogyi commented on SPARK-33635: --- BTW, I'm sure you know but using collect gathers all the

[jira] [Commented] (SPARK-33635) Performance regression in Kafka read

2020-12-09 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246625#comment-17246625 ] Gabor Somogyi commented on SPARK-33635: --- [~david.wyles] try to turn off Kafka consumer caching.

[jira] [Updated] (SPARK-32910) Remove UninterruptibleThread usage from KafkaOffsetReader

2020-12-02 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-32910: -- Affects Version/s: (was: 3.2.0) 3.1.0 > Remove

[jira] [Updated] (SPARK-33633) Expose fine grained state information on SS UI

2020-12-02 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-33633: -- Affects Version/s: (was: 3.2.0) 3.1.0 > Expose fine grained state

[jira] [Updated] (SPARK-33633) Expose fine grained state information on SS UI

2020-12-02 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-33633: -- Description: SPARK-33223 provides aggregated information but in order to find out the

[jira] [Created] (SPARK-33633) Expose fine grained state information on SS UI

2020-12-02 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-33633: - Summary: Expose fine grained state information on SS UI Key: SPARK-33633 URL: https://issues.apache.org/jira/browse/SPARK-33633 Project: Spark Issue Type:

[jira] [Commented] (SPARK-33629) spark.buffer.size not applied in driver from pyspark

2020-12-02 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242195#comment-17242195 ] Gabor Somogyi commented on SPARK-33629: --- I've started to work on this and going to file a PR soon.

[jira] [Created] (SPARK-33629) spark.buffer.size not applied in driver from pyspark

2020-12-02 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-33629: - Summary: spark.buffer.size not applied in driver from pyspark Key: SPARK-33629 URL: https://issues.apache.org/jira/browse/SPARK-33629 Project: Spark Issue

[jira] [Updated] (SPARK-32910) Remove UninterruptibleThread usage from KafkaOffsetReader

2020-12-01 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-32910: -- Affects Version/s: (was: 3.1.0) 3.2.0 > Remove

[jira] [Commented] (SPARK-32910) Remove UninterruptibleThread usage from KafkaOffsetReader

2020-12-01 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241403#comment-17241403 ] Gabor Somogyi commented on SPARK-32910: --- I think there will be no time to put it into 3.1 so

[jira] [Created] (SPARK-33491) Update Structured Streaming UI documentation page

2020-11-19 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-33491: - Summary: Update Structured Streaming UI documentation page Key: SPARK-33491 URL: https://issues.apache.org/jira/browse/SPARK-33491 Project: Spark Issue

[jira] [Commented] (SPARK-33143) Make SocketAuthServer socket timeout configurable

2020-11-16 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232824#comment-17232824 ] Gabor Somogyi commented on SPARK-33143: --- [~mszurap] the OS and network guys are still working on

[jira] [Commented] (SPARK-33143) Make SocketAuthServer socket timeout configurable

2020-11-13 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231270#comment-17231270 ] Gabor Somogyi commented on SPARK-33143: --- [~hyukjin.kwon] thanks for the confirmation, started to

[jira] [Commented] (SPARK-33143) Make SocketAuthServer socket timeout configurable

2020-11-13 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231244#comment-17231244 ] Gabor Somogyi commented on SPARK-33143: --- I've had a look at this issue and under some

[jira] [Updated] (SPARK-33287) Expose state custom metrics information on SS UI

2020-10-29 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-33287: -- Affects Version/s: (was: 3.1.0) 3.0.1 > Expose state custom

[jira] [Commented] (SPARK-33273) Fix Flaky Test: ThriftServerQueryTestSuite. subquery_scalar_subquery_scalar_subquery_select_sql

2020-10-29 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17222896#comment-17222896 ] Gabor Somogyi commented on SPARK-33273: --- I've just faced with this too:

[jira] [Updated] (SPARK-33287) Expose state custom metrics information on SS UI

2020-10-29 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-33287: -- Description: Since not all custom metrics hold useful information it would be good to add

[jira] [Created] (SPARK-33287) Expose state custom metrics information on SS UI

2020-10-29 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-33287: - Summary: Expose state custom metrics information on SS UI Key: SPARK-33287 URL: https://issues.apache.org/jira/browse/SPARK-33287 Project: Spark Issue

[jira] [Commented] (SPARK-33222) Expose missing information (graphs) on SS UI

2020-10-22 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218974#comment-17218974 ] Gabor Somogyi commented on SPARK-33222: --- I've started to implement it. > Expose missing

[jira] [Created] (SPARK-33224) Expose watermark information on SS UI

2020-10-22 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-33224: - Summary: Expose watermark information on SS UI Key: SPARK-33224 URL: https://issues.apache.org/jira/browse/SPARK-33224 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-33223) Expose state information on SS UI

2020-10-22 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-33223: - Summary: Expose state information on SS UI Key: SPARK-33223 URL: https://issues.apache.org/jira/browse/SPARK-33223 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-33222) Expose missing information (graphs) on SS UI

2020-10-22 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-33222: - Summary: Expose missing information (graphs) on SS UI Key: SPARK-33222 URL: https://issues.apache.org/jira/browse/SPARK-33222 Project: Spark Issue Type:

[jira] [Commented] (SPARK-25547) Pluggable jdbc connection factory

2020-10-13 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213152#comment-17213152 ] Gabor Somogyi commented on SPARK-25547: --- [~fsauer65] JDBC connection provider API is added here:

[jira] [Commented] (SPARK-32229) Application entry parsing fails because DriverWrapper registered instead of the normal driver

2020-10-12 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212327#comment-17212327 ] Gabor Somogyi commented on SPARK-32229: --- Started to work on this. > Application entry parsing

[jira] [Commented] (SPARK-33102) Use stringToSeq on SQL list typed parameters

2020-10-09 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1723#comment-1723 ] Gabor Somogyi commented on SPARK-33102: --- Filing a PR soon... > Use stringToSeq on SQL list typed

[jira] [Created] (SPARK-33102) Use stringToSeq on SQL list typed parameters

2020-10-09 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-33102: - Summary: Use stringToSeq on SQL list typed parameters Key: SPARK-33102 URL: https://issues.apache.org/jira/browse/SPARK-33102 Project: Spark Issue Type:

[jira] [Commented] (SPARK-32047) Add provider disable possibility just like in delegation token provider

2020-10-02 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17206208#comment-17206208 ] Gabor Somogyi commented on SPARK-32047: --- I'm intended to file a PR next week... > Add provider

[jira] [Updated] (SPARK-28367) Kafka connector infinite wait because metadata never updated

2020-09-17 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-28367: -- Description: Spark uses an old and deprecated API named poll(long) which never returns and

[jira] [Commented] (SPARK-32910) Remove UninterruptibleThread usage from KafkaOffsetReader

2020-09-17 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197617#comment-17197617 ] Gabor Somogyi commented on SPARK-32910: --- Started to work on this. > Remove UninterruptibleThread

[jira] [Created] (SPARK-32910) Remove UninterruptibleThread usage from KafkaOffsetReader

2020-09-17 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-32910: - Summary: Remove UninterruptibleThread usage from KafkaOffsetReader Key: SPARK-32910 URL: https://issues.apache.org/jira/browse/SPARK-32910 Project: Spark

[jira] [Commented] (SPARK-32032) Eliminate deprecated poll(long) API calls to avoid infinite wait in driver

2020-09-11 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17194269#comment-17194269 ] Gabor Somogyi commented on SPARK-32032: --- [~Bartalos] query is not progressing. I've just finished

[jira] [Commented] (SPARK-32032) Eliminate deprecated poll(long) API calls to avoid infinite wait in driver

2020-09-08 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191999#comment-17191999 ] Gabor Somogyi commented on SPARK-32032: --- As a temporary result of the effort I've created a

[jira] [Commented] (SPARK-32520) Flaky Test: KafkaSourceStressSuite.stress test with multiple topics and partitions

2020-08-03 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170073#comment-17170073 ] Gabor Somogyi commented on SPARK-32520: --- Commented on

[jira] [Commented] (SPARK-32519) test of org.apache.spark.sql.kafka010.KafkaSourceStressSuite failed for aarch64

2020-08-03 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17169850#comment-17169850 ] Gabor Somogyi commented on SPARK-32519: --- I'm on 3 weeks vacation but had a slight look at it. This

[jira] [Commented] (SPARK-32032) Eliminate deprecated poll(long) API calls to avoid infinite wait in driver

2020-07-29 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167214#comment-17167214 ] Gabor Somogyi commented on SPARK-32032: --- I'm working on the solution. > Eliminate deprecated

[jira] [Commented] (SPARK-32032) Eliminate deprecated poll(long) API calls to avoid infinite wait in driver

2020-07-29 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167213#comment-17167213 ] Gabor Somogyi commented on SPARK-32032: --- I've renamed the jira because the solution is not simply

[jira] [Updated] (SPARK-32032) Eliminate deprecated poll(long) API calls to avoid infinite wait in driver

2020-07-29 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-32032: -- Summary: Eliminate deprecated poll(long) API calls to avoid infinite wait in driver (was:

[jira] [Created] (SPARK-32482) Eliminate deprecated poll(long) API calls to avoid infinite wait in tests

2020-07-29 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-32482: - Summary: Eliminate deprecated poll(long) API calls to avoid infinite wait in tests Key: SPARK-32482 URL: https://issues.apache.org/jira/browse/SPARK-32482 Project:

[jira] [Created] (SPARK-32468) Fix timeout config issue in Kafka connector tests

2020-07-28 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-32468: - Summary: Fix timeout config issue in Kafka connector tests Key: SPARK-32468 URL: https://issues.apache.org/jira/browse/SPARK-32468 Project: Spark Issue

  1   2   3   4   5   6   7   >