[jira] [Commented] (SPARK-24432) Add support for dynamic resource allocation

2021-03-18 Thread Erik Erlandson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304397#comment-17304397 ] Erik Erlandson commented on SPARK-24432: [~dongjoon] should this be closed, now

[jira] [Updated] (SPARK-32159) New udaf(Aggregator) has an integration bug with UnresolvedMapObjects serialization

2020-07-07 Thread Erik Erlandson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-32159: --- Fix Version/s: 3.0.1 > New udaf(Aggregator) has an integration bug with UnresolvedMapObjects

[jira] [Updated] (SPARK-32159) New udaf(Aggregator) has an integration bug with UnresolvedMapObjects serialization

2020-07-02 Thread Erik Erlandson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-32159: --- Description: The new user defined aggregator feature (SPARK-27296) based on calling 'functi

[jira] [Updated] (SPARK-32159) New udaf(Aggregator) has an integration bug with UnresolvedMapObjects serialization

2020-07-02 Thread Erik Erlandson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-32159: --- Description: The new user defined aggregator feature (SPARK-27296) based on calling 'functi

[jira] [Commented] (SPARK-32159) New udaf(Aggregator) has an integration bug with UnresolvedMapObjects serialization

2020-07-02 Thread Erik Erlandson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150490#comment-17150490 ] Erik Erlandson commented on SPARK-32159: cc [~cloud_fan] > New udaf(Aggregator)

[jira] [Updated] (SPARK-32159) New udaf(Aggregator) has an integration bug with UnresolvedMapObjects serialization

2020-07-02 Thread Erik Erlandson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-32159: --- Description: The new user defined aggregator feature (SPARK-27296) based on calling 'functi

[jira] [Created] (SPARK-32159) New udaf(Aggregator) has an integration bug with UnresolvedMapObjects serialization

2020-07-02 Thread Erik Erlandson (Jira)
Erik Erlandson created SPARK-32159: -- Summary: New udaf(Aggregator) has an integration bug with UnresolvedMapObjects serialization Key: SPARK-32159 URL: https://issues.apache.org/jira/browse/SPARK-32159

[jira] [Commented] (SPARK-30520) Eliminate deprecation warnings for UserDefinedAggregateFunction

2020-06-15 Thread Erik Erlandson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136153#comment-17136153 ] Erik Erlandson commented on SPARK-30520: Starting in spark 3.0, any custom aggre

[jira] [Commented] (SPARK-7768) Make user-defined type (UDT) API public

2020-05-28 Thread Erik Erlandson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119054#comment-17119054 ] Erik Erlandson commented on SPARK-7768: --- Are there any issues with making this kind

[jira] [Commented] (SPARK-30424) Change ExpressionEncoder toRow method to return UnsafeRow

2020-01-14 Thread Erik Erlandson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17015183#comment-17015183 ] Erik Erlandson commented on SPARK-30424: The main place this change causes a com

[jira] [Created] (SPARK-30424) Change ExpressionEncoder toRow method to return UnsafeRow

2020-01-05 Thread Erik Erlandson (Jira)
Erik Erlandson created SPARK-30424: -- Summary: Change ExpressionEncoder toRow method to return UnsafeRow Key: SPARK-30424 URL: https://issues.apache.org/jira/browse/SPARK-30424 Project: Spark

[jira] [Created] (SPARK-30423) Deprecate UserDefinedAggregateFunction

2020-01-05 Thread Erik Erlandson (Jira)
Erik Erlandson created SPARK-30423: -- Summary: Deprecate UserDefinedAggregateFunction Key: SPARK-30423 URL: https://issues.apache.org/jira/browse/SPARK-30423 Project: Spark Issue Type: Task

[jira] [Created] (SPARK-30422) deprecate UserDefinedAggregateFunction in favor of SPARK-27296

2020-01-05 Thread Erik Erlandson (Jira)
Erik Erlandson created SPARK-30422: -- Summary: deprecate UserDefinedAggregateFunction in favor of SPARK-27296 Key: SPARK-30422 URL: https://issues.apache.org/jira/browse/SPARK-30422 Project: Spark

[jira] [Assigned] (SPARK-29865) k8s executor pods all have different prefixes in client mode

2019-11-14 Thread Erik Erlandson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson reassigned SPARK-29865: -- Assignee: Marcelo Masiero Vanzin > k8s executor pods all have different prefixes in c

[jira] [Resolved] (SPARK-29865) k8s executor pods all have different prefixes in client mode

2019-11-14 Thread Erik Erlandson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson resolved SPARK-29865. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26488 [https:

[jira] [Updated] (SPARK-27296) Efficient User Defined Aggregators

2019-10-19 Thread Erik Erlandson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-27296: --- Fix Version/s: (was: 3.0.0) > Efficient User Defined Aggregators >

[jira] [Commented] (SPARK-27296) Efficient User Defined Aggregators

2019-10-19 Thread Erik Erlandson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955216#comment-16955216 ] Erik Erlandson commented on SPARK-27296: This started with the goal of fixing th

[jira] [Updated] (SPARK-27296) Efficient User Defined Aggregators

2019-10-19 Thread Erik Erlandson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-27296: --- Fix Version/s: 3.0.0 Summary: Efficient User Defined Aggregators (was: User Defin

[jira] [Commented] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

2019-07-24 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892037#comment-16892037 ] Erik Erlandson commented on SPARK-27812: Agreed with [~skonto] that downgrading

[jira] [Commented] (SPARK-27296) User Defined Aggregating Functions (UDAFs) have a major efficiency problem

2019-07-06 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879747#comment-16879747 ] Erik Erlandson commented on SPARK-27296: I wrote up my benchmarking results [he

[jira] [Updated] (SPARK-27296) User Defined Aggregating Functions (UDAFs) have a major efficiency problem

2019-07-05 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-27296: --- Target Version/s: 3.0.0 > User Defined Aggregating Functions (UDAFs) have a major efficiency

[jira] [Assigned] (SPARK-27296) User Defined Aggregating Functions (UDAFs) have a major efficiency problem

2019-07-05 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson reassigned SPARK-27296: -- Assignee: Erik Erlandson > User Defined Aggregating Functions (UDAFs) have a major ef

[jira] [Commented] (SPARK-27296) User Defined Aggregating Functions (UDAFs) have a major efficiency problem

2019-07-02 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16877374#comment-16877374 ] Erik Erlandson commented on SPARK-27296: The basic approach as described above a

[jira] [Commented] (SPARK-27936) Support local dependency uploading from --py-files

2019-06-03 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854998#comment-16854998 ] Erik Erlandson commented on SPARK-27936: cc [~skonto] > Support local dependenc

[jira] [Created] (SPARK-27936) Support local dependency uploading from --py-files

2019-06-03 Thread Erik Erlandson (JIRA)
Erik Erlandson created SPARK-27936: -- Summary: Support local dependency uploading from --py-files Key: SPARK-27936 URL: https://issues.apache.org/jira/browse/SPARK-27936 Project: Spark Issue

[jira] [Commented] (SPARK-27872) Driver and executors use a different service account breaking pull secrets

2019-05-29 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851026#comment-16851026 ] Erik Erlandson commented on SPARK-27872: [~skonto], executors were never given a

[jira] [Assigned] (SPARK-23153) Support application dependencies in submission client's local file system

2019-05-22 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson reassigned SPARK-23153: -- Assignee: Stavros Kontopoulos Fix Version/s: 3.0.0 > Support application dep

[jira] [Resolved] (SPARK-23153) Support application dependencies in submission client's local file system

2019-05-22 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson resolved SPARK-23153. Resolution: Fixed > Support application dependencies in submission client's local file sys

[jira] [Commented] (SPARK-27296) User Defined Aggregating Functions (UDAFs) have a major efficiency problem

2019-03-28 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804176#comment-16804176 ] Erik Erlandson commented on SPARK-27296: My initial proposal would be to alter t

[jira] [Created] (SPARK-27296) User Defined Aggregating Functions (UDAFs) have a major efficiency problem

2019-03-27 Thread Erik Erlandson (JIRA)
Erik Erlandson created SPARK-27296: -- Summary: User Defined Aggregating Functions (UDAFs) have a major efficiency problem Key: SPARK-27296 URL: https://issues.apache.org/jira/browse/SPARK-27296 Projec

[jira] [Commented] (SPARK-26973) Kubernetes version support strategy on test nodes / backend

2019-02-22 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16775435#comment-16775435 ] Erik Erlandson commented on SPARK-26973: A couple other points: * Currently, k8

[jira] [Updated] (SPARK-24434) Support user-specified driver and executor pod templates

2018-11-26 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-24434: --- Fix Version/s: 3.0.0 > Support user-specified driver and executor pod templates > --

[jira] [Assigned] (SPARK-25828) Bumping Version of kubernetes.client to latest version

2018-10-26 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson reassigned SPARK-25828: -- Assignee: Ilan Filonenko > Bumping Version of kubernetes.client to latest version > -

[jira] [Resolved] (SPARK-25828) Bumping Version of kubernetes.client to latest version

2018-10-26 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson resolved SPARK-25828. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22820 [https:

[jira] [Commented] (SPARK-25828) Bumping Version of kubernetes.client to latest version

2018-10-24 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662978#comment-16662978 ] Erik Erlandson commented on SPARK-25828: cc [~skonto] > Bumping Version of kube

[jira] [Commented] (SPARK-25782) Add PCA Aggregator to support grouping

2018-10-19 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657398#comment-16657398 ] Erik Erlandson commented on SPARK-25782: An ML Estimator also arguably would be

[jira] [Commented] (SPARK-25782) Add PCA Aggregator to support grouping

2018-10-19 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657394#comment-16657394 ] Erik Erlandson commented on SPARK-25782: Thanks [~mttsndrs]! I agree it makes s

[jira] [Updated] (SPARK-25782) Add PCA Aggregator to support grouping

2018-10-19 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-25782: --- Target Version/s: 3.0.0 Component/s: ML Issue Type: New Feature (was: Imp

[jira] [Updated] (SPARK-25128) multiple simultaneous job submissions against k8s backend cause driver pods to hang

2018-09-06 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-25128: --- Target Version/s: 3.0.0 (was: 2.4.0, 2.3.3) Priority: Minor (was: Major) > mul

[jira] [Commented] (SPARK-25128) multiple simultaneous job submissions against k8s backend cause driver pods to hang

2018-09-06 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605916#comment-16605916 ] Erik Erlandson commented on SPARK-25128: Retargeting to next release sounds good

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-08-31 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599299#comment-16599299 ] Erik Erlandson commented on SPARK-24434: To amplify a little from my points abov

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-08-31 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599283#comment-16599283 ] Erik Erlandson commented on SPARK-24434: Stavros, yes, I knew you were working o

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-08-31 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599092#comment-16599092 ] Erik Erlandson commented on SPARK-24434: There are a few related, but separate,

[jira] [Resolved] (SPARK-25287) Check for JIRA_USERNAME and JIRA_PASSWORD up front in merge_spark_pr.py

2018-08-30 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson resolved SPARK-25287. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22294 [https:

[jira] [Assigned] (SPARK-25287) Check for JIRA_USERNAME and JIRA_PASSWORD up front in merge_spark_pr.py

2018-08-30 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson reassigned SPARK-25287: -- Assignee: Erik Erlandson > Check for JIRA_USERNAME and JIRA_PASSWORD up front in merg

[jira] [Created] (SPARK-25287) Check for JIRA_USERNAME and JIRA_PASSWORD up front in merge_spark_pr.py

2018-08-30 Thread Erik Erlandson (JIRA)
Erik Erlandson created SPARK-25287: -- Summary: Check for JIRA_USERNAME and JIRA_PASSWORD up front in merge_spark_pr.py Key: SPARK-25287 URL: https://issues.apache.org/jira/browse/SPARK-25287 Project:

[jira] [Resolved] (SPARK-25275) require memberhip in wheel to run 'su' (in dockerfiles)

2018-08-30 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson resolved SPARK-25275. Resolution: Fixed Fix Version/s: 2.4.0 > require memberhip in wheel to run 'su' (in

[jira] [Commented] (SPARK-25275) require memberhip in wheel to run 'su' (in dockerfiles)

2018-08-30 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16597935#comment-16597935 ] Erik Erlandson commented on SPARK-25275: {{merge_spark_pr.py}} failed to close t

[jira] [Created] (SPARK-25275) require memberhip in wheel to run 'su' (in dockerfiles)

2018-08-29 Thread Erik Erlandson (JIRA)
Erik Erlandson created SPARK-25275: -- Summary: require memberhip in wheel to run 'su' (in dockerfiles) Key: SPARK-25275 URL: https://issues.apache.org/jira/browse/SPARK-25275 Project: Spark I

[jira] [Commented] (SPARK-21097) Dynamic allocation will preserve cached data

2018-08-27 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594271#comment-16594271 ] Erik Erlandson commented on SPARK-21097: I'm wondering if this is going to be su

[jira] [Commented] (SPARK-7768) Make user-defined type (UDT) API public

2018-08-22 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16588934#comment-16588934 ] Erik Erlandson commented on SPARK-7768: --- We use `UserDefinedType`, for example here

[jira] [Commented] (SPARK-25128) multiple simultaneous job submissions against k8s backend cause driver pods to hang

2018-08-15 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16581537#comment-16581537 ] Erik Erlandson commented on SPARK-25128: [~mcheah], [~liyinan926], wdyt? > mult

[jira] [Created] (SPARK-25128) multiple simultaneous job submissions against k8s backend cause driver pods to hang

2018-08-15 Thread Erik Erlandson (JIRA)
Erik Erlandson created SPARK-25128: -- Summary: multiple simultaneous job submissions against k8s backend cause driver pods to hang Key: SPARK-25128 URL: https://issues.apache.org/jira/browse/SPARK-25128

[jira] [Commented] (SPARK-24817) Implement BarrierTaskContext.barrier()

2018-08-02 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567358#comment-16567358 ] Erik Erlandson commented on SPARK-24817: I have been looking at the use cases fo

[jira] [Commented] (SPARK-24817) Implement BarrierTaskContext.barrier()

2018-08-02 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567301#comment-16567301 ] Erik Erlandson commented on SPARK-24817: Thanks [~jiangxb] - I'd expect that des

[jira] [Commented] (SPARK-24817) Implement BarrierTaskContext.barrier()

2018-08-01 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566159#comment-16566159 ] Erik Erlandson commented on SPARK-24817: I'm curious about what the {{barrier}}

[jira] [Commented] (SPARK-24580) List scenarios to be handled by barrier execution mode properly

2018-08-01 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566154#comment-16566154 ] Erik Erlandson commented on SPARK-24580: This is blocking SPARK-24582 which is m

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

2018-07-31 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16564558#comment-16564558 ] Erik Erlandson commented on SPARK-24615: Am I understanding correctly that this

[jira] [Comment Edited] (SPARK-24793) Make spark-submit more useful with k8s

2018-07-12 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542102#comment-16542102 ] Erik Erlandson edited comment on SPARK-24793 at 7/12/18 7:01 PM: -

[jira] [Commented] (SPARK-24793) Make spark-submit more useful with k8s

2018-07-12 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542102#comment-16542102 ] Erik Erlandson commented on SPARK-24793: Also a good point that {{--kill}} and {

[jira] [Commented] (SPARK-24793) Make spark-submit more useful with k8s

2018-07-12 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541930#comment-16541930 ] Erik Erlandson commented on SPARK-24793: Another possible angle (not mutually ex

[jira] [Commented] (SPARK-24793) Make spark-submit more useful with k8s

2018-07-12 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541923#comment-16541923 ] Erik Erlandson commented on SPARK-24793: I am concerned that this is outside the

[jira] [Resolved] (SPARK-24534) Add a way to bypass entrypoint.sh script if no spark cmd is passed

2018-06-19 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson resolved SPARK-24534. Resolution: Fixed Fix Version/s: 2.4.0 > Add a way to bypass entrypoint.sh script i

[jira] [Commented] (SPARK-24534) Add a way to bypass entrypoint.sh script if no spark cmd is passed

2018-06-13 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511804#comment-16511804 ] Erik Erlandson commented on SPARK-24534: I think this has potential use for cust

[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates

2018-06-01 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16498337#comment-16498337 ] Erik Erlandson edited comment on SPARK-24434 at 6/1/18 5:53 PM: --

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-06-01 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16498337#comment-16498337 ] Erik Erlandson commented on SPARK-24434: My current take on UX around this featu

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-05-31 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496991#comment-16496991 ] Erik Erlandson commented on SPARK-24434: [~foxish] is there a technical (or ux)

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-05-30 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495642#comment-16495642 ] Erik Erlandson commented on SPARK-24434: [~skonto] given the number of ideas tha

[jira] [Commented] (SPARK-24091) Internally used ConfigMap prevents use of user-specified ConfigMaps carrying Spark configs files

2018-05-30 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495619#comment-16495619 ] Erik Erlandson commented on SPARK-24091: If we support user-supplied yaml, that

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-05-30 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495615#comment-16495615 ] Erik Erlandson commented on SPARK-24434: Is the template-based solution being ex

[jira] [Resolved] (SPARK-24435) Support user-supplied YAML that can be merged with k8s pod descriptions

2018-05-30 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson resolved SPARK-24435. Resolution: Duplicate > Support user-supplied YAML that can be merged with k8s pod descrip

[jira] [Created] (SPARK-24435) Support user-supplied YAML that can be merged with k8s pod descriptions

2018-05-30 Thread Erik Erlandson (JIRA)
Erik Erlandson created SPARK-24435: -- Summary: Support user-supplied YAML that can be merged with k8s pod descriptions Key: SPARK-24435 URL: https://issues.apache.org/jira/browse/SPARK-24435 Project:

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

2018-05-16 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477812#comment-16477812 ] Erik Erlandson commented on SPARK-24248: Is the design above using re-sync as the

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

2018-05-02 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461324#comment-16461324 ] Erik Erlandson commented on SPARK-24135: > In the case of the executor failing to

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

2018-05-02 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461154#comment-16461154 ] Erik Erlandson commented on SPARK-24135: IIRC the dynamic allocation heuristic wa

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

2018-05-01 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459954#comment-16459954 ] Erik Erlandson commented on SPARK-24135: I think it makes sense to detect these f

[jira] [Commented] (SPARK-23891) Debian based Dockerfile

2018-04-14 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438410#comment-16438410 ] Erik Erlandson commented on SPARK-23891: [~SercanKaraoglu] thanks for the informa

[jira] [Commented] (SPARK-23891) Debian based Dockerfile

2018-04-12 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436196#comment-16436196 ] Erik Erlandson commented on SPARK-23891: I do think that these reports are very u

[jira] [Commented] (SPARK-23891) Debian based Dockerfile

2018-04-12 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436185#comment-16436185 ] Erik Erlandson commented on SPARK-23891: The question of what OS base to use for

[jira] [Updated] (SPARK-23891) Debian based Dockerfile

2018-04-12 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-23891: --- Priority: Minor (was: Major) Issue Type: New Feature (was: Bug) > Debian based Docker

[jira] [Commented] (SPARK-23680) entrypoint.sh does not accept arbitrary UIDs, returning as an error

2018-03-16 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16402266#comment-16402266 ] Erik Erlandson commented on SPARK-23680: commit workflow indicates to set the Ass

[jira] [Resolved] (SPARK-23680) entrypoint.sh does not accept arbitrary UIDs, returning as an error

2018-03-16 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson resolved SPARK-23680. Resolution: Fixed Target Version/s: 2.3.1, 2.4.0 merged to master > entrypoint.s

[jira] [Updated] (SPARK-23680) entrypoint.sh does not accept arbitrary UIDs, returning as an error

2018-03-14 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-23680: --- Flags: Important Labels: easyfix (was: ) > entrypoint.sh does not accept arbitrary UIDs

[jira] [Commented] (SPARK-23680) entrypoint.sh does not accept arbitrary UIDs, returning as an error

2018-03-14 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398701#comment-16398701 ] Erik Erlandson commented on SPARK-23680: [~rmartine] thanks for catching this! It

[jira] [Commented] (SPARK-23324) Announce new Kubernetes back-end for 2.3 release notes

2018-02-02 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351020#comment-16351020 ] Erik Erlandson commented on SPARK-23324: cc [~sameer], [~foxish] > Announce new

[jira] [Created] (SPARK-23324) Announce new Kubernetes back-end for 2.3 release notes

2018-02-02 Thread Erik Erlandson (JIRA)
Erik Erlandson created SPARK-23324: -- Summary: Announce new Kubernetes back-end for 2.3 release notes Key: SPARK-23324 URL: https://issues.apache.org/jira/browse/SPARK-23324 Project: Spark Is

[jira] [Commented] (SPARK-23137) spark.kubernetes.executor.podNamePrefix is ignored

2018-01-17 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329721#comment-16329721 ] Erik Erlandson commented on SPARK-23137: +1, a more general "app prefix" seems mo

[jira] [Commented] (SPARK-22647) Docker files for image creation

2017-12-13 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289988#comment-16289988 ] Erik Erlandson commented on SPARK-22647: I'd like to propose migrating our images

[jira] [Commented] (SPARK-21277) Spark is invoking an incorrect serializer after UDAF completion

2017-07-05 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16075818#comment-16075818 ] Erik Erlandson commented on SPARK-21277: It would be ideal to document the requir

[jira] [Created] (SPARK-21277) Spark is invoking an incorrect serializer after UDAF completion

2017-07-01 Thread Erik Erlandson (JIRA)
Erik Erlandson created SPARK-21277: -- Summary: Spark is invoking an incorrect serializer after UDAF completion Key: SPARK-21277 URL: https://issues.apache.org/jira/browse/SPARK-21277 Project: Spark

[jira] [Commented] (SPARK-10915) Add support for UDAFs in Python

2017-06-27 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065787#comment-16065787 ] Erik Erlandson commented on SPARK-10915: This would be great for exposing {{TDige

[jira] [Created] (SPARK-21026) Document jenkins plug-ins assumed by the spark documentation build

2017-06-08 Thread Erik Erlandson (JIRA)
Erik Erlandson created SPARK-21026: -- Summary: Document jenkins plug-ins assumed by the spark documentation build Key: SPARK-21026 URL: https://issues.apache.org/jira/browse/SPARK-21026 Project: Spark

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

2016-12-08 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733439#comment-15733439 ] Erik Erlandson commented on SPARK-18278: As I understand it (and as I've built th

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

2016-12-03 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15718249#comment-15718249 ] Erik Erlandson commented on SPARK-18278: Not publishing images puts users in the

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

2016-12-03 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15718240#comment-15718240 ] Erik Erlandson commented on SPARK-18278: A possible scheme might be to publish th

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

2016-11-07 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15644919#comment-15644919 ] Erik Erlandson commented on SPARK-18278: Another comment on external plug-ins for

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

2016-11-07 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15644841#comment-15644841 ] Erik Erlandson commented on SPARK-18278: I agree with [~willbenton] that Kube is

[jira] [Updated] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

2016-11-04 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-18278: --- External issue URL: https://github.com/kubernetes/kubernetes/issues/34377 External issue

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

2016-11-04 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15637591#comment-15637591 ] Erik Erlandson commented on SPARK-18278: Current prototype: https://github.com/fo

[jira] [Created] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

2016-11-04 Thread Erik Erlandson (JIRA)
Erik Erlandson created SPARK-18278: -- Summary: Support native submission of spark jobs to a kubernetes cluster Key: SPARK-18278 URL: https://issues.apache.org/jira/browse/SPARK-18278 Project: Spark

  1   2   >