[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519445#comment-16519445 ] Thomas Graves commented on SPARK-24552: --- more details on hadoop committer side: So I think the

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519430#comment-16519430 ] Thomas Graves commented on SPARK-24552: --- this is actually a problem with hadoop committers, v1 and

[jira] [Commented] (SPARK-24611) Clean up OutputCommitCoordinator

[ https://issues.apache.org/jira/browse/SPARK-24611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519420#comment-16519420 ] Thomas Graves commented on SPARK-24611: --- [~joshrosen]  just noticed you were the last one to

[jira] [Commented] (SPARK-24622) Task attempts in other stage attempts not killed when one task attempt succeeds

[ https://issues.apache.org/jira/browse/SPARK-24622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519416#comment-16519416 ] Thomas Graves commented on SPARK-24622: --- Need to investigate further/test to make sure I am not

[jira] [Created] (SPARK-24622) Task attempts in other stage attempts not killed when one task attempt succeeds

Thomas Graves created SPARK-24622: - Summary: Task attempts in other stage attempts not killed when one task attempt succeeds Key: SPARK-24622 URL: https://issues.apache.org/jira/browse/SPARK-24622

[jira] [Updated] (SPARK-24519) MapStatus has 2000 hardcoded

[ https://issues.apache.org/jira/browse/SPARK-24519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24519: -- Description: MapStatus uses hardcoded value of 2000 partitions to determine if it should use

[jira] [Updated] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24552: -- Affects Version/s: 2.2.0 2.2.1 2.3.0

[jira] [Updated] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24552: -- Priority: Blocker (was: Critical) > Task attempt numbers are reused when stages are retried

[jira] [Commented] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512691#comment-16512691 ] Thomas Graves commented on SPARK-22148: --- ok, just update if you start working on it. thanks. >

[jira] [Commented] (SPARK-24539) HistoryServer does not display metrics from tasks that complete after stage failure

[ https://issues.apache.org/jira/browse/SPARK-24539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512643#comment-16512643 ] Thomas Graves commented on SPARK-24539: --- Its possible, I thought when I checked the history server

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512519#comment-16512519 ] Thomas Graves commented on SPARK-24552: --- sorry just realized the v2 api is still marked experiment

[jira] [Updated] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24552: -- Priority: Critical (was: Blocker) > Task attempt numbers are reused when stages are retried

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512504#comment-16512504 ] Thomas Graves commented on SPARK-24552: --- Note if this is a correctness bug and can cause data

[jira] [Updated] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24552: -- Priority: Blocker (was: Major) > Task attempt numbers are reused when stages are retried >

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512500#comment-16512500 ] Thomas Graves commented on SPARK-24552: --- I agree, I don't think changing the attempt number at

[jira] [Updated] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24415: -- Priority: Critical (was: Major) > Stage page aggregated executor metrics wrong when failures

[jira] [Commented] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495224#comment-16495224 ] Thomas Graves commented on SPARK-24415: --- ok so the issue here is in the AppStatusListener where

[jira] [Commented] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495204#comment-16495204 ] Thomas Graves commented on SPARK-24415: --- It also looks like in the history server they show up

[jira] [Commented] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495200#comment-16495200 ] Thomas Graves commented on SPARK-24415: --- this might actually be an order of events type thing. 

[jira] [Updated] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24415: -- Description: Running with spark 2.3 on yarn and having task failures and blacklisting, the

[jira] [Updated] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24415: -- Description: Running with spark 2.3 on yarn and having task failures and blacklisting, the

[jira] [Updated] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24415: -- Description: Running with spark 2.3 on yarn and having task failures and blacklisting, the

[jira] [Resolved] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and no active executors

[ https://issues.apache.org/jira/browse/SPARK-24413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24413. --- Resolution: Duplicate > Executor Blacklisting shouldn't immediately fail the application if

[jira] [Commented] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and no active executors

[ https://issues.apache.org/jira/browse/SPARK-24413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494279#comment-16494279 ] Thomas Graves commented on SPARK-24413: --- thanks for linking those we can just dup this to

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494265#comment-16494265 ] Thomas Graves commented on SPARK-24414: --- also just an fyi I also filed SPARK-24415, not sure if

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494237#comment-16494237 ] Thomas Graves commented on SPARK-24414: --- I am looking to see if we can just return an empty table

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494199#comment-16494199 ] Thomas Graves commented on SPARK-24414: --- looks like this was broken by SPARK-23147, so we probably

[jira] [Updated] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24415: -- Description: Running with spark 2.3 on yarn and having task failures and blacklisting, the

[jira] [Created] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

Thomas Graves created SPARK-24415: - Summary: Stage page aggregated executor metrics wrong when failures Key: SPARK-24415 URL: https://issues.apache.org/jira/browse/SPARK-24415 Project: Spark

[jira] [Updated] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24415: -- Attachment: Screen Shot 2018-05-29 at 2.15.38 PM.png > Stage page aggregated executor metrics

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494083#comment-16494083 ] Thomas Graves commented on SPARK-24414: --- to reproduce this simply start a shell:

[jira] [Created] (SPARK-24414) Stages page doesn't show all task attempts when failures

Thomas Graves created SPARK-24414: - Summary: Stages page doesn't show all task attempts when failures Key: SPARK-24414 URL: https://issues.apache.org/jira/browse/SPARK-24414 Project: Spark

[jira] [Commented] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and no active executors

[ https://issues.apache.org/jira/browse/SPARK-24413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493978#comment-16493978 ] Thomas Graves commented on SPARK-24413: --- [~imranr]  thoughts on this? > Executor Blacklisting

[jira] [Updated] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and no active executors

[ https://issues.apache.org/jira/browse/SPARK-24413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24413: -- Summary: Executor Blacklisting shouldn't immediately fail the application if dynamic

[jira] [Created] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and it doesn't have any other active executors

Thomas Graves created SPARK-24413: - Summary: Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and it doesn't have any other active executors Key: SPARK-24413

[jira] [Commented] (SPARK-6235) Address various 2G limits

[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16482746#comment-16482746 ] Thomas Graves commented on SPARK-6235: -- >> Still unsupported: * large task results * large blocks

[jira] [Commented] (SPARK-21033) fix the potential OOM in UnsafeExternalSorter

[ https://issues.apache.org/jira/browse/SPARK-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467803#comment-16467803 ] Thomas Graves commented on SPARK-21033: --- [~cloud_fan] the followup PR 

[jira] [Commented] (SPARK-24124) Spark history server should create spark.history.store.path and set permissions properly

[ https://issues.apache.org/jira/browse/SPARK-24124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458835#comment-16458835 ] Thomas Graves commented on SPARK-24124: --- [~vanzin]  any objections to this? > Spark history server

[jira] [Created] (SPARK-24124) Spark history server should create spark.history.store.path and set permissions properly

Thomas Graves created SPARK-24124: - Summary: Spark history server should create spark.history.store.path and set permissions properly Key: SPARK-24124 URL: https://issues.apache.org/jira/browse/SPARK-24124

[jira] [Assigned] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-22683: - Assignee: Julien Cuquemelle > DynamicAllocation wastes resources by allocating

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450131#comment-16450131 ] Thomas Graves commented on SPARK-22683: --- Note this added a new config 

[jira] [Resolved] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-22683. --- Resolution: Fixed Fix Version/s: 2.4.0 > DynamicAllocation wastes resources by

[jira] [Commented] (SPARK-23850) We should not redact username|user|url from UI by default

[ https://issues.apache.org/jira/browse/SPARK-23850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449054#comment-16449054 ] Thomas Graves commented on SPARK-23850: --- the url seems somewhat silly to me to, look at the

[jira] [Commented] (SPARK-23964) why does Spillable wait for 32 elements?

[ https://issues.apache.org/jira/browse/SPARK-23964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16444043#comment-16444043 ] Thomas Graves commented on SPARK-23964: --- so far in my testing I haven't seen any performance

[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440855#comment-16440855 ] Thomas Graves commented on SPARK-15703: --- this Jira is purely making the size of the event queue

[jira] [Commented] (YARN-8149) Revisit behavior of Re-Reservation in Capacity Scheduler

[ https://issues.apache.org/jira/browse/YARN-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436366#comment-16436366 ] Thomas Graves commented on YARN-8149: - thinking about this a little more, even with the current

[jira] [Commented] (YARN-8149) Revisit behavior of Re-Reservation in Capacity Scheduler

[ https://issues.apache.org/jira/browse/YARN-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436295#comment-16436295 ] Thomas Graves commented on YARN-8149: - are you going to do anything with starvation then or allocation

[jira] [Commented] (SPARK-23964) why does Spillable wait for 32 elements?

[ https://issues.apache.org/jira/browse/SPARK-23964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434479#comment-16434479 ] Thomas Graves commented on SPARK-23964: --- I'm not sure, I'm trying to figure out if there is a

[jira] [Updated] (SPARK-23964) why does Spillable wait for 32 elements?

[ https://issues.apache.org/jira/browse/SPARK-23964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-23964: -- Description: The spillable class has a check in maybeSpill as to when it tries to acquire

[jira] [Commented] (SPARK-23964) why does Spillable wait for 32 elements?

[ https://issues.apache.org/jira/browse/SPARK-23964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434454#comment-16434454 ] Thomas Graves commented on SPARK-23964: --- [~andrewor14]  [~matei] [~r...@databricks.com]   A few

[jira] [Updated] (SPARK-23964) why does Spillable wait for 32 elements?

[ https://issues.apache.org/jira/browse/SPARK-23964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-23964: -- Description: The spillable class has a check: if (elementsRead % 32 == 0 && currentMemory >=

[jira] [Updated] (SPARK-23964) why does Spillable wait for 32 elements?

[ https://issues.apache.org/jira/browse/SPARK-23964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-23964: -- Environment: (was: The spillable class has a check: if (elementsRead % 32 == 0 &&

[jira] [Created] (SPARK-23964) why does Spillable wait for 32 elements?

Thomas Graves created SPARK-23964: - Summary: why does Spillable wait for 32 elements? Key: SPARK-23964 URL: https://issues.apache.org/jira/browse/SPARK-23964 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-16630) Blacklist a node if executors won't launch on it.

[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432467#comment-16432467 ] Thomas Graves commented on SPARK-16630: --- sorry I don't follow, the list we get from the blacklist

[jira] [Commented] (SPARK-16630) Blacklist a node if executors won't launch on it.

[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432244#comment-16432244 ] Thomas Graves commented on SPARK-16630: --- yes I think it would make sense as the union of all

[jira] [Commented] (SPARK-16630) Blacklist a node if executors won't launch on it.

[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427377#comment-16427377 ] Thomas Graves commented on SPARK-16630: --- the problem is that spark.executor.instances (or dynamic

[jira] [Resolved] (SPARK-23567) spark.redaction.regex should not include user by default, docs not updated

[ https://issues.apache.org/jira/browse/SPARK-23567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-23567. --- Resolution: Duplicate > spark.redaction.regex should not include user by default, docs not

[jira] [Commented] (SPARK-23039) Fix the bug in alter table set location.

[ https://issues.apache.org/jira/browse/SPARK-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423996#comment-16423996 ] Thomas Graves commented on SPARK-23039: --- seems to be a dup of  # SPARK-23057 > Fix the bug in

[jira] [Commented] (SPARK-23850) We should not redact username|user|url from UI by default

[ https://issues.apache.org/jira/browse/SPARK-23850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422965#comment-16422965 ] Thomas Graves commented on SPARK-23850: --- ping [~ash...@gmail.com] [~onursatici] [~LI,Xiao]

[jira] [Created] (SPARK-23850) We should not redact username|user|url from UI by default

Thomas Graves created SPARK-23850: - Summary: We should not redact username|user|url from UI by default Key: SPARK-23850 URL: https://issues.apache.org/jira/browse/SPARK-23850 Project: Spark

[jira] [Updated] (SPARK-23806) Broadcast. unpersist can cause fatal exception when used with dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-23806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-23806: -- Description: Very similar to https://issues.apache.org/jira/browse/SPARK-22618 . But this

[jira] [Created] (SPARK-23806) Broadcast. unpersist can cause fatal exception when used with dynamic allocation

Thomas Graves created SPARK-23806: - Summary: Broadcast. unpersist can cause fatal exception when used with dynamic allocation Key: SPARK-23806 URL: https://issues.apache.org/jira/browse/SPARK-23806

[jira] [Commented] (SPARK-22618) RDD.unpersist can cause fatal exception when used with dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417466#comment-16417466 ] Thomas Graves commented on SPARK-22618: --- I'll file a separate Jira for it and put up a pr >

[jira] [Commented] (SPARK-22618) RDD.unpersist can cause fatal exception when used with dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416309#comment-16416309 ] Thomas Graves commented on SPARK-22618: --- thanks for fixing this, hitting it now in spark 2.2, I

[jira] [Commented] (SPARK-16630) Blacklist a node if executors won't launch on it.

[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390232#comment-16390232 ] Thomas Graves commented on SPARK-16630: --- yes yarn tells you the # of nodemanagers. allocateResponse

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387963#comment-16387963 ] Thomas Graves commented on SPARK-22683: --- I left comments on the open PR already, lets move the

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387938#comment-16387938 ] Thomas Graves commented on SPARK-22683: --- [~jcuquemelle] do you have time to update the PR,

[jira] [Commented] (SPARK-16630) Blacklist a node if executors won't launch on it.

[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387846#comment-16387846 ] Thomas Graves commented on SPARK-16630: --- yes something along these lines is what I was thinking. we 

[jira] [Comment Edited] (SPARK-23567) spark.redaction.regex should not include user by default, docs not updated

[ https://issues.apache.org/jira/browse/SPARK-23567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383723#comment-16383723 ] Thomas Graves edited comment on SPARK-23567 at 3/2/18 3:57 PM: --- I also

[jira] [Commented] (SPARK-22479) SaveIntoDataSourceCommand logs jdbc credentials

[ https://issues.apache.org/jira/browse/SPARK-22479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383733#comment-16383733 ] Thomas Graves commented on SPARK-22479: --- Also the example above shows the password, but the

[jira] [Commented] (SPARK-23567) spark.redaction.regex should not include user by default, docs not updated

[ https://issues.apache.org/jira/browse/SPARK-23567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383723#comment-16383723 ] Thomas Graves commented on SPARK-23567: --- I also question whether the url should be redacted by

[jira] [Created] (SPARK-23567) spark.redaction.regex should not include user by default, docs not updated

Thomas Graves created SPARK-23567: - Summary: spark.redaction.regex should not include user by default, docs not updated Key: SPARK-23567 URL: https://issues.apache.org/jira/browse/SPARK-23567

[jira] [Commented] (SPARK-22479) SaveIntoDataSourceCommand logs jdbc credentials

[ https://issues.apache.org/jira/browse/SPARK-22479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383709#comment-16383709 ] Thomas Graves commented on SPARK-22479: --- [~aash] [~onursatici] this seems to have redacted user

[jira] [Commented] (YARN-7935) Expose container's hostname to applications running within the docker container

[ https://issues.apache.org/jira/browse/YARN-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374598#comment-16374598 ] Thomas Graves commented on YARN-7935: - thanks for the explanation Mridul. I'm fine with waiting on the

[jira] [Commented] (YARN-7935) Expose container's hostname to applications running within the docker container

[ https://issues.apache.org/jira/browse/YARN-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373039#comment-16373039 ] Thomas Graves commented on YARN-7935: - [~mridulm80] what is the spark Jira for this?  If this goes in

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357346#comment-16357346 ] Thomas Graves commented on SPARK-23309: --- sorry I haven't had time to make a query/dataset to

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356168#comment-16356168 ] Thomas Graves commented on SPARK-22683: --- I agree, I think default behavior stays 1.  I ran a few

[jira] [Comment Edited] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356168#comment-16356168 ] Thomas Graves edited comment on SPARK-22683 at 2/7/18 10:24 PM: I agree,

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355920#comment-16355920 ] Thomas Graves commented on SPARK-22683: --- If the config is set to 1 which keeps the current behavior

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1637#comment-1637 ] Thomas Graves commented on SPARK-22683: --- ok thanks,  I would like to try this out myself on a few

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354336#comment-16354336 ] Thomas Graves commented on SPARK-23309: --- I pulled in that patch

[jira] [Comment Edited] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350901#comment-16350901 ] Thomas Graves edited comment on SPARK-23309 at 2/2/18 8:29 PM: --- I should

[jira] [Comment Edited] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350901#comment-16350901 ] Thomas Graves edited comment on SPARK-23309 at 2/2/18 8:29 PM: --- I should

[jira] [Comment Edited] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350813#comment-16350813 ] Thomas Graves edited comment on SPARK-23309 at 2/2/18 8:15 PM: --- I'm still

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350901#comment-16350901 ] Thomas Graves commented on SPARK-23309: --- I should ask is there a log statement or query plan I can

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350900#comment-16350900 ] Thomas Graves commented on SPARK-23309: --- So the last test I did was spark 2.3 with the old hive

[jira] [Comment Edited] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350813#comment-16350813 ] Thomas Graves edited comment on SPARK-23309 at 2/2/18 7:04 PM: --- I'm still

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350813#comment-16350813 ] Thomas Graves commented on SPARK-23309: --- I'm still seeing spark 2.3 slower by about 15% for the

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350533#comment-16350533 ] Thomas Graves commented on SPARK-23309: --- Note the schema of "something" here is a "string". I'll

[jira] [Resolved] (SPARK-23304) Spark SQL coalesce() against hive not working

[ https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-23304. --- Resolution: Invalid > Spark SQL coalesce() against hive not working >

[jira] [Commented] (SPARK-23304) Spark SQL coalesce() against hive not working

[ https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350440#comment-16350440 ] Thomas Graves commented on SPARK-23304: --- ok so I guess by that logic then the coalesce won't every

[jira] [Commented] (SPARK-23304) Spark SQL coalesce() against hive not working

[ https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350423#comment-16350423 ] Thomas Graves commented on SPARK-23304: --- it doesn't look like sql("xyz").rdd.partitions.length

[jira] [Commented] (SPARK-23304) Spark SQL coalesce() against hive not working

[ https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350428#comment-16350428 ] Thomas Graves commented on SPARK-23304: --- well I guess that give you end # of partitions and not the

[jira] [Commented] (SPARK-23304) Spark SQL coalesce() against hive not working

[ https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349555#comment-16349555 ] Thomas Graves commented on SPARK-23304: --- I don't have any hive tables backed by parquet to compare

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349553#comment-16349553 ] Thomas Graves commented on SPARK-23309: --- [~dongjoon] is there any native way with the native hive

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349551#comment-16349551 ] Thomas Graves commented on SPARK-23309: --- seeing the same time difference after adding in the   

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349483#comment-16349483 ] Thomas Graves commented on SPARK-23309: --- sure, I can also run with the  --conf

[jira] [Commented] (SPARK-23304) Spark SQL coalesce() against hive not working

[ https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349480#comment-16349480 ] Thomas Graves commented on SPARK-23304: --- I just ran the query (show()) and saw the # of partitions. 

[jira] [Updated] (SPARK-23304) Spark SQL coalesce() against hive not working

[ https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-23304: -- Attachment: spark23_oldorc_explain_convermetastoreorcfalse.txt > Spark SQL coalesce() against

[jira] [Commented] (SPARK-23304) Spark SQL coalesce() against hive not working

[ https://issues.apache.org/jira/browse/SPARK-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349446#comment-16349446 ] Thomas Graves commented on SPARK-23304: --- It still seems like a bug to me since the coalesce isn't

<    3   4   5   6   7   8   9   10   11   12   >