[jira] [Updated] (SPARK-26560) Repeating select on udf function throws analysis exception - function not registered

[ https://issues.apache.org/jira/browse/SPARK-26560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26560: --- Component/s: (was: Spark Core) SQL > Repeating select on udf funct

[jira] [Updated] (SPARK-26589) proper `median` method for spark dataframe

[ https://issues.apache.org/jira/browse/SPARK-26589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26589: --- Component/s: (was: Spark Core) SQL > proper `median` method for sp

[jira] [Updated] (SPARK-26769) partition prunning in inner join

[ https://issues.apache.org/jira/browse/SPARK-26769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26769: --- Component/s: (was: Spark Core) SQL > partition prunning in inner j

[jira] [Updated] (SPARK-26800) JDBC - MySQL nullable option is ignored

[ https://issues.apache.org/jira/browse/SPARK-26800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26800: --- Component/s: (was: Spark Core) SQL > JDBC - MySQL nullable opt

[jira] [Commented] (SPARK-26855) SparkSubmitSuite fails on a clean build

[ https://issues.apache.org/jira/browse/SPARK-26855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766531#comment-16766531 ] Marcelo Vanzin commented on SPARK-26855: Could R-related tests be moved to the "R"

[jira] [Resolved] (SPARK-1502) Spark on Yarn: add config option to not include yarn/mapred cluster classpath

[ https://issues.apache.org/jira/browse/SPARK-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-1502. --- Resolution: Won't Fix I don't think a new setting is worth it. You can use

[jira] [Assigned] (SPARK-1502) Spark on Yarn: add config option to not include yarn/mapred cluster classpath

[ https://issues.apache.org/jira/browse/SPARK-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-1502: - Assignee: (was: Marcelo Vanzin) > Spark on Yarn: add config option to not incl

[jira] [Assigned] (SPARK-1502) Spark on Yarn: add config option to not include yarn/mapred cluster classpath

[ https://issues.apache.org/jira/browse/SPARK-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-1502: - Assignee: Marcelo Vanzin (was: Thomas Graves) > Spark on Yarn: add config opt

[jira] [Resolved] (SPARK-1537) Add integration with Yarn's Application Timeline Server

[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-1537. --- Resolution: Won't Fix I don't think it makes sense to integrate with the ATS at this point

[jira] [Resolved] (SPARK-19649) Spark YARN client throws exception if job succeeds and max-completed-applications=0

[ https://issues.apache.org/jira/browse/SPARK-19649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-19649. Resolution: Won't Fix There isn't a reliable way to do this in Spark without the RM

[jira] [Resolved] (SPARK-17667) Make locking fine grained in YarnAllocator#enqueueGetLossReasonRequest

[ https://issues.apache.org/jira/browse/SPARK-17667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-17667. Resolution: Won't Fix PR was abandoned. Let's close this one. > Make locking f

[jira] [Resolved] (SPARK-13852) handle the InterruptedException caused by YARN HA switch

[ https://issues.apache.org/jira/browse/SPARK-13852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-13852. Resolution: Won't Fix See discussion in PR. > handle the InterruptedException cau

[jira] [Resolved] (SPARK-15974) Create a socket on YARN AM start-up

[ https://issues.apache.org/jira/browse/SPARK-15974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-15974. Resolution: Won't Fix The way this feature is proposed, I don't agree

[jira] [Resolved] (SPARK-19941) Spark should not schedule tasks on executors on decommissioning YARN nodes

[ https://issues.apache.org/jira/browse/SPARK-19941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-19941. Resolution: Duplicate > Spark should not schedule tasks on executors on decommission

[jira] [Assigned] (SPARK-17209) Support manual credential updating in the run-time for Spark on YARN

[ https://issues.apache.org/jira/browse/SPARK-17209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-17209: -- Assignee: Marcelo Vanzin > Support manual credential updating in the run-t

[jira] [Assigned] (SPARK-17209) Support manual credential updating in the run-time for Spark on YARN

[ https://issues.apache.org/jira/browse/SPARK-17209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-17209: -- Assignee: (was: Marcelo Vanzin) > Support manual credential updating in the

[jira] [Resolved] (SPARK-18898) Exception not failing Scala applications (in yarn)

[ https://issues.apache.org/jira/browse/SPARK-18898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-18898. Resolution: Cannot Reproduce I'm very sure this works in cluster mode. In client mode

[jira] [Resolved] (SPARK-15955) Failed Spark application returns with exitcode equals to zero

[ https://issues.apache.org/jira/browse/SPARK-15955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-15955. Resolution: Cannot Reproduce I'm pretty sure this has worked reliably for a while

[jira] [Resolved] (SPARK-19894) Tasks entirely assigned to one executor on Yarn-cluster mode for default-rack

[ https://issues.apache.org/jira/browse/SPARK-19894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-19894. Resolution: Won't Fix See discussion in PR. My reading is that it was decided

[jira] [Resolved] (SPARK-21933) Spark Streaming request more executors than excepted without DynamicAllocation

[ https://issues.apache.org/jira/browse/SPARK-21933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21933. Resolution: Won't Fix See discussion in PR. > Spark Streaming request more execut

[jira] [Resolved] (SPARK-22199) Spark Job on YARN fails with executors "Slave registration failed"

[ https://issues.apache.org/jira/browse/SPARK-22199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-22199. Resolution: Cannot Reproduce So much has changed since 1.6 that I'll make a guess

[jira] [Resolved] (SPARK-22341) [2.3.0] cannot run Spark on Yarn when Yarn impersonation is turned off

[ https://issues.apache.org/jira/browse/SPARK-22341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-22341. Resolution: Fixed This was fixed, forgot to close. > [2.3.0] cannot run Spark on Y

[jira] [Commented] (SPARK-26101) Spark Pipe() executes the external app by yarn username not the current username

[ https://issues.apache.org/jira/browse/SPARK-26101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766475#comment-16766475 ] Marcelo Vanzin commented on SPARK-26101: Spark does tell YARN which user it wants. But if YARN

[jira] [Resolved] (SPARK-22760) where driver is stopping, and some executors lost because of YarnSchedulerBackend.stop, then there is a problem.

[ https://issues.apache.org/jira/browse/SPARK-22760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-22760. Resolution: Won't Fix See discussion in PR. It's just a misleading exception. Not worth

[jira] [Resolved] (SPARK-23497) Sparklyr Applications doesn't disconnect spark driver in client mode

[ https://issues.apache.org/jira/browse/SPARK-23497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23497. Resolution: Invalid See above. > Sparklyr Applications doesn't disconnect spark dri

[jira] [Resolved] (SPARK-24205) java.util.concurrent.locks.LockSupport.parkNanos

[ https://issues.apache.org/jira/browse/SPARK-24205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24205. Resolution: Invalid There isn't enough information here to see what's going on. If you

[jira] [Resolved] (SPARK-24700) AM shutdown terminates streaming application but marks it as successful

[ https://issues.apache.org/jira/browse/SPARK-24700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24700. Resolution: Cannot Reproduce This needs more details, like logs if available. AFAIK

[jira] [Commented] (SPARK-26650) Yarn Client throws 'ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration'

[ https://issues.apache.org/jira/browse/SPARK-26650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766454#comment-16766454 ] Marcelo Vanzin commented on SPARK-26650: So nothing is being *thrown* here. It's just

[jira] [Assigned] (SPARK-25869) Spark on YARN: the original diagnostics is missing when job failed maxAppAttempts times

[ https://issues.apache.org/jira/browse/SPARK-25869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25869: -- Assignee: (was: Marcelo Vanzin) > Spark on YARN: the original diagnost

[jira] [Assigned] (SPARK-25869) Spark on YARN: the original diagnostics is missing when job failed maxAppAttempts times

[ https://issues.apache.org/jira/browse/SPARK-25869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25869: -- Assignee: Marcelo Vanzin > Spark on YARN: the original diagnostics is missing w

[jira] [Resolved] (SPARK-26101) Spark Pipe() executes the external app by yarn username not the current username

[ https://issues.apache.org/jira/browse/SPARK-26101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26101. Resolution: Not A Problem If you want YARN to start child processes as specific users

[jira] [Assigned] (SPARK-26650) Yarn Client throws 'ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration'

[ https://issues.apache.org/jira/browse/SPARK-26650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26650: -- Assignee: (was: Marcelo Vanzin) > Yarn Client throws 'ClassNotFoundExcept

[jira] [Assigned] (SPARK-26650) Yarn Client throws 'ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration'

[ https://issues.apache.org/jira/browse/SPARK-26650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26650: -- Assignee: Marcelo Vanzin > Yarn Client throws 'ClassNotFoundExcept

[jira] [Resolved] (SPARK-2624) Datanucleus jars not accessible in yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-2624. --- Resolution: Cannot Reproduce Pretty sure this is not an issue since 2.0. > Datanucleus j

[jira] [Resolved] (SPARK-7007) Add metrics source for ExecutorAllocationManager to expose internal status

[ https://issues.apache.org/jira/browse/SPARK-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-7007. --- Resolution: Fixed This was fixed but not closed. https://github.com/apache/spark/commit

[jira] [Resolved] (SPARK-7941) Cache Cleanup Failure when job is killed by Spark

[ https://issues.apache.org/jira/browse/SPARK-7941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-7941. --- Resolution: Cannot Reproduce I'm pretty sure this has been fixed since 1.3. All YARN temp

[jira] [Resolved] (SPARK-11315) Add YARN extension service to publish Spark events to YARN timeline service

[ https://issues.apache.org/jira/browse/SPARK-11315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-11315. Resolution: Won't Fix Pretty sure we've given up ATS integration at this point in time

[jira] [Resolved] (SPARK-9468) Avoid scheduling tasks on preemption executors

[ https://issues.apache.org/jira/browse/SPARK-9468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-9468. --- Resolution: Won't Fix See discussion in PR. > Avoid scheduling tasks on preempt

[jira] [Assigned] (SPARK-10288) Add a rest client for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-10288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-10288: -- Assignee: (was: Marcelo Vanzin) > Add a rest client for Spark on Y

[jira] [Assigned] (SPARK-10288) Add a rest client for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-10288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-10288: -- Assignee: Marcelo Vanzin > Add a rest client for Spark on Y

[jira] [Comment Edited] (SPARK-10766) Add some configurations for the client process in yarn-cluster mode.

[ https://issues.apache.org/jira/browse/SPARK-10766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766438#comment-16766438 ] Marcelo Vanzin edited comment on SPARK-10766 at 2/12/19 8:49 PM

[jira] [Resolved] (SPARK-13232) YARN executor node label expressions

[ https://issues.apache.org/jira/browse/SPARK-13232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-13232. Resolution: Won't Fix It seems the conclusion in the bug is that this is a YARN bug (YARN

[jira] [Resolved] (SPARK-10766) Add some configurations for the client process in yarn-cluster mode.

[ https://issues.apache.org/jira/browse/SPARK-10766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-10766. Resolution: Not A Problem There's a bunch of config options already

[jira] [Resolved] (SPARK-11182) HDFS Delegation Token will be expired when calling "UserGroupInformation.getCurrentUser.addCredentials" in HA mode

[ https://issues.apache.org/jira/browse/SPARK-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-11182. Resolution: Not A Problem HDFS bug. > HDFS Delegation Token will be expired when call

[jira] [Resolved] (SPARK-11323) Add History Service Provider to service application histories from YARN timeline server

[ https://issues.apache.org/jira/browse/SPARK-11323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-11323. Resolution: Won't Fix I think it's safe to say we've given up on this. > Add Hist

[jira] [Resolved] (SPARK-11662) Call startExecutorDelegationTokenRenewer() ahead of client app submission

[ https://issues.apache.org/jira/browse/SPARK-11662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-11662. Resolution: Cannot Reproduce Pretty sure this works these days. > C

[jira] [Resolved] (SPARK-13829) Spark submit with keytab can't submit to a non-HDFS yarn cluster

[ https://issues.apache.org/jira/browse/SPARK-13829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-13829. Resolution: Cannot Reproduce I'm very sure the current code handles things fine when

[jira] [Resolved] (SPARK-12516) Properly handle NM failure situation for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-12516. Resolution: Cannot Reproduce Seems like work-preserving restart fixes the issue

[jira] [Resolved] (SPARK-14173) Ignoring config property “spark.executor.extraJavaOptions”

[ https://issues.apache.org/jira/browse/SPARK-14173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-14173. Resolution: Cannot Reproduce I'm pretty sure this works since we use it all the time

[jira] [Resolved] (SPARK-24751) [k8s] Relaunch failed executor pods

[ https://issues.apache.org/jira/browse/SPARK-24751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24751. Resolution: Cannot Reproduce Closing based on above information. > [k8s] Relaunch fai

[jira] [Resolved] (SPARK-24105) Spark 2.3.0 on kubernetes

[ https://issues.apache.org/jira/browse/SPARK-24105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24105. Resolution: Won't Fix Most probably covered by pod templates. Also the bug summary

[jira] [Resolved] (SPARK-24247) [K8S] currentNodeToLocalTaskCount is unused in KubernetesClusterSchedulerBackend

[ https://issues.apache.org/jira/browse/SPARK-24247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24247. Resolution: Not A Problem No such variable. > [K8S] currentNodeToLocalTaskCo

[jira] [Resolved] (SPARK-24249) Spark on kubernetes, pods crashes with spark sql job.

[ https://issues.apache.org/jira/browse/SPARK-24249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24249. Resolution: Invalid This was filed with a non-ASF fork of Spark; also, this is most

[jira] [Resolved] (SPARK-24857) required the sample code test the spark steaming job in kubernates and write the data in remote hdfs file system

[ https://issues.apache.org/jira/browse/SPARK-24857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24857. Resolution: Invalid There's not enough data for the original problem (e.g. logs

[jira] [Resolved] (SPARK-26125) Delegation Token seems not appropriately stored on secrets of Kubernetes/Kerberized HDFS

[ https://issues.apache.org/jira/browse/SPARK-26125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26125. Resolution: Fixed Fix Version/s: 3.0.0 Pretty sure this was fixed as part

[jira] [Assigned] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24793: -- Assignee: (was: Anirudh Ramanathan) > Make spark-submit more useful with

[jira] [Resolved] (SPARK-24903) Allow driver container name to be configurable in k8 spark

[ https://issues.apache.org/jira/browse/SPARK-24903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24903. Resolution: Won't Fix According to comment on the PR, this is covered by pod templaes

[jira] [Resolved] (SPARK-25194) Kubernetes - Define cpu and memory limit to init container

[ https://issues.apache.org/jira/browse/SPARK-25194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25194. Resolution: Won't Fix Init containers were removed in 2.4. > Kubernetes - Define

[jira] [Resolved] (SPARK-26401) [k8s] Init container drops necessary config options for pulling jars from azure storage

[ https://issues.apache.org/jira/browse/SPARK-26401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26401. Resolution: Won't Fix Init containers were removed in 2.4; your configuration should

[jira] [Resolved] (SPARK-26400) [k8s] Init container silently swallows errors when fetching jars from remote url

[ https://issues.apache.org/jira/browse/SPARK-26400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26400. Resolution: Won't Fix Init containers were removed in 2.4, so standard Spark error

[jira] [Resolved] (SPARK-24179) History Server for Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-24179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24179. Resolution: Won't Fix You should be able to create your own pod to run the SHS; all

[jira] [Resolved] (SPARK-26789) [k8s] pyspark needs to upload local resources to driver and executor pods

[ https://issues.apache.org/jira/browse/SPARK-26789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26789. Resolution: Duplicate > [k8s] pyspark needs to upload local resources to dri

Re: building docker images for GPU

I think I remember someone mentioning a thread about this on the PR discussion, and digging a bit I found this: http://apache-spark-developers-list.1001551.n3.nabble.com/Toward-an-quot-API-quot-for-spark-images-used-by-the-Kubernetes-back-end-td23622.html It started a discussion but I haven't

[jira] [Resolved] (SPARK-25254) docker-image-tool should allow not building images for R and Python

[ https://issues.apache.org/jira/browse/SPARK-25254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25254. Resolution: Duplicate > docker-image-tool should allow not building images fo

Re: [VOTE] Release Apache Spark 2.3.3 (RC2)

+1. Ran our regression tests for YARN and Hive, all look good. On Tue, Feb 5, 2019 at 5:07 PM Takeshi Yamamuro wrote: > > Please vote on releasing the following candidate as Apache Spark version > 2.3.3. > > The vote is open until February 8 6:00PM (PST) and passes if a majority +1 > PMC votes

[jira] [Commented] (LIVY-508) Support custom auth filter for LivyServer

[ https://issues.apache.org/jira/browse/LIVY-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765223#comment-16765223 ] Marcelo Vanzin commented on LIVY-508: - Done, also added you to the right jira group (forgot to do

[jira] [Resolved] (LIVY-508) Support custom auth filter for LivyServer

[ https://issues.apache.org/jira/browse/LIVY-508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-508. - Resolution: Fixed > Support custom auth filter for LivySer

Re: Multiplicity of GitBox messages

I checked which mailing lists exist for commons, and it looks like different repos are configured differently. e.g. commons-lang has github updates going to issues@, commons-numbers to notifications@, and a bunch, as we can see, to dev@. As long as it's consistent, I'm ok with anything, although

Re: [VOTE] Release Apache Spark 2.3.3 (RC2)

Hi Takeshi, Since we only really have one +1 binding vote, do you want to extend this vote a bit? I've been stuck on a few things but plan to test this (setting things up now), but it probably won't happen before the deadline. On Tue, Feb 5, 2019 at 5:07 PM Takeshi Yamamuro wrote: > > Please

Re: Making travis logs less verbose

If you know how to silence messages from the setup phase (apt / pip / git), go for it. Those seem kinda hidden by Travis, but maybe there's a setting I'm not familiar with. Maven also has a -B option that makes things a little less verbose in non-interactive terminals. I think -quiet might be a

Re: Multiplicity of GitBox messages

I agree it would be best to have people watch the repos they're interested in. If no one objects, I'll open an infra ticket to remove dev@ from commons-crypto updates. (Will leave it to others to update other repos.) On Fri, Feb 8, 2019 at 1:57 PM Eric Barnhill wrote: > > Is it the consensus

[jira] [Assigned] (SPARK-26766) Remove the list of filesystems from HadoopDelegationTokenProvider.obtainDelegationTokens

[ https://issues.apache.org/jira/browse/SPARK-26766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26766: -- Assignee: Gabor Somogyi > Remove the list of filesystems f

[jira] [Resolved] (SPARK-26766) Remove the list of filesystems from HadoopDelegationTokenProvider.obtainDelegationTokens

[ https://issues.apache.org/jira/browse/SPARK-26766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26766. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23698 [https

[spark] branch master updated: [SPARK-26766][CORE] Remove the list of filesystems from HadoopDelegationTokenProvider.obtainDelegationTokens

This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new d0443a7 [SPARK-26766][CORE] Remove the list

[jira] [Resolved] (SPARK-21755) Spark 2.1.1 UI page not displaying any dynamic updates on job progress after showing progress for initial few minutes of job run.

[ https://issues.apache.org/jira/browse/SPARK-21755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21755. Resolution: Not A Bug I'm closing this for a few reasons. - There's been a lot

[jira] [Resolved] (SPARK-19528) external shuffle service registration timeout is very short with heavy workloads when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-19528. Resolution: Duplicate I believe this is the same issue as SPARK-24355. There's a new

[jira] [Resolved] (SPARK-23974) Do not allocate more containers as expected in dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-23974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23974. Resolution: Not A Problem Closing based on the above comment. > Do not allocate m

[jira] [Assigned] (SPARK-16158) Support pluggable dynamic allocation heuristics

[ https://issues.apache.org/jira/browse/SPARK-16158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-16158: -- Assignee: Marcelo Vanzin > Support pluggable dynamic allocation heurist

[jira] [Assigned] (SPARK-16158) Support pluggable dynamic allocation heuristics

[ https://issues.apache.org/jira/browse/SPARK-16158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-16158: -- Assignee: (was: Marcelo Vanzin) > Support pluggable dynamic allocat

[jira] [Assigned] (SPARK-21097) Dynamic allocation will preserve cached data

[ https://issues.apache.org/jira/browse/SPARK-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-21097: -- Assignee: (was: Marcelo Vanzin) > Dynamic allocation will preserve cached d

[jira] [Assigned] (SPARK-21097) Dynamic allocation will preserve cached data

[ https://issues.apache.org/jira/browse/SPARK-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-21097: -- Assignee: Marcelo Vanzin > Dynamic allocation will preserve cached d

[jira] [Assigned] (LIVY-551) Add "doAs" impersonation support

[ https://issues.apache.org/jira/browse/LIVY-551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned LIVY-551: --- Assignee: Kevin Risden > Add "doAs" imperson

[jira] [Resolved] (LIVY-551) Add "doAs" impersonation support

[ https://issues.apache.org/jira/browse/LIVY-551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-551. - Resolution: Fixed Issue resolved by pull request 141 [https://github.com/apache/incubator-livy

[incubator-livy] branch master updated: [LIVY-551] Add "doAs" impersonation support

This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-livy.git The following commit(s) were added to refs/heads/master by this push: new 80daade [LIVY-551] Add "

[jira] [Resolved] (SPARK-26842) java.lang.IllegalArgumentException: Unsupported class file major version 55

[ https://issues.apache.org/jira/browse/SPARK-26842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26842. Resolution: Invalid Java 11 is not supported in 2.4

[jira] [Resolved] (SPARK-26733) Clean up entrypoint.sh

[ https://issues.apache.org/jira/browse/SPARK-26733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26733. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23655 [https

[jira] [Assigned] (SPARK-26733) Clean up entrypoint.sh

[ https://issues.apache.org/jira/browse/SPARK-26733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26733: -- Assignee: Marcelo Vanzin > Clean up entrypoint

[spark] branch master updated: [SPARK-26733][K8S] Cleanup entrypoint.sh.

This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c624f5d [SPARK-26733][K8S] Cleanup

Re: Livy-0.6 release?

arcelo, what particular patches are blocking Livy 0.6 release? > > I see 3 jiras with 0.6 as Fix Version - not sure if that's correct way to > find blockers. > https://goo.gl/9axfsw > > > Thank you! > Ruslan Dautkhanov > > > On Mon, Jan 28, 2019 at 2:24 PM Marcelo Vanzi

[jira] [Resolved] (LIVY-41) Need a way to submit codes through Named Session instead of Session ID numbers

[ https://issues.apache.org/jira/browse/LIVY-41?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-41. Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 48 [https://github.com

[incubator-livy] branch master updated: [LIVY-41] Let users access sessions by session name

This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-livy.git The following commit(s) were added to refs/heads/master by this push: new def6318 [LIVY-41] Let users access

[jira] [Assigned] (SPARK-10614) SystemClock uses non-monotonic time in its wait logic

[ https://issues.apache.org/jira/browse/SPARK-10614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-10614: -- Assignee: (was: Marcelo Vanzin) > SystemClock uses non-monotonic time in

[jira] [Assigned] (SPARK-10614) SystemClock uses non-monotonic time in its wait logic

[ https://issues.apache.org/jira/browse/SPARK-10614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-10614: -- Assignee: Marcelo Vanzin > SystemClock uses non-monotonic time in its wait lo

[jira] [Issue Comment Deleted] (SPARK-24432) Add support for dynamic resource allocation

[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24432: --- Comment: was deleted (was: vanzin closed pull request #22722: [SPARK-24432][k8s] Add

[jira] [Assigned] (SPARK-24432) Add support for dynamic resource allocation

[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24432: -- Assignee: Marcelo Vanzin > Add support for dynamic resource allocat

[jira] [Assigned] (SPARK-24432) Add support for dynamic resource allocation

[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24432: -- Assignee: (was: Marcelo Vanzin) > Add support for dynamic resource allocat

[jira] [Updated] (SPARK-26726) Synchronize the amount of memory used by the broadcast variable to the UI display

[ https://issues.apache.org/jira/browse/SPARK-26726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26726: --- Fix Version/s: 2.3.3 > Synchronize the amount of memory used by the broadcast varia

[jira] [Assigned] (SPARK-26726) Synchronize the amount of memory used by the broadcast variable to the UI display

[ https://issues.apache.org/jira/browse/SPARK-26726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26726: -- Assignee: hantiantian > Synchronize the amount of memory used by the broadc

[spark] branch branch-2.3 updated: [SPARK-26726] Synchronize the amount of memory used by the broadcast variable to the UI display

This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch branch-2.3 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.3 by this push: new 94a4b46 [SPARK-26726] Synchronize

[jira] [Resolved] (SPARK-26726) Synchronize the amount of memory used by the broadcast variable to the UI display

[ https://issues.apache.org/jira/browse/SPARK-26726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26726. Resolution: Fixed Fix Version/s: 2.4.1 3.0.0 Issue resolved

<    4   5   6   7   8   9   10   11   12   13   >