[python-uk] Vacancy Notice

2020-10-27 Thread Steve Holden
My employer's Platform group has vacancies, including one for a full-stack Python developer. The position is fully remote (I was hired without meeting anyone in the company face to face). The only downside is you might end up working in my team. See

[Boatanchors] Fall 2020 Classic Exchange CW

2020-10-24 Thread howard holden
The Fall CW Classic Exchange will take place Sunday Nov 1 1400Z to Nov 2 0800Z and Tuesday Nov 3 1400Z to Nov 4 0800Z. The CX is a no-pressure contest celebrating the older commercial and Home Brew equipment that was the pride and joy of ham shacks many decades ago. The object is to encourage

[spark] branch branch-3.0 updated: [SPARK-30821][K8S] Handle executor failure with multiple containers

2020-10-24 Thread holden
This is an automated email from the ASF dual-hosted git repository. holden pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new f7c7f4f8 [SPARK-30821][K8S] Handle

[jira] [Resolved] (SPARK-30821) Executor pods with multiple containers will not be rescheduled unless all containers fail

2020-10-24 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-30821. -- Fix Version/s: 3.1.0 3.0.2 Resolution: Fixed > Executor p

[spark] branch master updated: [SPARK-30821][K8S] Handle executor failure with multiple containers

2020-10-24 Thread holden
This is an automated email from the ASF dual-hosted git repository. holden pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new f659527 [SPARK-30821][K8S] Handle executor

[jira] [Created] (SPARK-33231) Make podCreationTimeout configurable

2020-10-23 Thread Holden Karau (Jira)
Holden Karau created SPARK-33231: Summary: Make podCreationTimeout configurable Key: SPARK-33231 URL: https://issues.apache.org/jira/browse/SPARK-33231 Project: Spark Issue Type: Improvement

[Python-Dev] Re: Speeding up CPython

2020-10-21 Thread Steve Holden
On Wed, Oct 21, 2020 at 8:37 AM Paul Moore wrote: > On Wed, 21 Oct 2020 at 08:14, Christian Heimes > wrote: > > > > On 21/10/2020 00.14, Steven D'Aprano wrote: > > > On Tue, Oct 20, 2020 at 06:04:37PM +0100, Paul Moore wrote: > > > > > >> What I don't see is where the money's coming from. It's

[jira] [Created] (SPARK-33200) Validate that all shuffle files are migrated during E2E testing

2020-10-20 Thread Holden Karau (Jira)
Holden Karau created SPARK-33200: Summary: Validate that all shuffle files are migrated during E2E testing Key: SPARK-33200 URL: https://issues.apache.org/jira/browse/SPARK-33200 Project: Spark

Re: Scala vs Python for ETL with Spark

2020-10-17 Thread Holden Karau
Scala and Python have their advantages and disadvantages with Spark. In my experience with performance is super important you’ll end up needing to do some of your work in the JVM, but in many situations what matters work is what your team and company are familiar with and the ecosystem of tooling

[Python-Dev] Re: Remove module's __version__ attributes in the stdlib

2020-10-16 Thread Steve Holden
Since some code clearly accesses __version__, would it make sense to equip all stdlib modules with a __version__ property that returned the Python version, suitably prefixed? Kind regards, Steve On Fri, Oct 16, 2020 at 5:28 AM Karthikeyan wrote: > On Fri, Oct 16, 2020, 12:45 AM Serhiy

[jira] [Created] (SPARK-33154) Handle shuffle blocks being removed during decommissioning

2020-10-14 Thread Holden Karau (Jira)
Holden Karau created SPARK-33154: Summary: Handle shuffle blocks being removed during decommissioning Key: SPARK-33154 URL: https://issues.apache.org/jira/browse/SPARK-33154 Project: Spark

[jira] [Resolved] (SPARK-33151) Jenkins PRB is not responding

2020-10-14 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-33151. -- Resolution: Fixed > Jenkins PRB is not respond

[jira] [Created] (SPARK-33151) Jenkins PRB is not responding

2020-10-14 Thread Holden Karau (Jira)
Holden Karau created SPARK-33151: Summary: Jenkins PRB is not responding Key: SPARK-33151 URL: https://issues.apache.org/jira/browse/SPARK-33151 Project: Spark Issue Type: Bug

[Python-Dev] Re: [python-committers] Resignation from Stefan Krah

2020-10-13 Thread Steve Holden
Full marks to the SC for transparency. That's a healthy sign that the community acknowledges its disciplinary processes must also be open to scrutiny, and rather better than dealing with matters in a Star Council. Kind regards, Steve On Fri, Oct 9, 2020 at 12:10 AM Thomas Wouters wrote: > >

[Boatanchors] Fall 2020 Classic Exchange Phone

2020-10-12 Thread howard holden
The AM-SSB-FM CX will run from Sunday Oct 18 1400Z to Oct 19 0800Z and Tuesday Oct 20 1400Z to Oct 21 0800Z The CX is a no-pressure contest celebrating the older commercial and Home Brew equipment that was the pride and joy of ham shacks many decades ago. The object is to encourage

[jira] [Created] (SPARK-33097) Add Decommissioning messages to the Spark listener bus

2020-10-08 Thread Holden Karau (Jira)
Holden Karau created SPARK-33097: Summary: Add Decommissioning messages to the Spark listener bus Key: SPARK-33097 URL: https://issues.apache.org/jira/browse/SPARK-33097 Project: Spark Issue

[tesseract-ocr] Indigenous Language OCR w/ Tesseract 4.0 in Mac OSX - uncharset_extractor "command not found"

2020-10-08 Thread Josh Holden
Any advice anyone has on this would be much appreciated! I am doing this for a paid contract, not my own research, so I really would like to get past this roadblock. Thanks so much. Sincerely, Dr. Josh Holden Postdoctoral researcher ALT Lab, University of Alberta -- You received t

Re: Official support of CREATE EXTERNAL TABLE

2020-10-07 Thread Holden Karau
onable use cases. I think it's >>> always safer to keep the behavior the same as before. If we want to change >>> the behavior and follow option 2, we need use cases to justify it. >>> >>> For now, the only use case I see is for Hive compatibility and allow >

Re: Official support of CREATE EXTERNAL TABLE

2020-10-06 Thread Holden Karau
As someone who's had the job of porting different SQL dialects to Spark, I'm also very much in favor of keeping EXTERNAL, and I think Ryan's suggestion of leaving it up to the catalogs on how to handle this makes sense. On Tue, Oct 6, 2020 at 1:54 PM Ryan Blue wrote: > I would summarize both

New ASF guidelines for docker images

2020-10-04 Thread Holden Karau
Hi Folks, It seems like there are new guidelines for incubator projects that might make it easier for us to publish Spark docker images. While we are not an incubator project, I think this is probably a sign we should revisit publishing docker images. Cheers, Holden -- Forwarded

[jira] [Created] (SPARK-33049) Decommission Core Integration Test is flaky.

2020-10-01 Thread Holden Karau (Jira)
Holden Karau created SPARK-33049: Summary: Decommission Core Integration Test is flaky. Key: SPARK-33049 URL: https://issues.apache.org/jira/browse/SPARK-33049 Project: Spark Issue Type: Bug

RE: [WIRELESS-LAN] Status of Wi-Fi 6 Client Drivers?

2020-10-01 Thread Michael Holden
The new encryptions can also cause issues with some of the supplicants. We’ve seen some clients requesting user/pass when connecting to an Open network running the Open Wireless Enhanced (OWE). Google Pixel 3’s used to Seg fault and rebooting the phone when connecting to WPA3-Personal along

Re: [VOTE][Format] Allow for 256-bit Decimal's in the Arrow specification

2020-09-29 Thread Holden Karau
+1 (non-binding) On Tue, Sep 29, 2020 at 6:08 PM Sutou Kouhei wrote: > +1 > > In > "Re: [VOTE][Format] Allow for 256-bit Decimal's in the Arrow > specification" on Tue, 29 Sep 2020 13:38:04 -0700, > Jacques Nadeau wrote: > > > +1 > > > > On Tue, Sep 29, 2020 at 11:19 AM Wes McKinney >

[jira] [Assigned] (SPARK-32381) Expose the ability for users to use parallel file & avoid location information discovery in RDDs

2020-09-24 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau reassigned SPARK-32381: Assignee: Chao Sun > Expose the ability for users to use parallel file & avoid l

[jira] [Resolved] (SPARK-32381) Expose the ability for users to use parallel file & avoid location information discovery in RDDs

2020-09-24 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-32381. -- Fix Version/s: 3.1.0 Resolution: Fixed > Expose the ability for users to use paral

[spark] branch master updated: [SPARK-32381][CORE][SQL] Move and refactor parallel listing & non-location sensitive listing to core

2020-09-24 Thread holden
This is an automated email from the ASF dual-hosted git repository. holden pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 8ccfbc1 [SPARK-32381][CORE][SQL] Move

Re: Running K8s integration tests for changes in core?

2020-09-24 Thread Holden Karau
eye on the queue/build results > today. > > shane > > On Thu, Aug 20, 2020 at 2:28 PM Holden Karau wrote: > >> Sounds good, thanks for the heads up. I hope you get some time to relax :) >> >> On Thu, Aug 20, 2020 at 2:26 PM shane knapp ☠ >> wrote: >>

[jira] [Commented] (SPARK-32980) Launcher Client tests flake with minikube

2020-09-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201073#comment-17201073 ] Holden Karau commented on SPARK-32980: -- Our method of getting the service assumes the service

[jira] [Created] (SPARK-32980) Launcher Client tests flake with minikube

2020-09-23 Thread Holden Karau (Jira)
Holden Karau created SPARK-32980: Summary: Launcher Client tests flake with minikube Key: SPARK-32980 URL: https://issues.apache.org/jira/browse/SPARK-32980 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-32979) Spark K8s decom test is broken

2020-09-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-32979. -- Resolution: Duplicate Duplicate of SPARK-32937 > Spark K8s decom test is bro

[jira] [Created] (SPARK-32979) Spark K8s decom test is broken

2020-09-23 Thread Holden Karau (Jira)
Holden Karau created SPARK-32979: Summary: Spark K8s decom test is broken Key: SPARK-32979 URL: https://issues.apache.org/jira/browse/SPARK-32979 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-32937) DecomissionSuite in k8s integration tests is failing.

2020-09-19 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198867#comment-17198867 ] Holden Karau commented on SPARK-32937: -- So there are additional proposed changes to this chunk

[jira] [Commented] (SPARK-32937) DecomissionSuite in k8s integration tests is failing.

2020-09-19 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198863#comment-17198863 ] Holden Karau commented on SPARK-32937: -- I want to be clear, I'm asking for a revert then fix. I do

[jira] [Commented] (SPARK-32937) DecomissionSuite in k8s integration tests is failing.

2020-09-19 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198860#comment-17198860 ] Holden Karau commented on SPARK-32937: -- It looks like someone changed the logging message again

[Python-Dev] Re: Hygenic macros PEP.

2020-09-16 Thread Steve Holden
On Wed, Sep 16, 2020 at 3:53 PM Mark Shannon wrote: > [...] > > maybe!(a.b) > > which would translate to: > > $tmp = a; None if $tmp is None else ($tmp.b) > > This reminds me very much of PL/1's "compile-time procedures," which forty years ago were pretty much like PL/1

Re: How to clear spark Shuffle files

2020-09-14 Thread Holden Karau
There's a second new mechanism which uses TTL for cleanup of shuffle files. Can you share more about your use case? On Mon, Sep 14, 2020 at 1:33 PM Edward Mitchell wrote: > We've also had some similar disk fill issues. > > For Java/Scala RDDs, shuffle file cleanup is done as part of the JVM >

[jira] [Commented] (SPARK-32881) NoSuchElementException occurs during decommissioning

2020-09-14 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195700#comment-17195700 ] Holden Karau commented on SPARK-32881: -- Thanks for the catch, I'll take a look at this issue

[jira] [Created] (SPARK-32866) Docker buildx now requires --push

2020-09-12 Thread Holden Karau (Jira)
Holden Karau created SPARK-32866: Summary: Docker buildx now requires --push Key: SPARK-32866 URL: https://issues.apache.org/jira/browse/SPARK-32866 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-32745) Add an option to reject block migrations when under disk pressure

2020-08-30 Thread Holden Karau (Jira)
Holden Karau created SPARK-32745: Summary: Add an option to reject block migrations when under disk pressure Key: SPARK-32745 URL: https://issues.apache.org/jira/browse/SPARK-32745 Project: Spark

[jira] [Commented] (YUNIKORN-380) Deploying YUNIKORN 0.9 RC2 on Minikube 1.3.1 results in port conflict

2020-08-30 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/YUNIKORN-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17187396#comment-17187396 ] Holden Karau commented on YUNIKORN-380: --- That makes sense. I'll see about raising a PR

Re: [Maria-discuss] Missing table rows when importing a MySQL 5.7.8 database into MariaDB 10.4.8 database

2020-08-26 Thread Elliot Holden
out the first that won't make it to maria server? If you'll find such row, the next step I suggest to report a bug on Jira providing the table definition and the INSERT(s) which would be pretty easy to reproduce and hopefully to fix too. Thanks as well! Andrei -- Elliot Holden ‪(404)

[jira] [Resolved] (SPARK-32643) [Cleanup] Consolidate state kept in ExecutorDecommissionInfo with TaskSetManager.tidToExecutorKillTimeMapping

2020-08-26 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-32643. -- Fix Version/s: 3.1.0 Resolution: Fixed > [Cleanup] Consolidate state k

[jira] [Assigned] (SPARK-32643) [Cleanup] Consolidate state kept in ExecutorDecommissionInfo with TaskSetManager.tidToExecutorKillTimeMapping

2020-08-26 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau reassigned SPARK-32643: Assignee: Devesh Agrawal > [Cleanup] Consolidate state kept in ExecutorDecommissionI

[spark] branch master updated: [SPARK-32643][CORE][K8S] Consolidate state decommissioning in the TaskSchedulerImpl realm

2020-08-26 Thread holden
This is an automated email from the ASF dual-hosted git repository. holden pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new b786f31 [SPARK-32643][CORE][K8S] Consolidate

[spark-website] branch asf-site updated: Update committer guide

2020-08-24 Thread holden
This is an automated email from the ASF dual-hosted git repository. holden pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new a3f618b Update committer guide

[jira] [Created] (YUNIKORN-380) Deploying YUNIKORN 0.9 RC2 on Minikube 1.3.1 results in port conflict

2020-08-23 Thread Holden Karau (Jira)
Holden Karau created YUNIKORN-380: - Summary: Deploying YUNIKORN 0.9 RC2 on Minikube 1.3.1 results in port conflict Key: YUNIKORN-380 URL: https://issues.apache.org/jira/browse/YUNIKORN-380 Project

[jira] [Created] (YUNIKORN-380) Deploying YUNIKORN 0.9 RC2 on Minikube 1.3.1 results in port conflict

2020-08-23 Thread Holden Karau (Jira)
Holden Karau created YUNIKORN-380: - Summary: Deploying YUNIKORN 0.9 RC2 on Minikube 1.3.1 results in port conflict Key: YUNIKORN-380 URL: https://issues.apache.org/jira/browse/YUNIKORN-380 Project

[jira] [Commented] (YUNIKORN-379) Default log level is too verbose

2020-08-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/YUNIKORN-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17182865#comment-17182865 ] Holden Karau commented on YUNIKORN-379: --- Also related to this issue is the need to add

[jira] [Created] (YUNIKORN-379) Default log level is too verbose

2020-08-23 Thread Holden Karau (Jira)
Holden Karau created YUNIKORN-379: - Summary: Default log level is too verbose Key: YUNIKORN-379 URL: https://issues.apache.org/jira/browse/YUNIKORN-379 Project: Apache YuniKorn Issue Type

Re: [PSF-Community] Invitation to PSF Trademarks Working Group

2020-08-22 Thread Steve Holden
As someone who's worked on this committee, I would say that the questions that come along are often interesting, and the work can give you valuable insight into a less-technical part of Python's governance. It relies on the common sense of its members to consistently apply policy in a way that

Re: Running K8s integration tests for changes in core?

2020-08-20 Thread Holden Karau
kers w/minikube and docker, >>> but it shouldn't be too bad as the full integration test takes ~45m, vs 4+ >>> hrs for the regular PRB. >>> >>> i can enable this in about 1m of time if the consensus is for us to want >>> this. >>> >>> On We

Re: [Maria-discuss] Missing table rows when importing a MySQL 5.7.8 database into MariaDB 10.4.8 database

2020-08-20 Thread Elliot Holden
l_server database_name < wp_backup.sql So the above works I am guessing it's 29 in 'all' of 'all rows are imported'. Does wp_backup.sql actually have *all* 600? -- Elliot Holden ‪(404)884-8593‬ ell...@elliotmywebguy.com https

Re: [Maria-discuss] Missing table rows when importing a MySQL 5.7.8 database into MariaDB 10.4.8 database

2020-08-20 Thread Elliot Holden
orting into a test database out there: mysql -u username -pPassword -h remote_mysql_server database_name < wp_backup.sql So the above works I am guessing it's 29 in 'all' of 'all rows are imported'. Does wp_backup.sql actually have *all* 600? -- Elliot Holden ‪(404)

Re: [Maria-discuss] Missing table rows when importing a MySQL 5.7.8 database into MariaDB 10.4.8 database

2020-08-19 Thread Elliot Holden
re. Then check the result of "SHOW WARNINGS" and "|SHOW INNODB STATUS"| | | |Also the "--opt" should be the default in mysqldump but you could verify that your sql dump file does have charset setup and everything | Regards, On 19/08/2020 06:44, Elliot Holden wrote:

Re: [Maria-discuss] Missing table rows when importing a MySQL 5.7.8 database into MariaDB 10.4.8 database

2020-08-19 Thread Elliot Holden
with the Latin1 character set. Try creating your MariaDB database with UTF8 as it's character set. Cheers Peter On Wed, 19 Aug 2020 at 14:44, Elliot Holden <mailto:ell...@elliotmywebguy.com>> wrote: Sending this again because I got no response the first time and di

Re: Running K8s integration tests for changes in core?

2020-08-19 Thread Holden Karau
Sounds good. In the meantime would folks committing things in core run the K8s PRB or run it locally? A second change this morning was committed that broke the K8s PR tests. On Tue, Aug 18, 2020 at 9:53 PM Prashant Sharma wrote: > +1, we should enable. > > On Wed, Aug 19, 2020 at 9:18

Re: [Maria-discuss] Missing table rows when importing a MySQL 5.7.8 database into MariaDB 10.4.8 database

2020-08-18 Thread Elliot Holden
. I will keep searching as well. Thanks E. On 8/11/20 11:25 PM, Elliot Holden wrote: Hello All, New to the list, so maybe an expert has the easy answer to this one... When I use mysqldump to backup a MySQL 5.7.8 database then import that database into a MariaDB 10.4.8 database, at least one

Running K8s integration tests for changes in core?

2020-08-18 Thread Holden Karau
core changes as well as K8s changes? Cheers, Holden :) -- Twitter: https://twitter.com/holdenkarau Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau

[jira] [Created] (SPARK-32657) Decommissioning Integration Test checks for old string

2020-08-18 Thread Holden Karau (Jira)
Holden Karau created SPARK-32657: Summary: Decommissioning Integration Test checks for old string Key: SPARK-32657 URL: https://issues.apache.org/jira/browse/SPARK-32657 Project: Spark Issue

[jira] [Created] (SPARK-32642) Add support for ESS in Spark sidecar

2020-08-17 Thread Holden Karau (Jira)
Holden Karau created SPARK-32642: Summary: Add support for ESS in Spark sidecar Key: SPARK-32642 URL: https://issues.apache.org/jira/browse/SPARK-32642 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-32642) Add support for ESS in Spark sidecar

2020-08-17 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17179257#comment-17179257 ] Holden Karau commented on SPARK-32642: -- I'm exploring this. > Add support for ESS in Spark side

[jira] [Resolved] (SPARK-31198) Use graceful decommissioning as part of dynamic scaling

2020-08-13 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-31198. -- Fix Version/s: 3.1.0 Resolution: Fixed > Use graceful decommissioning as p

[spark] branch master updated: [SPARK-31198][CORE] Use graceful decommissioning as part of dynamic scaling

2020-08-12 Thread holden
This is an automated email from the ASF dual-hosted git repository. holden pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 548ac7c [SPARK-31198][CORE] Use graceful

[Maria-discuss] Missing table rows when importing a MySQL 5.7.8 database into MariaDB 10.4.8 database

2020-08-11 Thread Elliot Holden
eign keys and encryption for tables Transactions: YES   XA: YES   Savepoints: YES -- Elliot Holden ‪(404)884-8593‬ ell...@elliotmywebguy.com https://www.ElliotMyWebGuy.com ___ Mailing list: https://launchpad.net/~maria-discuss Post to

[jira] [Commented] (SPARK-32530) SPIP: Kotlin support for Apache Spark

2020-08-11 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175547#comment-17175547 ] Holden Karau commented on SPARK-32530: -- I can understand the desire to have this be inside the ASF

Re: [VOTE] Release Spark 2.4.7 (RC1)

2020-08-08 Thread Holden Karau
I'm going to go ahead and vote -0 then based on that then. On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun wrote: > Hi, All. > > Unfortunately, there is an on-going discussion about the new decimal > correctness. > > Although we fixed one correctness issue at master and backported it > partially

Re: spark-on-k8s is still experimental?

2020-08-05 Thread Holden Karau
e/spark/pull/29368. > > Anyway, many thanks! > > > On Tue, Aug 4, 2020 at 12:26 AM Holden Karau wrote: > >> There was discussion around removing the statement and declaring it GA >> but I believe it was decided to leave it in until an external shuffle >> service is

[jira] [Resolved] (SPARK-31197) Exit the executor once all tasks & migrations are finished

2020-08-05 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-31197. -- Fix Version/s: 3.1.0 Target Version/s: 3.1.0 Resolution: Fixed > E

[jira] [Assigned] (SPARK-31197) Exit the executor once all tasks & migrations are finished

2020-08-05 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau reassigned SPARK-31197: Assignee: Holden Karau > Exit the executor once all tasks & migrations are f

[spark] branch master updated: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning

2020-08-05 Thread holden
This is an automated email from the ASF dual-hosted git repository. holden pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 375d348 [SPARK-31197][CORE] Shutdown executor

Re: Someone should get started on migrating your build jobs to ci-builds.apache.org

2020-08-05 Thread Holden Karau
The Spark project is using github actions successfully. We’re currently trying to get more workers added (apparently all of the ASF repos get one pool together). But honestly it’s been faster for us even with the current number of workers so I think that should be fine for mahout. On Wed, Aug 5,

Re: [VOTE] Update the committer guidelines to clarify when to commit changes.

2020-08-04 Thread Holden Karau
10:18 PM Xiao Li wrote: > >> +1 >> >> Xiao >> >> On Fri, Jul 31, 2020 at 9:32 AM Mridul Muralidharan >> wrote: >> >>> >>> +1 >>> >>> Thanks, >>> Mridul >>> >>> On Thu, Jul 30, 2020 at 4:49 PM

Re: Removing references to Master

2020-08-04 Thread Holden Karau
I think this is a good idea, and yes keeping it backwards compatible initially is important since we missed the boat on Spark 3. I like the Controller/Leader one since I think that does a good job of reflecting the codes role. On Tue, Aug 4, 2020 at 7:01 AM Tom Graves wrote: > Hey everyone, > >

Re: spark-on-k8s is still experimental?

2020-08-03 Thread Holden Karau
There was discussion around removing the statement and declaring it GA but I believe it was decided to leave it in until an external shuffle service is supported on K8s. On Mon, Aug 3, 2020 at 2:45 AM JackyLee wrote: > +1. It has been worked well in our company and we has used it to support >

Re: [VOTE] Update the committer guidelines to clarify when to commit changes.

2020-07-30 Thread Holden Karau
+1 from myself :) On Thu, Jul 30, 2020 at 2:53 PM Jungtaek Lim wrote: > +1 (non-binding, I guess) > > Thanks for raising the issue and sorting it out! > > On Fri, Jul 31, 2020 at 6:47 AM Holden Karau wrote: > >> Hi Spark Developers, >> >> After the discu

[VOTE] Update the committer guidelines to clarify when to commit changes.

2020-07-30 Thread Holden Karau
collaboration in building Apache Spark. I believe we share the goal of creating a welcoming community around the project. On a personal note, it is my belief that consistently applying this policy around commits can help to make a more accessible and welcoming community. Kind Regards, Holden

[jira] [Resolved] (SPARK-32417) Flaky test: BlockManagerDecommissionIntegrationSuite.verify that an already running task which is going to cache data succeeds on a decommissioned executor

2020-07-30 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-32417. -- Fix Version/s: 3.1.0 Assignee: Devesh Agrawal Resolution: Fixed > Flaky t

[spark] branch master updated: [SPARK-32417] Fix flakyness of BlockManagerDecommissionIntegrationSuite

2020-07-30 Thread holden
This is an automated email from the ASF dual-hosted git repository. holden pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 6032c5b [SPARK-32417] Fix flakyness

[jira] [Resolved] (SPARK-32199) Clear shuffle state when decommissioned nodes/executors are finally lost

2020-07-30 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-32199. -- Fix Version/s: 3.1.0 Assignee: Devesh Agrawal Resolution: Fixed > Cl

[spark] branch master updated: [SPARK-32199][SPARK-32198] Reduce job failures during decommissioning

2020-07-30 Thread holden
This is an automated email from the ASF dual-hosted git repository. holden pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 366a178 [SPARK-32199][SPARK-32198] Reduce job

[jira] [Assigned] (SPARK-32198) Don't fail running jobs when decommissioned executors finally go away

2020-07-30 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau reassigned SPARK-32198: Assignee: Devesh Agrawal > Don't fail running jobs when decommissioned executors fina

[jira] [Resolved] (SPARK-32198) Don't fail running jobs when decommissioned executors finally go away

2020-07-30 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-32198. -- Fix Version/s: 3.1.0 Target Version/s: 3.1.0 Resolution: Fixed > Don't f

Re: [DISCUSS] Amend the commiter guidelines on the subject of -1s & how we expect PR discussion to be treated.

2020-07-25 Thread Holden Karau
It sounds like with the slight wording change we’re in agreement so I’ll bounce this by an editor friend to fix my grammar/spelling before I put it up for a vote. On Sat, Jul 25, 2020 at 9:23 PM Hyukjin Kwon wrote: > +1 thanks Holden. > > On Fri, 24 Jul 2020, 22:34 Tom Graves, > wr

Re: Starting work on last Scala 2.13 updates

2020-07-24 Thread Holden Karau
This is awesome progress :) On Fri, Jul 24, 2020 at 8:56 AM Sean Owen wrote: > Status update - we should have Scala 2.13 compiling, with the > exception of the REPL. > Looks like 99% or so of tests pass too, but the remaining ones might > be hard to debug. I haven't looked hard yet. >

[jira] [Commented] (SPARK-32264) More resources in Github Actions

2020-07-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164066#comment-17164066 ] Holden Karau commented on SPARK-32264: -- It's being routed inside of GitHub as of my last contact

Re: Exposing Spark parallelized directory listing & non-locality listing in core

2020-07-23 Thread Holden Karau
Awesome that sounds great :) On Thu, Jul 23, 2020 at 3:43 AM Steve Loughran wrote: > > > On Wed, 22 Jul 2020 at 18:50, Holden Karau wrote: > >> Wonderful. To be clear the patch is more to start the discussion about >> how we want to do it and less what

[spark] branch master updated: [SPARK-32217] Plumb whether a worker would also be decommissioned along with executor

2020-07-22 Thread holden
This is an automated email from the ASF dual-hosted git repository. holden pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new f8d29d3 [SPARK-32217] Plumb whether a worker

[jira] [Resolved] (SPARK-32217) Track whether the worker is also being decommissioned along with an executor

2020-07-22 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-32217. -- Fix Version/s: 3.1.0 Assignee: Devesh Agrawal Resolution: Fixed > Tr

[Python-Dev] Re: How to customize CPython to a minimal set

2020-07-22 Thread Steve Holden
Sadly micropython is not intended to support numerical libraries and other such complex modules: the support for the Python standard library is pretty much non-existent. Kind regards, Steve On Wed, Jul 22, 2020 at 3:10 PM Huang, Yang wrote: > Thank you for all your comments. > > I cannot

[jira] [Created] (SPARK-32397) Snapshot artifacts can have differing timestamps, making it hard to consume

2020-07-22 Thread Holden Karau (Jira)
Holden Karau created SPARK-32397: Summary: Snapshot artifacts can have differing timestamps, making it hard to consume Key: SPARK-32397 URL: https://issues.apache.org/jira/browse/SPARK-32397 Project

Re: [DISCUSS] Amend the commiter guidelines on the subject of -1s & how we expect PR discussion to be treated.

2020-07-22 Thread Holden Karau
On Wed, Jul 22, 2020 at 7:39 AM Imran Rashid < iras...@apache.org > wrote: > Hi Holden, > > thanks for leading this discussion, I'm in favor in general. I have one > specific question -- these two sections seem to contradict each other > slightly: > > > If ther

Re: Exposing Spark parallelized directory listing & non-locality listing in core

2020-07-22 Thread Holden Karau
Wonderful. To be clear the patch is more to start the discussion about how we want to do it and less what I think is the right way. On Wed, Jul 22, 2020 at 10:47 AM Steve Loughran wrote: > > > On Wed, 22 Jul 2020 at 00:51, Holden Karau wrote: > >> Hi Folks,

Exposing Spark parallelized directory listing & non-locality listing in core

2020-07-21 Thread Holden Karau
potential different paths we could do for implementation and wanted to see if anyone had thoughts - https://github.com/apache/spark/pull/29179. Cheers, Holden -- Twitter: https://twitter.com/holdenkarau Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn

[jira] [Resolved] (SPARK-24266) Spark client terminates while driver is still running

2020-07-21 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-24266. -- Fix Version/s: 3.1.0 Target Version/s: 3.1.0 (was: 2.4.7, 3.1.0) Resolution

[spark] branch master updated: [SPARK-24266][K8S] Restart the watcher when we receive a version changed from k8s

2020-07-21 Thread holden
This is an automated email from the ASF dual-hosted git repository. holden pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 0432379 [SPARK-24266][K8S] Restart the watcher

[DISCUSS] Amend the commiter guidelines on the subject of -1s & how we expect PR discussion to be treated.

2020-07-21 Thread Holden Karau
out on the dev@ list to start the discussion. Kind Regards, Holden -- Twitter: https://twitter.com/holdenkarau Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau

[jira] [Created] (SPARK-32381) Expose the ability for users to use parallel file & avoid location information discovery in RDDs

2020-07-21 Thread Holden Karau (Jira)
Holden Karau created SPARK-32381: Summary: Expose the ability for users to use parallel file & avoid location information discovery in RDDs Key: SPARK-32381 URL: https://issues.apache.org/jira/browse/SPARK-3

[jira] [Commented] (SPARK-26345) Parquet support Column indexes

2020-07-21 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162313#comment-17162313 ] Holden Karau commented on SPARK-26345: -- We don't assign issues normally until after the merge

Re: [PySpark] Revisiting PySpark type annotations

2020-07-21 Thread Holden Karau
Yeah I think this could be a great project now that we're only Python 3.5+. One potential is making this an Outreachy project to get more folks from different backgrounds involved in Spark. On Tue, Jul 21, 2020 at 12:33 PM Driesprong, Fokko wrote: > Since we've recently dropped support for

Re: [OSS DIGEST] The major changes of Apache Spark from June 3 to June 16

2020-07-21 Thread Holden Karau
Got it, I missed the date in the reading :) On Tue, Jul 21, 2020 at 11:23 AM Xingbo Jiang wrote: > Hi Holden, > > This is the digest for commits merged between *June 3 and June 16.* The > commits you mentioned would be included in the future digests. > > Cheers, > > X

<    3   4   5   6   7   8   9   10   11   12   >