Re: Javascript Based UDFs

I don't know how these frameworks work, but I'd hope that it takes JavaScript and gives you some invokeable object. It's just Java executing like anything else, nothing special, no special consideration for being in a task? Why would it need another JVM? On Mon, Jun 27, 2022, 11:18 PM Matt Hawes

Re: Javascript Based UDFs

Thanks for the reply! I had originally thought that this would incur a cost of spinning up a VM every time the UDF is called but thinking about it again you might be right. I guess if I make the VM accessible via a transient property on the UDF class then it would only be initialized once per

Re: Javascript Based UDFs

Rather than reimplement a new UDF, why not indeed just use an embedded interpreter? if something can turn javascript into something executable you can wrap that in a normal Java/Scala UDF and go. On Mon, Jun 27, 2022 at 10:42 PM Matt Hawes wrote: > Hi all, I'm thinking about trying to implement

Javascript Based UDFs

Hi all, I'm thinking about trying to implement the ability to write spark UDFs using javascript. For the use case I have in mind, a lot of the code is already written in javascript and so it would be very convenient to be able to call this directly from spark. I wanted to post here first before

Docker images for Spark 3.3.0 release are now available

Hi all, The official Docker images for Spark 3.3.0 release are now available! - To run Spark with Scala/Java API only: https://hub.docker.com/r/apache/spark - To run Python on Spark: https://hub.docker.com/r/apache/spark-py - To run R on Spark: https://hub.docker.com/r/apache/spark-r

Observed consistent test failure in master (ParquetIOSuite)

Hi, I just observed the test failure in ParquetIOSuite which I can consistently reproduce with IntelliJ. Haven't had a chance to run a test with maven/sbt. I filed SPARK-39622 for this failure. It'd be awesome if someone having context looks

[FINAL CALL] - Travel Assistance to ApacheCon New Orleans 2022

To all committers and non-committers. This is a final call to apply for travel/hotel assistance to get to and stay in New Orleans for ApacheCon 2022. Applications have been extended by one week and so the application deadline is now the 8th July 2022. The rest of this email is a copy of what