Re: spark.catalog.listFunctions type signatures

Hi Jacek, Thanks for the hints, I would rather have the information statically rather than build a logical plan. I'm using Apache Calcite to build SQL expressions and then I feed them to spark to run, so the pipeline goes like this: initial query in SQL (from the user) + schema definition (from

Re: spark.catalog.listFunctions type signatures

Hi, Interesting question indeed! The closest I could get would be to use lookupFunctionBuilder(name: FunctionIdentifier): Option[FunctionBuilder] [1] followed by extracting the dataType from T in `type FunctionBuilder = Seq[Expression] => T` which can be Expression (regular functions) or

spark.catalog.listFunctions type signatures

Hi, I'm using Apache Calcite to run some SQL transformations on Apache sparks SQL statements. I would like to extract the type signature out of spark.catalog.listFunctions to be able to register them in Calcite with their proper signature. >From the API, I can get the fully qualified class name

Re: Topics for Spark online classes & webinars

https://join.slack.com/t/sparkcommunitytalk/shared_invite/zt-1rk11diac-hzGbOEdBHgjXf02IZ1mvUA Mich Talebzadeh, Lead Solutions Architect/Engineering Lead Palantir Technologies Limited view my Linkedin profile

Re: Topics for Spark online classes & webinars

Hi Bjorn, you just need to create an account on slack and join any topic I believe HTH Mich Talebzadeh, Lead Solutions Architect/Engineering Lead Palantir Technologies Limited view my Linkedin profile

Re: Topics for Spark online classes & webinars

Do I need to get an invite before joining? tir. 28. mar. 2023 kl. 18:51 skrev Mich Talebzadeh < mich.talebza...@gmail.com>: > Hi all, > > There is a section in slack called webinars > > > https://sparkcommunitytalk.slack.com/x-p4977943407059-5006939220983-5006939446887/messages/C0501NBTNQG > >

Re: Topics for Spark online classes & webinars

Hi all, There is a section in slack called webinars https://sparkcommunitytalk.slack.com/x-p4977943407059-5006939220983-5006939446887/messages/C0501NBTNQG Asma Zgolli, agreed to prepare materials for Spark internals and/or comparing spark 3 and 2. I like to contribute to "Spark Streaming &

Re: What is the range of the PageRank value of graphx

That is, every pagerank value has no relationship to 1, right? As long as we focus on the size of each pagerank value in Graphx, we don't need to focus on the range, is that right? | | 李杰 | | leedd1...@163.com | Replied Message | From | Sean Owen | | Date | 3/28/2023 22:33 | | To |

Re: What is the range of the PageRank value of graphx

>From the docs: * Note that this is not the "normalized" PageRank and as a consequence pages that have no * inlinks will have a PageRank of alpha. In particular, the pageranks may have some values * greater than 1. On Tue, Mar 28, 2023 at 9:11 AM lee wrote: > When I calculate pagerank using

What is the range of the PageRank value of graphx

When I calculate pagerank using HugeGraph, each pagerank value is less than 1, and the total of pageranks is 1. However, the PageRank value of graphx is often greater than 1, so what is the range of the PageRank value of graphx？ || 李杰 | | leedd1...@163.com |

Re: Slack for PySpark users

I created one at slack called pyspark Mich Talebzadeh, Lead Solutions Architect/Engineering Lead Palantir Technologies Limited view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your

Re: Topics for Spark online classes & webinars

Hello everyone, I suggest using the slack for the spark community created recently to collaborate and work together on these topics and use the LinkedIn page to publish the events and the webinars. Cheers, Asma Le jeu. 16 mars 2023 à 01:39, Denny Lee a écrit : > What we can do is get into the

Re: Slack for PySpark users

Hello @Mich Talebzadeh , I suggest we use this slack to plan and organize "Online classes for spark topics". Best, Asma Le mar. 28 mars 2023 à 14:37, Shani Alisar a écrit : > Hi all, > > We recently opened an unofficial spark community slack workspace > Please join so we can increase the

Re: Slack for PySpark users

Hi all, We recently opened an unofficial spark community slack workspace Please join so we can increase the community and knowledge - Link Cheers, Shani >> From: שוהם יהודה >> Date: 28 March 2023