Thanks Elliot ! Let me check it out !
On Mon, 17 Apr, 2023, 10:08 pm Elliot West, wrote:
> Hi Ankit,
>
> While not a part of Spark, there is a project called 'WaggleDance' that
> can federate multiple Hive metastores so that they are accessible via a
> single URI:
Hi Team
I was trying to run spark using `sbt console` on the terminal. I am
able to build the project successfully using build.sbt and the following
piece of code runs fine on IntelliJ. The only issue I am facing while
running the same on terminal is that the Executor keeps running and is
There is a DSv2-based Hive connector in Apache Kyuubi[1] that supports
connecting multiple HMS in a single Spark application.
Some limitations
- currently only supports Spark 3.3
- has a known issue when using w/ `spark-sql`, but OK w/ spark-shell and
normal jar-based Spark application.
[1]
Hi Ankit,
While not a part of Spark, there is a project called 'WaggleDance' that can
federate multiple Hive metastores so that they are accessible via a single
URI: https://github.com/ExpediaGroup/waggle-dance
This may be useful or perhaps serve as inspiration.
Thanks,
Elliot.
On Mon, 17 Apr
Greetings Everyone!
We are in need to ship spark (driver and executor) logs (not spark event
logs) from K8 to cloud bucket ADLS/S3.
Using fluentbit we are able to ship the log files but only to one single
path container/logs/.
This will cause a huge number of files in a single folder and will
++
User Mailing List
Just a reminder, anyone who can help on this.
Thanks a lot !
Ankit Prakash Gupta
On Wed, Apr 12, 2023 at 8:22 AM Ankit Gupta wrote:
> Hi All
>
> The question is regarding the support of multiple Remote Hive Metastore
> catalogs with Spark. Starting Spark 3, multiple
