Re: Specifying YARN Node (Label) for LLAP AM

Thank you for the information, Aaron. I explored the MR3 link you provided and found it intriguing. However, the latest email I received from another member seemed to deviate from the technical discussion's focus, potentially leading us off track and hinder objectivity. Therefore, with regret, I

Re: Specifying YARN Node (Label) for LLAP AM

Hello, For more recent benchmark results, please see [1] where we compare Trino 418, Spark 3.4.0, and Hive 3.1.3 (on MR3 1.7) using TPC-DS 10TB. Spark takes about 19600 seconds to complete all the queries, whereas Trino and Hive take about 7400 seconds only. The experiment does not use Hive-LLAP,

Re: Specifying YARN Node (Label) for LLAP AM

You might also be interested in knowing that there has been discussions about deprecating Hive on Spark: https://lists.apache.org/thread/sspltkv3ovbsjmoct72p4m1ooqk2g740 On Sat, 2023-08-19 at 10:17 +, Aaron Grubb wrote: Hi Mich, It's not a question of cannot but rather a) is it worth

Re: Specifying YARN Node (Label) for LLAP AM

Hi Mich, It's not a question of cannot but rather a) is it worth converting our pipelines from Hive to Spark and b) is Spark more performant than LLAP, and in both cases the answer seems to be no. 2016 is a lifetime ago in technological time and since then there's been a major release of Hive

Re: Specifying YARN Node (Label) for LLAP AM

interesting! In 2016 I gave a presentation in London, in Future of DataOrganised by Hortonworks July 20, 2016, Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations! Then I thought Spark as an underlying

Re: Specifying YARN Node (Label) for LLAP AM

Hi Mich, Yes, that's correct On Fri, 2023-08-18 at 15:24 +0100, Mich Talebzadeh wrote: Hi, Are you using LLAP (Long live and prosper) as a Hive engine? HTH Mich Talebzadeh, Solutions Architect/Engineering Lead London United Kingdom

Re: Specifying YARN Node (Label) for LLAP AM

Hi, Are you using LLAP (Long live and prosper) as a Hive engine? HTH Mich Talebzadeh, Solutions Architect/Engineering Lead London United Kingdom view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh

RE: Specifying YARN Node (Label) for LLAP AM

For those interested, I managed to define a way to launch the LLAP application master and daemons on separate, targeted machines. It was inspired by an article I found [1] and implemented using YARN Node Labels [2] and Placement Constraints [3] with a modification to the file

Re: Specifying YARN Node (Label) for LLAP AM

Hi Sungwoo, Thanks for your reply but I was referring exclusively to the LLAP application master which is not Tez-related. Thanks, Aaron On Wed, 2023-03-22 at 20:02 +0900, Sungwoo Park wrote: Hello, A similar issue was discussed in the Tez mailing list a long time ago:

Re: Specifying YARN Node (Label) for LLAP AM

Hello, A similar issue was discussed in the Tez mailing list a long time ago: https://lists.apache.org/thread/0vjor12lpcncg43rn6vddw8yc1k62c81 Tez still does not support specifying node labels for AMs, but as explained in the response, this is quite easy to implement if you can re-compile Tez.

Specifying YARN Node (Label) for LLAP AM

Hi all, I have a Hadoop cluster (3.3.4) with 6 nodes of equal resource size that run HDFS and YARN and 1 node with lower resources which only runs YARN that I use for Hive AMs, the LLAP AM, Spark AMs and Hive file merge containers. The HDFS nodes are set up such that the queue for LLAP on the