Re: Spark reading from HBase using hbase-connectors - any benefit from localization?

Hi Mich, Thanks a lot for the insight, it was very helpful. Aaron On Thu, 2023-01-05 at 23:44 +, Mich Talebzadeh wrote: Hi Aaron, Thanks for the details. It is a general practice when running Spark on premise to use Hadoop clusters.

Re: Spark reading from HBase using hbase-connectors - any benefit from localization?

Hi Aaron, Thanks for the details. It is a general practice when running Spark on premise to use Hadoop clusters. This comes from the notion of data locality. Data locality in simp

Re: Spark reading from HBase using hbase-connectors - any benefit from localization?

Hi Mich, Thanks for your reply. In hindsight I realize I didn't provide enough information about the infrastructure for the question to be answered properly. We are currently running a Hadoop cluster with nodes that have the following services: - HDFS NameNode (3.3.4) - YARN NodeManager (3.3.4

Re: Spark reading from HBase using hbase-connectors - any benefit from localization?

Few questions - As I understand you already have a Hadoop cluster. Are you going to put your spark as Hadoopp nodes? - Where is your HBase cluster? Is it sharing nodes with Hadoop or has its own cluster I looked at that link and it does not say much. Essentially you want to use HBase