Our write and read are 50/50. We need to bulk load file system metadata into Atlas and tag department by department, then search.
Feng From: Jiaxin Ye <[email protected]> Sent: Sunday, October 18, 2020 1:47 AM To: [email protected] Subject: Re: Multiple Atlas instance in on Hadoop cluster Is your case read heavy or write heavy? I was able to change the code to make it read-only on multiple instances on the cluster. I haven’t tried to make write on multiple instances though. I did try to make the Kafka topic and consumer multi-partitioned to scale up the write with some assumptions of our Hive system. Thanks, Jiaxin On Sat, Oct 17, 2020 at 07:36 XU Feng <[email protected]<mailto:[email protected]>> wrote: Yes Dennis, we can scale HBase and Solr out, but I am sure if it’s good enough. The REST service and Janus graph still run on single active node. Thanks, Feng Xu From: Dennis Suhari <[email protected]<mailto:[email protected]>> Sent: Friday, October 16, 2020 6:08 AM To: [email protected]<mailto:[email protected]> Subject: Re: Multiple Atlas instance in on Hadoop cluster Hi Feng, but you can „scale out“ by extending HBASE cluster e.g. if you use this as storage. Br, Dennis Von meinem iPhone gesendet Am 14.10.2020 um 20:06 schrieb XU Feng <[email protected]<mailto:[email protected]>>: I am new to Apache Atlas, I know Atlas HA cannot scale out and this is a big limitation for my use cases. I was wondering if I could configure multiple Atlas HA instances in one Hadoop cluster to work around. Thank you! Feng
