Hi Rima, I am working on the datasketches extension to support parallel queries (distributed aggregation). I expect to get this done in a matter of days. Also we have just made some improvements to HLL merge speed in the core library. These changes were not released yet, but available in the master branch. We have another HLL performance improvement in mind. I will work on it once I finish the parallel query support.
On Fri, Apr 14, 2023 at 3:33 AM Bhowmick, Rima <[email protected]> wrote: > Hello Team, > > > > Here is the snapshot of the existing application: > > > > TechStack: Postgres DB, Hive, Tableau UI > > Postgres Plugin: DataSketches > > > > Flow in brief: > > - Hadoop Data pipeline job pushes pre-aggregated(using hive > datasketches algo) active card data, along with other details to Hive. > - Another job populates that data to Postgres DB, finally having 3 > years data of 4 regions for multiple countries. > - Tableau dashboard having live connection to Postgres DB. > - Tableau Query calling Postgres DB, to aggregate the > binary/pre-aggregated data to get distinct card count (using DataSketches > algorithm) and fetch data based on multiple filter conditions. > - Usually data would be of 3yrs for the span of 2 months, means total > 6 months of data to aggregate for a country on multiple conditions. > > > > Usually this aggregation query response is quite slow. We have tried lot > of different ways to resolve this, > > > > Mainly datasketches part is making most of the time in execution. > > > > Thanks & Regards, > > Rima Bhowmick > > Marketing Brand Analytics > > [image: Logo Description automatically generated] >
