??????[DISCUSSION] Multi-tenant support by refactoring datamaps

[Jacky Li]

Hi,


+1 for moving the DataMapSchema json file to database folder, for supporting 
multi-tenancy.


Furthermore, I suggest we further refactor the datamap. The reason is that now 
the Sencodary Index feature have been introduced into CarbonData, and it stores 
the index metadata as the table property in the main table, and Index datamap 
actually also only associated with one main table only, so we can do the same 
for index datamap.


Propose to refactor as following:
1. For index datamap like bloom filter and lucene datamap, move their metadata 
(DataMapSchema) to the table property of the main table. Just like the way SI 
has done. 


2. Then DataMapSchema is only for Materialized View. We can rename it to 
MVSchema and clean up to keep only required fields for MV only.


3. Add separate commands for CREATE MATERIALIZED VIEW and CREATE INDEX, unify 
the Index SQL syntax for bloomfilter, lucene and SI.


4. After these refactory, for MV we can enlarge its support scope for 
non-carbon table. This could be a big benefit for user as he can accelerate 
OLAP queries on orc/parquet tables, for example.


Regards,
Jacky




------------------ ???????? ------------------
??????:&nbsp;"Indhumathi M"<indhumathi...@gmail.com&gt;;
????????:&nbsp;2020??2??13??(??????) ????3:28
??????:&nbsp;"dev"<dev@carbondata.apache.org&gt;;

????:&nbsp;[DISCUSSION] Multi-tenant support by refactoring datamaps



Hello all,

Currently, when user creates a datamap, system will store the datamap
metadata in a configurable system folder in HDFS or S3. And also, since we
use same naming conventions
as datamap name for datamapschema, users cannot create datamap with same
name which is already present in storage.

System folder currently holds the following files,
1. DataMapSchema -&gt; a json file containing schema for one datamap.
2. DataMapStatus -&gt; status for each datamap

In cloud scenarios, when one user creates SYSTEM_FOLDER and stores metadata
for materalized views and index datamap's such as bloom and lucene, other
user's are not able to access the SYSTEM_FOLDER.

In order to support multi-tenancy for datamaps, i am planning to move
system_folder under
each database level, so that users can access it. As system folder is moved
across database folder,users can create datamap with same name under
different databases.

Datamaps will be saved to database folder specified while creating datamap.

Any suggestions/inputs from the community is appreciated.

Thanks
Indhumathi