I see it in the following way: 1. Long-running tasks, like data-compaction, storage optimization, etc. should be in spark and Java; 2. Inspecting cli, like get schema, get information about files, get information about attributes, get schema version, get information about partitioning, last read/write, etc. should be in python or cpp. Or even both of them: we can use python for cli-ui (for example, with https://github.com/tiangolo/typer) and cpp behind it via pybind11). That allows us to simplify the codebase.
Fo me it is important to allow people to use such a cli without configuring of Java and Spark, so that is the reason of such a two- versions CLI. What do you think about it? On Wed, 2024-04-17 at 09:30 +0000, Weibin Zeng wrote: > The proposal looks good to me. Since the tool/cli should handle large > data ingestion, I suggest they can implement base on the GraphAr > spark library which can handle large data process by increase the > cluster scale. What do you think? > > On 2024/04/16 19:30:16 Sem wrote: > > I have a proposal for OSPP. I would like to have the additional > > functionality: > > 1. Effective data ingestion tool/cli that allows to add new > > edges/vertices without changing schema; > > 2. A tool/cli for optimization of underlaying data: sort all the > > edges, > > generate new offset; > > > > In this case ingestion of the new data may be organized via tool > > from > > p.1 and writing in a non optimized but fast way and optimize data > > as a > > separate job from time to time via tool from p.2. > > > > In my understanding, data ingestion is very important functionality > > but > > there is a lot of work, so it can be even splitted to subtasks. > > > > On Tue, 2024-04-16 at 16:33 +0800, weibin.zen wrote: > > > GraphAr community has successfully applied for and been accepted > > > to > > > participate in the Open Source Promotion Plan (OSPP) 2024[1]. We > > > are > > > currently in the process of gathering project proposals for OSPP > > > 2024, and we have until April 27th to finalize our submissions. > > > > > > For more detailed information regarding project, please refer to > > > the > > > following link [2]. > > > > > > For discussions, suggestions, or inquiries about the project > > > proposals kindly respond to this email thread. > > > > > > [1] > > > https://summer-ospp.ac.cn/org/orgdetail/e7aea313-3d6d-420d-a96a-f1124 > > > 1bdfa96?lang=en > > > [2] https://summer-ospp.ac.cn/help/en/#how-to-apply-for-projects > > > > > > Best > > > weibin.zen > > > > > > > > > ------------------------------------------------------------------- > > -- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
