luoyuxia commented on code in PR #157: URL: https://github.com/apache/paimon-rust/pull/157#discussion_r3005799838
########## docs/src/getting-started.md: ########## @@ -0,0 +1,178 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Getting Started + +## Installation + +Add `paimon` to your `Cargo.toml`: + +```toml +[dependencies] +paimon = "0.0.0" +tokio = { version = "1", features = ["full"] } +``` + +By default, the `storage-fs` (local filesystem) and `storage-memory` (in-memory) backends are enabled. To use additional storage backends, enable the corresponding feature flags: + +```toml +[dependencies] +paimon = { version = "0.0.0", features = ["storage-s3"] } +``` + +Available storage features: + +| Feature | Backend | +|------------------|------------------| +| `storage-fs` | Local filesystem | +| `storage-memory` | In-memory | +| `storage-s3` | Amazon S3 | +| `storage-oss` | Alibaba Cloud OSS| +| `storage-all` | All of the above | + +## Catalog Management + +`FileSystemCatalog` manages databases and tables stored on a local (or remote) filesystem. + +### Create a Catalog + +```rust +use paimon::FileSystemCatalog; + +let catalog = FileSystemCatalog::new("/tmp/paimon-warehouse")?; +``` + +### Manage Databases + +```rust +use paimon::Catalog; // import the trait +use std::collections::HashMap; + +// Create a database +catalog.create_database("my_db", false, HashMap::new()).await?; + +// List databases +let databases = catalog.list_databases().await?; + +// Drop a database (cascade = true to drop all tables inside) +catalog.drop_database("my_db", false, true).await?; +``` + +### Manage Tables + +```rust +use paimon::catalog::Identifier; +use paimon::spec::{DataType, IntType, VarCharType, Schema}; + +// Define a schema +let schema = Schema::builder() + .column("id", DataType::Int(IntType::new())) + .column("name", DataType::VarChar(VarCharType::string_type())) + .build()?; + +// Create a table +let identifier = Identifier::new("my_db", "my_table"); +catalog.create_table(&identifier, schema, false).await?; + +// List tables in a database +let tables = catalog.list_tables("my_db").await?; + +// Get a table handle +let table = catalog.get_table(&identifier).await?; +``` + +## Reading a Table + +Paimon Rust uses a scan-then-read pattern: first scan the table to produce splits, then read data from those splits as Arrow `RecordBatch` streams. + +```rust +use futures::StreamExt; + +// Get a table from the catalog +let table = catalog.get_table(&Identifier::new("my_db", "my_table")).await?; + +// Create a read builder +let read_builder = table.new_read_builder(); + +// Step 1: Scan — produces a Plan containing DataSplits +let scan = read_builder.new_scan(); +let plan = scan.plan().await?; + +// Step 2: Read — consumes splits and returns Arrow RecordBatches +let reader = read_builder.new_read()?; +let mut stream = reader.to_arrow(plan.splits())?; + +while let Some(batch) = stream.next().await { + let batch = batch?; + println!("Got batch with {} rows", batch.num_rows()); Review Comment: nit: may be print the record batch ``` println!("RecordBatch: {batch:#?}"); ``` ########## docs/mkdocs.yml: ########## @@ -0,0 +1,65 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +site_name: Apache Paimon Rust +site_description: The Rust implementation of Apache Paimon +site_url: https://paimon.apache.org/paimon-rust/ Review Comment: seem it's not available now. Another question is do we need a separate website to hold the docs just like cpp client `https://alibaba.github.io/paimon-cpp/index.html` ########## docs/src/releases.md: ########## @@ -0,0 +1,43 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Releases + +## Release Policy + +Apache Paimon Rust follows [Semantic Versioning](https://semver.org/). All releases are published to [crates.io](https://crates.io/crates/paimon). + +## Upcoming + +### 0.1.0 (In Development) + +The first release of Apache Paimon Rust. Track progress at the [0.1.0 milestone](https://github.com/apache/paimon-rust/issues/3). + +Planned features: + +- Paimon table format reader +- Local filesystem and S3 storage backends +- REST Catalog client +- Apache DataFusion integration +- Partitioned table support +- C FFI bindings Review Comment: Also add go bindings? ########## docs/src/releases.md: ########## @@ -0,0 +1,43 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Releases + +## Release Policy + +Apache Paimon Rust follows [Semantic Versioning](https://semver.org/). All releases are published to [crates.io](https://crates.io/crates/paimon). + +## Upcoming + +### 0.1.0 (In Development) + +The first release of Apache Paimon Rust. Track progress at the [0.1.0 milestone](https://github.com/apache/paimon-rust/issues/3). + +Planned features: + +- Paimon table format reader +- Local filesystem and S3 storage backends Review Comment: we also support oss ########## docs/src/index.md: ########## @@ -0,0 +1,48 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Apache Paimon Rust + +The Rust implementation of [Apache Paimon](https://paimon.apache.org/) — a streaming data lake platform with high-speed data ingestion, changelog tracking, and efficient real-time analytics. + +## Overview + +Apache Paimon Rust provides native Rust libraries for reading and writing Paimon tables, enabling high-performance data lake access from the Rust ecosystem. + +Key features: + +- Native Rust reader for Paimon table format +- Support for local filesystem and S3 storage backends Review Comment: dito: we also support oss ########## docs/src/architecture.md: ########## @@ -0,0 +1,60 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Architecture + +## Overview + +Apache Paimon Rust is organized as a Cargo workspace with multiple crates, each responsible for a distinct layer of functionality. + +## Crate Structure + +### `crates/paimon` — Core Library + +The core crate implements the Paimon table format, including: + +- **Catalog** — REST Catalog client for discovering and managing databases and tables Review Comment: ```suggestion - **Catalog** — Catalog client for discovering and managing databases and tables ``` Since not only rest catalog, but also filesystem catalog -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
