AliRana30 opened a new issue, #875:
URL: https://github.com/apache/wayang/issues/875

   ### Description
   
   Wayang currently supports embedded execution via `wayang-java` (Java 
Streams) and `wayang-sqlite3` (row-oriented embedded SQL), as well as 
server-based platforms (Spark, Flink, Postgres, BigQuery, and the proposed 
ClickHouse platform in #871). 
   
   However, Wayang currently lacks an **in-process, vectorized columnar OLAP 
platform**. 
   
   I propose adding a new platform module: `wayang-platforms/wayang-duckdb`. 
   
   DuckDB is widely regarded as the "SQLite of OLAP". Adding DuckDB enables 
Wayang to execute analytical queries (aggregations, joins, filter pushdowns) 
with vectorized columnar speed directly in-process, without requiring users to 
launch or maintain a remote cluster (like Spark) or external database daemon 
(like Postgres or ClickHouse).
   
   ### Value & Motivation
   
   1. **High-Speed In-Process OLAP**: Gives Wayang a fast, local columnar 
execution tier for medium-sized analytical workloads that are too large for 
simple Java Streams but too small to justify cluster overhead.
   2. **Zero Infrastructure & Reliable CI**: Uses the official 
`org.duckdb:duckdb_jdbc` driver (MIT license). Because DuckDB is in-process, 
full end-to-end integration tests (`DuckDBOperatorsIT`) run directly in CI 
without requiring external servers or Docker containers.
   3. **Native Parquet/CSV Compatibility**: Lays the foundation for pushing 
file-based scans directly into a vectorized SQL engine.
   
   ### Proposed Architecture & Layout
   
   Following the architecture of `wayang-sqlite3` and `wayang-jdbc-template`:
   
   | Component | Class / File | Purpose |
   |---|---|---|
   | Entry point | `DuckDB.java` | Static helper for `plugin()`, `platform()` |
   | Platform | `DuckDBPlatform extends JdbcPlatformTemplate` | Platform 
definition, configuration ID `duckdb` |
   | Operators | 
`DuckDB{TableSource,Filter,Projection,Join,GlobalReduce,ReduceBy,Sort,TableSink}Operator`
 | Extend corresponding `Jdbc*` operators |
   | Mappings | `Mappings.java` (operator mappings + `Mappings.ALL`) | 
Translates Wayang operators to DuckDB physical operators |
   | Channels | `ChannelConversions` | Bridges DuckDB SQL -> Java Streams / 
Spark RDDs |
   | Config | `wayang-duckdb-defaults.properties` | JDBC URL (`jdbc:duckdb:` 
in-memory or file-backed), driver class, cost parameters |
   | Build | `wayang-platforms/pom.xml` | Register new 
`<module>wayang-duckdb</module>` |
   
   ### Dialect & Engine Specifics
   
   - **Connection handling**: Default to in-process in-memory database 
(`jdbc:duckdb:`) or user-configured database file paths 
(`wayang.duckdb.jdbc.url`).
   - **Table Sink**: DuckDB fully supports standard `CREATE TABLE <t> AS SELECT 
...` and `INSERT INTO <t> SELECT ...`, cleanly matching `JdbcTableSinkOperator`.
   - **Quoting & Identifiers**: Case preservation and identifier escaping 
conform to standard ANSI SQL quotes (`"identifier"`).
   
   ### Proposed Phased Implementation
   
   To keep code reviews concise and self-contained:
   
   1. **PR 1: Module Skeleton & Basic Operators**
      - Create `wayang-platforms/wayang-duckdb` module.
      - Add `DuckDBPlatform`, `DuckDB.java`, plugin registration, and 
configuration properties.
      - Implement `DuckDBTableSource`, `DuckDBFilterOperator`, and 
`DuckDBProjectionOperator`.
      - Unit tests covering SQL clause generation.
   2. **PR 2: Complex Operators & Integration Tests**
      - Implement `Join`, `GlobalReduce`, `ReduceBy`, `Sort`, and `TableSink` 
operators.
      - Add `DuckDBOperatorsIT` running real queries against an in-process 
DuckDB instance (100% CI compatible).
   3. **PR 3: Documentation & Application Demo**
      - Add module `README.md`, an example demo in `wayang-applications`, and 
update the platform list in `wayang-docs`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to