This is an automated email from the ASF dual-hosted git repository. JackieTien97 pushed a commit to branch ty/module-agent-guidance in repository https://gitbox.apache.org/repos/asf/iotdb.git
commit 7631362f53a9339390c5f178de31c9dfa38bdc3d Author: JackieTien97 <[email protected]> AuthorDate: Fri Oct 23 16:23:39 2026 +0800 Add scoped agent guidance and reusable unordered IT assertions --- AGENTS.md | 80 ++++++++++++++++--- distribution/AGENTS.md | 78 ++++++++++++++++++ integration-test/AGENTS.md | 16 ++++ .../db/it/aggregation/IoTDBTimeDurationIT.java | 40 ++-------- .../org/apache/iotdb/db/it/utils/TestUtils.java | 21 +++++ iotdb-core/ainode/AGENTS.md | 92 ++++++++++++++++++++++ iotdb-core/antlr/AGENTS.md | 54 +++++++++++++ iotdb-core/calc-commons/AGENTS.md | 49 ++++++++++++ iotdb-core/node-commons/AGENTS.md | 49 ++++++++++++ iotdb-core/relational-grammar/AGENTS.md | 55 +++++++++++++ scripts/AGENTS.md | 86 ++++++++++++++++++++ 11 files changed, 574 insertions(+), 46 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index cb1d54cae57..b58cd703af5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -28,6 +28,47 @@ For setup and compatibility notes, see [Coding agents](CONTRIBUTING.md#coding-ag Apache IoTDB is a time series database for IoT data. It uses a distributed architecture with ConfigNodes (metadata/coordination) and DataNodes (storage/query). Data is stored in TsFile columnar format (separate repo: https://github.com/apache/tsfile). Current version is 2.0.11-SNAPSHOT. +## Branch Relationship and Backports + +- `master` is the primary development branch for the 2.x line. `dev/1.3` maintains the 1.3.x + line. They have diverged in supported features, implementation, APIs, and dependencies + (including TsFile); `dev/1.3` is not a mirror of `master`. +- A request to cherry-pick (CP) a `master` PR to `dev/1.3` authorizes only that PR's changes and + the minimum compatibility adaptations needed to preserve their logic on the target branch. + **The backport must be logically and behaviorally equivalent to the original PR. Do not make + additional changes or use the backport as an opportunity to improve the surrounding code.** + +### Cherry-picking from master to dev/1.3 + +1. Read the source PR's final diff, relevant surrounding code, and regression tests. Identify + the exact source commit(s) and target base revision before editing. Base the backport branch + on the requested `dev/1.3` revision, and follow that branch's applicable instructions and + build/test configuration. +2. Prefer `git cherry-pick -x` for the requested commits. When conflicts or branch differences + require manual adaptation, preserve source traceability in the commit/PR description and + resolve each change by its intended behavior. Do not blindly choose one side of a conflict + or copy whole files from `master` over target-branch implementations. +3. Preserve the source PR's conditions, execution order, return values, exceptions/status codes, + side effects, boundary cases, and concurrency/resource-lifecycle guarantees. Adapt package + names, method signatures, or APIs only where the target requires it, using equivalent target + facilities. A clean cherry-pick or successful compilation alone does not prove equivalence. +4. Keep the diff limited to the requested backport. Do not add unrelated fixes, refactors, + optimizations, renames, formatting changes, dependency upgrades, or configuration changes. + Preserve unrelated `dev/1.3` behavior and keep formatter changes limited to the affected code. + Do not silently cherry-pick prerequisite PRs or import newer `master` features to make the + patch compile. +5. If a source change has no target counterpart, a prerequisite is missing, or equivalence cannot + be established, identify the exact gap and the minimum work needed to resolve it. Obtain + direction before expanding scope or dropping part of the source change; do not substitute + a weaker fix or describe an incomplete backport as equivalent. +6. Port the relevant regression tests with equivalent scenarios and assertions, adapting only + to the target test framework. Verify the original failure case and affected boundary/error + paths on `dev/1.3`; do not weaken assertions or skip failing cases to make the backport pass. +7. Review the complete diff against the target base before finishing: account for every source + change (including any already present on the target), and map every target hunk to a source + change or a necessary compatibility adaptation. Report the source PR/commit(s), target base, + adaptations and why they preserve semantics, and actual validation results/limitations. + ## Build Commands ```bash @@ -56,12 +97,30 @@ mvn spotless:apply mvn spotless:check ``` -## Integration Tests +## Module-specific Instructions + +Before editing files or running build/test commands for the areas below, read and follow the +corresponding instructions. For changes spanning multiple areas, read each applicable file, +including when the task starts from the repository root. + +| Area | Instructions | Module-specific guidance | +| --- | --- | --- | +| AINode | [iotdb-core/ainode/AGENTS.md](iotdb-core/ainode/AGENTS.md) | Python environment, generated sources, formatting, tests, and binary packaging | +| Tree-model SQL grammar | [iotdb-core/antlr/AGENTS.md](iotdb-core/antlr/AGENTS.md) | Lexer keywords, identifier compatibility, and grammar verification | +| Table-model SQL grammar | [iotdb-core/relational-grammar/AGENTS.md](iotdb-core/relational-grammar/AGENTS.md) | Reserved/non-reserved keywords, identifier compatibility, and grammar verification | +| Shared node models and infrastructure | [iotdb-core/node-commons/AGENTS.md](iotdb-core/node-commons/AGENTS.md) | Shared paths, schema, partition/auth/configuration models, and RPC/service abstractions | +| Reusable query computation | [iotdb-core/calc-commons/AGENTS.md](iotdb-core/calc-commons/AGENTS.md) | Reusable operators, column transformations, aggregation, joins, and sorting | +| Distribution and assembly descriptors | [distribution/AGENTS.md](distribution/AGENTS.md) | Artifact identities, package contents, Edge configuration, and packaging verification | +| Launchers, environment scripts, and operational tools | [scripts/AGENTS.md](scripts/AGENTS.md) | Platform compatibility, process ownership, cleanup behavior, and script verification | +| Integration tests | [integration-test/AGENTS.md](integration-test/AGENTS.md) | IT authoring, test profiles, targeted verification, and Edge/IDE runs | + +Read both commons guides before adding or moving shared code between these modules. Logic tied to +a concrete node's storage, lifecycle, or execution scheduling belongs in that node's module. -Before editing or running integration tests, read and follow -[integration-test/AGENTS.md](integration-test/AGENTS.md). It contains the IT authoring rules, -formatting and build commands, test profiles, targeted verification, and Edge/IDE instructions. -Keep IT-specific rules and commands in that module-level file. +For changes to `EdgeNode` or Edge configuration defaults outside these directories, also read the +distribution and scripts instructions. For SQL parsing or keyword changes in DataNode code, also +read the relevant grammar instructions. Keep shared rules in this file and specialized rules in +the corresponding module file; link to existing guidance instead of copying it. ## Code Style @@ -80,12 +139,10 @@ Keep IT-specific rules and commands in that module-level file. ### Edge Distribution -- `EdgeNode` (`iotdb-core/confignode/src/main/java/org/apache/iotdb/edge/EdgeNode.java`) starts the ConfigNode and DataNode in one JVM for resource-constrained single-node deployments. -- `distribution/src/assembly/edge.xml` produces the additional `apache-iotdb-<version>-edge-bin.zip` artifact. Keep its `edge-bin` assembly id unique so it cannot replace an existing attached artifact. -- Edge-specific system defaults live under `iotdb-core/node-commons/src/assembly/resources/conf/edge/`. Standard node, all-in-one, and integration-test assemblies must continue to exclude `edge/**`. -- Edge launchers must reuse the configuration-aware port checks in `scripts/conf/iotdb-common.sh`. Stop scripts must match both the `EdgeNode` main class and the exact `IOTDB_HOME`; never fall back to a global process-name kill. -- Package node-independent tools that work with the combined process. Exclude scripts that require standalone node launchers or environment files, including destructive, daemon-management, and health-check scripts. -- After packaging changes, run a clean distribution build, check that every existing distribution artifact is still produced, and inspect the Edge zip contents and startup/shutdown behavior. +`EdgeNode` (`iotdb-core/confignode/src/main/java/org/apache/iotdb/edge/EdgeNode.java`) starts the +ConfigNode and DataNode in one JVM for resource-constrained single-node deployments. Its package +layout and operational requirements are maintained in the distribution and scripts instructions +listed above. ### Dual Data Model @@ -164,7 +221,6 @@ The project uses compile-time i18n via the `build-helper-maven-plugin`. The prop ### Code Style - **Always run `mvn spotless:apply` after editing Java files**: Spotless runs `spotless:check` automatically during the `compile` phase. Format violations cause an immediate BUILD FAILURE. Make it a habit to run `mvn spotless:apply -pl <module>` right after editing, not at the end. -- **Always run `black` and `isort` after editing Python files under `iotdb-core/ainode/`**: The AINode Code Style Check CI runs `black --check .` and `isort --check-only --profile black .` on that directory. Run `cd iotdb-core/ainode && black . && isort --profile black .` before committing. Requires `pip install black==25.1.0 isort==6.0.1`. - **Gson version compatibility**: `JsonObject.isEmpty()` / `JsonArray.isEmpty()` may not be available in the Gson version used by this project. Use `size() > 0` instead and add a comment explaining why. ## Git Commit diff --git a/distribution/AGENTS.md b/distribution/AGENTS.md new file mode 100644 index 00000000000..f1ee9f1c2e3 --- /dev/null +++ b/distribution/AGENTS.md @@ -0,0 +1,78 @@ +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +# Distribution Guidelines + +These instructions apply to `distribution/` and supplement the +[repository instructions](../AGENTS.md). + +## Assembly ownership and artifact identity + +- Treat [pom.xml](pom.xml) and `src/assembly/` as the sources of package layout. Edit the + original configuration or script source, not staged files or archives under `target/`. +- Preserve the default `all-bin`, `library-udf-bin`, `library-pipe-bin`, and `edge-bin` + assemblies. `with-ainode` and `with-cpp` add their own distributions; keep profile descriptors + additive so enabling one does not replace the default descriptor list. +- Assembly IDs determine attached artifact classifiers. Keep each ID unique, especially + `edge-bin`, and check final archive/base-directory names and SHA-512 checksum includes when + adding or renaming an artifact. Preserve existing outputs for every affected profile. +- Keep LICENSE, NOTICE, and bundled dependency licenses in the resulting archives; reuse + `src/assembly/common-files.xml` and the AINode-specific component where applicable. +- Preserve executable file modes and Maven filtering for shared environment scripts, including + `iotdb-common.sh` and `windows/iotdb-common.bat`. Check includes/excludes against both Unix + and Windows paths. Read [scripts/AGENTS.md](../scripts/AGENTS.md) when changing script contents + or availability, and [AINode instructions](../iotdb-core/ainode/AGENTS.md) for AINode packaging. + +## Edge package boundaries + +- [src/assembly/edge.xml](src/assembly/edge.xml) produces + `apache-iotdb-<version>-edge-bin.zip` for the combined ConfigNode/DataNode process. +- Edge system defaults come from + `iotdb-core/node-commons/src/assembly/resources/conf/edge/`; Edge logging configuration comes + from `distribution/src/assembly/resources/conf-edge/`. Keep these separate from standard + configuration. Standard node, all-in-one, and integration-test packages must not acquire + Edge defaults; preserve `edge/**` exclusions wherever shared configuration directories are + copied recursively. +- Package the Edge start/stop pair and compatible CLI/tools. Exclude tools requiring standalone + node launchers or environment files, including standalone daemon, destroy, and health-check + scripts. The explicit Edge daemon/destroy replacements are intentional exceptions; retain + their inclusion without restoring the standalone tools. +- Check the standalone node and integration-test assembly descriptors as well when changing + shared configuration or script selection; a correct Edge archive does not establish that + other packages are unaffected. + +## Packaging verification + +For changes to assembly descriptors, packaged resources, or artifact wiring: + +1. Run the clean distribution build from the + [root build instructions](../AGENTS.md#build-commands). Enable any affected optional profile + as well; an old archive in `target/` is not evidence of a successful new build. +2. Inspect the produced archives and SHA-512 files under `distribution/target/`. Verify all + expected artifacts for the active profiles, archive roots, configuration, dependency jars, + licenses, Unix executable modes, and Windows entries. Confirm unwanted tools and Edge-only + defaults are absent from the other packages. +3. Inspect the Edge zip and verify startup/shutdown from a freshly extracted package using an + isolated installation. For Edge package selection changes, also run the script/assembly + checks described in [scripts/AGENTS.md](../scripts/AGENTS.md#verification). +4. For Edge runtime changes, follow the build and Tree/Table read-write verification in + [integration-test/AGENTS.md](../integration-test/AGENTS.md#edge-tests). Keep the IT commands + there rather than duplicating them here. diff --git a/integration-test/AGENTS.md b/integration-test/AGENTS.md index 15690a76477..9f3df1541e3 100644 --- a/integration-test/AGENTS.md +++ b/integration-test/AGENTS.md @@ -68,6 +68,22 @@ These instructions apply to `integration-test/` and supplement the repository-le - Assert the behavior under test and its recovery result. Only tolerate expected cleanup errors in narrowly scoped cleanup code; do not swallow failures from the operation being verified. +### Result ordering across DataNodes + +- When row order is not part of the query's contract and may differ across DataNodes, use + `TestUtils.resultSetEqualTestUnordered(sql, expectedHeader, expectedRetArray)` for tree-model + SQL cases. When the test already owns a result set (including table-model queries), use + `TestUtils.assertResultSetEqualUnordered(resultSet, expectedHeader, expectedRows)` and close + the original result set after the assertion. Reuse these helpers instead of writing per-node + comparison loops in individual tests. +- These helpers check every node independently, including headers, values, and duplicate counts. + The default `ClusterTestResultSet` compares nodes at the same cursor position; collecting or + sorting rows after reading through that wrapper cannot avoid row-order mismatches. Do not + replace the helpers with set equality or single-node checks that weaken these assertions. +- Keep ordered assertions when SQL guarantees the final result order. For `LIMIT`/`OFFSET`, + ensure the selected rows are deterministic before applying unordered equality; sorting cannot + repair a nondeterministic subset. Add sufficient `ORDER BY` tie-breakers when needed. + ## Verification This module is excluded from the default build. Run the commands below from the repository root, diff --git a/integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/IoTDBTimeDurationIT.java b/integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/IoTDBTimeDurationIT.java index c6d8079abb4..ce2ea3eef36 100644 --- a/integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/IoTDBTimeDurationIT.java +++ b/integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/IoTDBTimeDurationIT.java @@ -20,7 +20,6 @@ package org.apache.iotdb.db.it.aggregation; import org.apache.iotdb.it.env.EnvFactory; -import org.apache.iotdb.it.env.cluster.node.DataNodeWrapper; import org.apache.iotdb.it.framework.IoTDBTestRunner; import org.apache.iotdb.itbase.category.ClusterIT; import org.apache.iotdb.itbase.category.LocalStandaloneIT; @@ -32,20 +31,10 @@ import org.junit.Test; import org.junit.experimental.categories.Category; import org.junit.runner.RunWith; -import java.sql.Connection; -import java.sql.ResultSet; -import java.sql.ResultSetMetaData; -import java.sql.SQLException; -import java.sql.Statement; -import java.util.Arrays; -import java.util.HashSet; -import java.util.Set; - import static org.apache.iotdb.db.it.utils.TestUtils.assertTestFail; import static org.apache.iotdb.db.it.utils.TestUtils.prepareData; import static org.apache.iotdb.db.it.utils.TestUtils.resultSetEqualTest; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertTrue; +import static org.apache.iotdb.db.it.utils.TestUtils.resultSetEqualTestUnordered; @RunWith(IoTDBTestRunner.class) @Category({LocalStandaloneIT.class, ClusterIT.class}) @@ -225,7 +214,7 @@ public class IoTDBTimeDurationIT { } @Test - public void testTimeDurationManyColumnOnManyDataRegion() throws SQLException { + public void testTimeDurationManyColumnOnManyDataRegion() { // normal String[] expectedHeader = new String[] { @@ -357,27 +346,10 @@ public class IoTDBTimeDurationIT { // group by tags expectedHeader = new String[] {"city", "time_duration(s1)"}; retArray = new String[] {"Beijing,1677570933,", "Nanjing,1677570938,"}; - // GROUP BY TAGS does not guarantee row order, so check each DataNode independently. - for (DataNodeWrapper dataNode : EnvFactory.getEnv().getDataNodeWrapperList()) { - try (Connection connection = EnvFactory.getEnv().getConnection(dataNode); - Statement statement = connection.createStatement(); - ResultSet resultSet = - statement.executeQuery( - "select time_duration(s1) from root.db1.**,root.db.** group by tags(city)")) { - ResultSetMetaData metaData = resultSet.getMetaData(); - assertEquals(expectedHeader.length, metaData.getColumnCount()); - for (int i = 0; i < expectedHeader.length; i++) { - assertEquals(expectedHeader[i], metaData.getColumnName(i + 1)); - } - Set<String> remainingRows = new HashSet<>(Arrays.asList(retArray)); - while (resultSet.next()) { - String row = - resultSet.getString("city") + "," + resultSet.getString("time_duration(s1)") + ","; - assertTrue(dataNode.getIpAndPortString() + ": " + row, remainingRows.remove(row)); - } - assertTrue(dataNode.getIpAndPortString() + ": " + remainingRows, remainingRows.isEmpty()); - } - } + resultSetEqualTestUnordered( + "select time_duration(s1) from root.db1.**,root.db.** group by tags(city)", + expectedHeader, + retArray); // group by time expectedHeader = diff --git a/integration-test/src/test/java/org/apache/iotdb/db/it/utils/TestUtils.java b/integration-test/src/test/java/org/apache/iotdb/db/it/utils/TestUtils.java index cc2556578b9..a0d768ec771 100644 --- a/integration-test/src/test/java/org/apache/iotdb/db/it/utils/TestUtils.java +++ b/integration-test/src/test/java/org/apache/iotdb/db/it/utils/TestUtils.java @@ -47,6 +47,7 @@ import java.text.DateFormat; import java.time.ZoneId; import java.time.ZoneOffset; import java.util.ArrayList; +import java.util.Arrays; import java.util.Collection; import java.util.Collections; import java.util.HashMap; @@ -280,6 +281,26 @@ public class TestUtils { resultSetEqualTest(sql, expectedHeader, expectedRetArray, null); } + /** + * Executes a tree-model query and checks unordered rows on every read node, including duplicate + * multiplicity. Use only when the query's final result does not require a particular row order or + * select a nondeterministic subset of rows. + */ + public static void resultSetEqualTestUnordered( + String sql, String[] expectedHeader, String[] expectedRetArray) { + try (Connection connection = EnvFactory.getEnv().getConnection(); + Statement statement = connection.createStatement()) { + connection.setClientInfo("time_zone", "+00:00"); + try (ResultSet resultSet = statement.executeQuery(sql)) { + assertResultSetEqualUnordered( + resultSet, String.join(",", expectedHeader) + ",", Arrays.asList(expectedRetArray)); + } + } catch (SQLException e) { + e.printStackTrace(); + fail(e.getMessage()); + } + } + public static void tableResultSetEqualTest( String sql, String[] expectedHeader, String[] expectedRetArray, String database) { tableResultSetEqualTest( diff --git a/iotdb-core/ainode/AGENTS.md b/iotdb-core/ainode/AGENTS.md new file mode 100644 index 00000000000..b894f3a88a1 --- /dev/null +++ b/iotdb-core/ainode/AGENTS.md @@ -0,0 +1,92 @@ +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +# AINode Guidelines + +These instructions apply to `iotdb-core/ainode/` and supplement the +[repository instructions](../../AGENTS.md). + +## Python environment and source ownership + +- Use an isolated Python environment matching [pyproject.toml](pyproject.toml); the runtime + currently requires Python 3.11 (`>=3.11.0,<3.12.0`). The Python version in the formatting-only + CI job does not define the supported runtime. +- AINode implementation lives in `iotdb/ainode/core/`. Keep configuration parsing and defaults + (`config.py` and `constant.py` there) aligned with `resources/conf/iotdb-ainode.properties`. +- Maven copies the Python client and generated Thrift modules from `iotdb-client/client-py/iotdb` + into this module. Fix their original sources or Thrift IDL rather than the copied files under + `iotdb/`. Build metadata under `iotdb/ainode/conf/` is generated from `resources/pom.properties` + and Git; do not edit generated metadata or files under `build/`, `dist/`, or `target/`. + +## Formatting + +Always run Black and isort after editing Python files here. Use the versions pinned by +[AINode Code Style Check](../../.github/workflows/greeting-ainode.yml), currently +`black==25.1.0` and `isort==6.0.1`. Use a separate formatter environment if the Black dependency +in `pyproject.toml` differs from the CI pin. + +From `iotdb-core/ainode/`, apply and check formatting: + +```bash +black . +isort --profile black . +black --check . +isort --check-only --profile black . +``` + +## Targeted verification + +- Existing Python tests use `unittest`. From this module directory, with the runtime environment + active, select the affected test file, for example: + + ```bash + python -m unittest discover -s tests -p 'test_config.py' + ``` + +- Model tests such as `tests/test_sundial_flow_loss.py` require the relevant ML dependencies. + Preserve checkpoint parameter names/shapes and tensor shape/gradient contracts when changing + model code. Keep small unit cases independent of model downloads or a running cluster where + possible; do not treat a CPU-only run as GPU validation. +- For registration, RPC, or cluster inference changes, read + [integration-test/AGENTS.md](../../integration-test/AGENTS.md). The cluster verification setup + is in [Cluster IT - 1C1D1A](../../.github/workflows/cluster-it-1c1d1a.yml): it uses + `AIClusterIT`, `with-integration-tests`, and `with-ainode`, and runs on a GPU runner. + +## Binary packaging + +- AINode is opt-in through `with-ainode`. From the repository root, build the module and its + dependencies with a Python 3.11 executable available under the selected name: + + ```bash + mvn clean package -pl iotdb-core/ainode -am -P with-ainode -DskipTests -Dpython.exe.bin=python3.11 + ``` + +- The Maven `compile` phase invokes `build_binary.py`, which installs Poetry/dependencies, + regenerates `poetry.lock`, and builds or restores a PyInstaller binary. It is not a lightweight + Python syntax check. The script stores its environment and binary cache under + `~/.cache/iotdb-ainode-build/` on Unix or `%LOCALAPPDATA%\iotdb-ainode-build\` on Windows; + `TMPDIR` and pip cache settings do not relocate that script-managed directory. Check cache + placement against the active workspace policy before invoking it. +- For dependency, dynamic-import, or bundled-resource changes, inspect `ainode.spec` and + `ainode.xml` and verify the packaged executable, not only imports from the source tree. + Keep AINode-specific NOTICE and license resources aligned with the bundled contents. +- Read [distribution/AGENTS.md](../../distribution/AGENTS.md) when changing the final AINode + distribution, and [scripts/AGENTS.md](../../scripts/AGENTS.md) when changing its launchers. diff --git a/iotdb-core/antlr/AGENTS.md b/iotdb-core/antlr/AGENTS.md new file mode 100644 index 00000000000..4547a4ee850 --- /dev/null +++ b/iotdb-core/antlr/AGENTS.md @@ -0,0 +1,54 @@ +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +# Tree-Model SQL Grammar Guidelines + +These instructions apply to `iotdb-core/antlr/` and supplement the +[repository instructions](../../AGENTS.md). + +## Keyword and identifier compatibility + +- When adding a keyword token to + [SqlLexer.g4](src/main/antlr4/org/apache/iotdb/db/qp/sql/SqlLexer.g4), **add the same token to + the `keyWords` rule in + [IdentifierParser.g4](src/main/antlr4/org/apache/iotdb/db/qp/sql/IdentifierParser.g4) in the + same change**. A new dedicated lexer token no longer matches the generic `ID` alternative; + omitting it from `keyWords` can break its use as an unquoted measurement or path node name. +- Preserve the existing identifier and path rules in + [IoTDBSqlParser.g4](src/main/antlr4/org/apache/iotdb/db/qp/sql/IoTDBSqlParser.g4), which imports + `IdentifierParser`. Do not make users quote previously valid unquoted names to work around a + missing keyword entry, or change existing reserved/special-token behavior incidentally. +- This synchronization applies to keyword tokens. Do not add punctuation, operators, + whitespace, lexer fragments, or literal rules to `keyWords` merely because they are defined + in the lexer. + +## Verification + +- Verify both the newly introduced SQL syntax and the same word used as an unquoted measurement + and path component in create/insert/query statements. Include case variants; testing only + quoted identifiers does not cover this compatibility requirement. The existing + [IoTDBSyntaxConventionIdentifierIT](../../integration-test/src/test/java/org/apache/iotdb/db/it/IoTDBSyntaxConventionIdentifierIT.java) + provides identifier/path regression coverage. Follow + [integration-test/AGENTS.md](../../integration-test/AGENTS.md) before editing or running ITs. +- Edit the `.g4` sources and regenerate through Maven; do not edit generated Java, `.tokens`, + or `.interp` files. Rebuild the affected grammar and DataNode consumers and run the affected + parser/identifier tests. Successful grammar generation alone does not prove that existing + names still parse correctly. diff --git a/iotdb-core/calc-commons/AGENTS.md b/iotdb-core/calc-commons/AGENTS.md new file mode 100644 index 00000000000..40c83d5a25b --- /dev/null +++ b/iotdb-core/calc-commons/AGENTS.md @@ -0,0 +1,49 @@ +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +# Calc Commons Guidelines + +These instructions apply to `iotdb-core/calc-commons/` and supplement the +[repository instructions](../../AGENTS.md). + +## Responsibilities and code placement + +- This module provides reusable query computation and execution for server and stream execution + environments. Place reusable operators, column transformations, aggregation, joins, sorting, + windows, and their execution/memory utilities here, under `org.apache.iotdb.calc`. +- Shared node models and infrastructure belong in [node-commons](../node-commons/AGENTS.md). + Keep concrete storage/DataRegion integration, fragment lifecycle, and node scheduler wiring + in the owning node module; separate reusable computation from that integration. +- Existing SQL constants are also consumed by ConfigNode and CLI. Such usage does not make this + module a destination for arbitrary shared utilities. Choose by responsibility and actual + dependencies; do not relocate existing code solely because of its package or class name. + +## Dependency boundary + +- The dependency direction is **`calc-commons` -> `node-commons`**, never the reverse. Do not + introduce Maven dependencies or Java imports on concrete DataNode/ConfigNode implementations. +- Obtain host services through common contexts or small provider interfaces. For example, + `CommonOperatorContext` supplies the execution context, while file-service providers abstract + temporary/object-file access. Keep their node-specific implementations in the consuming module. + Do not downcast to DataNode's `OperatorContext` or access a concrete node singleton from shared + computation code. +- When changing a shared interface, inspect its implementations and callers across consuming + modules. Preserve use from standalone/test contexts without requiring a running DataNode. diff --git a/iotdb-core/node-commons/AGENTS.md b/iotdb-core/node-commons/AGENTS.md new file mode 100644 index 00000000000..9cace7b8cf8 --- /dev/null +++ b/iotdb-core/node-commons/AGENTS.md @@ -0,0 +1,49 @@ +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +# Node Commons Guidelines + +These instructions apply to `iotdb-core/node-commons/` and supplement the +[repository instructions](../../AGENTS.md). + +## Responsibilities and code placement + +- This module provides models, contracts, and infrastructure shared across nodes, under + `org.apache.iotdb.commons`: paths, schema, partitions, authentication/authorization, + configuration, RPC clients, and common service abstractions. +- Shared AST/PlanNode definitions and serialization contracts also live here. Changes must + account for all consuming nodes, including their serialization/deserialization counterparts. + Keep concrete node startup, storage operations, and execution orchestration in the node module. +- Reusable execution operators, column transformations, and query computation generally belong + in [calc-commons](../calc-commons/AGENTS.md). Existing grouping/hash and UDF primitives here mean + that the current boundary has overlap; inspect consumers and dependencies before deciding + placement. Do not move code merely because it is query-related or called a common utility. + +## Dependency boundary + +- The dependency direction is **`calc-commons` -> `node-commons`**. This module must not depend + on `calc-commons` or concrete DataNode/ConfigNode implementations through Maven or Java imports. + Shared APIs, protocol types, and lower-level libraries remain appropriate dependencies. +- When common code needs behavior supplied by a node, define the smallest necessary interface + here and implement it in the consuming module, as with `IPlanNodeDeserializerProvider`. + Do not pull a concrete node implementation into this module to satisfy one caller. +- Check all consumers before changing shared types or interfaces. Keep existing cross-node + contracts intact unless changing those contracts is explicitly part of the task. diff --git a/iotdb-core/relational-grammar/AGENTS.md b/iotdb-core/relational-grammar/AGENTS.md new file mode 100644 index 00000000000..3edf66d3885 --- /dev/null +++ b/iotdb-core/relational-grammar/AGENTS.md @@ -0,0 +1,55 @@ +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +# Table-Model SQL Grammar Guidelines + +These instructions apply to `iotdb-core/relational-grammar/` and supplement the +[repository instructions](../../AGENTS.md). + +## Reserved and non-reserved keywords + +- For each new keyword token in the lexer portion of + [RelationalSql.g4](src/main/antlr4/org/apache/iotdb/db/relational/grammar/sql/RelationalSql.g4), + determine whether it is intentionally reserved. **If it is not a reserved word, add the token + to the `nonReserved` rule in the same change.** Defining only the lexer token can make a + previously valid unquoted database, table, column, or alias name stop parsing, because the + `identifier` rule accepts these keyword tokens through `nonReserved`. +- Keep `nonReserved` as a list of token alternatives only; do not add nested parser rules. + The DataNode SQL parser's `exitNonReserved` implementation depends on this structure. +- Do not reserve a word accidentally by omitting it from `nonReserved`, or make an existing + reserved word non-reserved merely to resolve a grammar conflict. Intentional reservation + must be part of the SQL change's contract and covered by compatibility tests. This rule is + about keyword tokens, not operators, punctuation, whitespace, fragments, or literals. + +## Verification + +- Cover the new SQL syntax and non-reserved keyword use as unquoted identifiers in the relevant + database/table/column/alias positions, including case variants. Quoted-only tests do not + protect unquoted-name compatibility. For intentionally reserved words, verify rejection in + unquoted identifier positions and acceptance with supported quoting. +- The existing + [StreamKeywordTest](../datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/relational/sql/parser/StreamKeywordTest.java) + demonstrates reserved and non-reserved identifier checks. For integration coverage, follow + [integration-test/AGENTS.md](../../integration-test/AGENTS.md) and use the table-model profile. +- Edit the `.g4` sources and regenerate through Maven; do not edit generated Java, `.tokens`, + or `.interp` files. Rebuild the affected grammar and DataNode consumers and run the affected + parser/identifier tests. Successful grammar generation alone does not establish identifier + compatibility or correct AST construction. diff --git a/scripts/AGENTS.md b/scripts/AGENTS.md new file mode 100644 index 00000000000..25346351ead --- /dev/null +++ b/scripts/AGENTS.md @@ -0,0 +1,86 @@ +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +# Script Guidelines + +These instructions apply to `scripts/` and supplement the +[repository instructions](../AGENTS.md). + +## Shared sources and platform behavior + +- `conf/` holds environment setup and shared helpers, `sbin/` holds launchers, and `tools/` + holds operational utilities. Their Windows counterparts live in the respective `windows/` + directories. Review the counterpart when changing shared behavior and keep arguments, + configuration handling, and exit status consistent where applicable. +- Scripts are copied into several distributions. Read + [distribution/AGENTS.md](../distribution/AGENTS.md) before changing packaged file names, + locations, or dependencies. Update original sources here rather than copies under `target/`. +- Resolve installation paths relative to the script or the supported environment override, + not the caller's working directory. Preserve `IOTDB_HOME`, `IOTDB_CONF`, data/log directory, + and `JAVA_HOME` handling. Quote paths and forwarded arguments, including paths with spaces. +- Reuse `conf/iotdb-common.sh` and `conf/windows/iotdb-common.bat` for shared behavior. Preserve + Maven-filtered placeholders such as `@tsfile.locale.opt@`; inspect the packaged result when + changing them. Keep supported Linux/macOS branches and explicitly guard platform-specific + operations such as systemd integration. + +## Edge lifecycle and cleanup + +- Edge startup must use the configuration-aware ConfigNode and DataNode port checks in the + shared helpers. Preserve custom configuration paths and fail before launching when required + validation fails; do not substitute hardcoded default-port checks. +- Edge stop logic must match both the `org.apache.iotdb.edge.EdgeNode` main class and the exact + installation `IOTDB_HOME`, including when recovering from a missing/stale PID file. Validate + ownership again before forced termination; never fall back to a global process-name kill. +- Preserve Edge daemon tracking of the forked JVM through `edge.pid`. During cleanup, stop only + the matching installation's systemd service before stopping the process so it cannot restart + while its data is being removed. Propagate stop/service failures instead of continuing. +- Preserve cleanup confirmation and force-option behavior. Read effective configured data + paths, including legacy configuration and multiple data tiers, and validate every target + before deleting any. Retain protection for filesystem/home roots, parent directories, and + symlink aliases. Report deletion failures instead of printing a success message. + +## Verification + +- Run `bash -n` on each edited Bash script. This checks syntax only; exercise changed argument, + path, process, and failure behavior with an isolated installation or the existing harness. +- For Edge operational scripts or their assembly selection, run from the repository root: + + ```bash + python3 .github/scripts/test-edge-ops.py + ``` + + The harness covers cleanup, daemon behavior, and assembly selection with temporary fixtures. + Keep new cases similarly isolated; do not use an existing installation or real system service + as a cleanup fixture. +- For Windows Edge launchers, also run on Windows from the repository root: + + ```powershell + powershell -NoProfile -File .github/scripts/test-edge-windows.ps1 + ``` + + These checks are wired into [Edge IT](../.github/workflows/edge-it.yml). Report platform-specific + checks that could not run; Unix results do not establish Windows behavior. +- For packaged launcher or environment changes, rebuild the affected distribution and smoke-test + its extracted scripts. Verify custom configuration, an unrelated working directory, occupied + ports, and isolation from another installation when those behaviors are affected. Follow + [distribution verification](../distribution/AGENTS.md#packaging-verification) for archive checks + and [Edge IT instructions](../integration-test/AGENTS.md#edge-tests) for Tree/Table read-write + verification.
