This is an automated email from the ASF dual-hosted git repository. jin pushed a commit to branch serena in repository https://gitbox.apache.org/repos/asf/incubator-hugegraph-toolchain.git
commit 19fc6d9f264bca27d84847b0edf902b777bdcb7e Author: imbajin <[email protected]> AuthorDate: Wed Nov 5 18:59:15 2025 +0800 feat: init serena onboarding & project memory files Introduces the .serena directory with onboarding documentation, project architecture, code style, workflows, and related memory files for HugeGraph Toolchain. These files provide guidelines and quick start information for new developers, covering project overview, design patterns, code conventions, testing, and development workflows. --- .serena/.gitignore | 1 + .serena/memories/README_INDEX.md | 244 ++++++++ .../memories/architecture_and_design_patterns.md | 572 ++++++++++++++++++ .serena/memories/code_style_and_conventions.md | 312 ++++++++++ .serena/memories/common_development_workflows.md | 658 +++++++++++++++++++++ .serena/memories/project_overview.md | 126 ++++ .serena/memories/task_completion_checklist.md | 373 ++++++++++++ .serena/memories/testing_infrastructure.md | 635 ++++++++++++++++++++ .serena/project.yml | 84 +++ 9 files changed, 3005 insertions(+) diff --git a/.serena/.gitignore b/.serena/.gitignore new file mode 100644 index 00000000..14d86ad6 --- /dev/null +++ b/.serena/.gitignore @@ -0,0 +1 @@ +/cache diff --git a/.serena/memories/README_INDEX.md b/.serena/memories/README_INDEX.md new file mode 100644 index 00000000..ba720238 --- /dev/null +++ b/.serena/memories/README_INDEX.md @@ -0,0 +1,244 @@ +# Memory Index - HugeGraph Toolchain Project + +## Onboarding Complete ✓ + +This project has been successfully initialized with Serena MCP. Below is an index of all available memory files. + +## Available Memories (7 Core Files) + +### Core Project Information + +1. **project_overview.md** (125 lines) + - Project purpose and goals + - All 6 modules (client, loader, hubble, tools, client-go, spark-connector) + - Technology stack (Java 8, Node.js 18.20.8, Go, React, Spring Boot) + - Module dependencies and relationships + - External dependencies + - Project structure + +### Code Quality and Style + +2. **code_style_and_conventions.md** (311 lines) + - Java code style (indentation, naming, formatting) + - Naming conventions (classes, methods, variables) + - Import rules and prohibited imports + - Maven POM style + - Frontend code style (TypeScript, CSS/Less) + - Go code style + - Design patterns used in each module + - File organization standards + - Commit message format + +3. **task_completion_checklist.md** (372 lines) ⭐ **IMPORTANT** + - Code quality checks before committing + - Testing requirements + - Build verification steps + - Documentation update checklist + - Git pre-commit checklist + - Pull request checklist + - CI/CD pipeline details + - Common issues and solutions + - Release-specific tasks + +### Architecture and Design + +4. **architecture_and_design_patterns.md** (571 lines) + - Overall system architecture + - Module-specific architectures: + - hugegraph-client: Layered architecture, Manager pattern + - hugegraph-loader: Pipeline architecture + - hugegraph-hubble: Frontend (React+MobX) + Backend (Spring Boot) + - hugegraph-tools: Command pattern + - Design patterns (Factory, Builder, Strategy, Observer, Repository) + - Cross-cutting concerns (error handling, logging) + - Configuration management + +### Testing + +5. **testing_infrastructure.md** (634 lines) + - Testing philosophy (unit, integration, functional) + - Test organization and structure + - Module-specific testing: + - hugegraph-client: UnitTestSuite, ApiTestSuite, FuncTestSuite + - hugegraph-loader: Test profiles (unit, file, hdfs, jdbc, kafka) + - hugegraph-hubble: Backend (Spring Test) + Frontend (Jest) + - hugegraph-client-go: Go standard testing + - CI/CD testing pipelines + - Test coverage tools and targets + - Common testing patterns + - Debugging tests + +### Development Workflows + +6. **common_development_workflows.md** (657 lines) + - Daily development workflows: + - Starting new features + - Fixing bugs + - Adding tests + - Refactoring code + - Module-specific workflows + - Troubleshooting common issues + - Release workflow + - Useful development commands + - Git hooks setup + - IDE configuration (IntelliJ IDEA, VS Code) + +## Quick Start Guide + +### For New Developers + +1. **Read First**: + - `project_overview.md` - Understand what the project is + - `common_development_workflows.md` - Learn essential commands and workflows + +2. **Before Making Changes**: + - `code_style_and_conventions.md` - Learn coding standards + - `task_completion_checklist.md` - Know what to check before committing + +3. **When Working on Code**: + - `architecture_and_design_patterns.md` - Understand design patterns + +4. **When Writing Tests**: + - `testing_infrastructure.md` - Learn testing approach + +### For System Setup + +**Prerequisites** (macOS): +```bash +# Java 11 (required) +/usr/libexec/java_home -V +export JAVA_HOME=$(/usr/libexec/java_home -v 11) + +# Maven +brew install maven + +# Node.js 18.20.8 (for Hubble) +nvm install 18.20.8 +nvm use 18.20.8 +npm install -g yarn + +# Python 3 (for Hubble build) +brew install python3 +pip3 install -r hugegraph-hubble/hubble-dist/assembly/travis/requirements.txt +``` + +**Build Entire Project**: +```bash +mvn clean install -DskipTests -Dmaven.javadoc.skip=true -ntp +``` + +**Run Tests**: +```bash +# Client tests +cd hugegraph-client +mvn test -Dtest=UnitTestSuite -ntp + +# Loader tests +cd hugegraph-loader +mvn test -P unit -ntp + +# Hubble tests +cd hugegraph-hubble/hubble-fe +yarn test +``` + +## Essential Commands Cheat Sheet + +### Build Commands +```bash +# Full project +mvn clean install -DskipTests -Dmaven.javadoc.skip=true -ntp + +# Specific module (e.g., client) +mvn install -pl hugegraph-client -am -DskipTests -ntp + +# Hubble (requires dependencies built first) +mvn install -pl hugegraph-client,hugegraph-loader -am -DskipTests -ntp +cd hugegraph-hubble +mvn -e compile package -Dmaven.test.skip=true -ntp +``` + +### Testing Commands +```bash +# Client unit tests +cd hugegraph-client && mvn test -Dtest=UnitTestSuite -ntp + +# Loader tests +cd hugegraph-loader && mvn test -P unit -ntp + +# Single test +mvn test -Dtest=ClassName#methodName -ntp +``` + +### Code Quality +```bash +# Checkstyle +mvn checkstyle:check + +# License check +mvn apache-rat:check + +# EditorConfig validation +mvn editorconfig:check +``` + +### Git Commands (IMPORTANT: Always use --no-pager) +```bash +# View history +git --no-pager log --oneline -10 + +# View changes +git --no-pager diff HEAD~1 +``` + +**See `common_development_workflows.md` for complete command reference** + +## Key Project Facts + +- **Language**: Java 8 (main), Go, TypeScript +- **Build Tool**: Maven 3.x +- **Test Framework**: JUnit 4 + Mockito +- **Frontend**: React + TypeScript + MobX (Node.js 18.20.8) +- **Backend**: Spring Boot +- **Version**: 1.7.0 +- **License**: Apache 2.0 +- **Repository**: https://github.com/apache/hugegraph-toolchain + +## Common Pitfalls to Avoid + +1. ❌ **DON'T** use `git log` without `--no-pager` flag +2. ❌ **DON'T** commit without running checkstyle and tests +3. ❌ **DON'T** use star imports (`import org.apache.*`) +4. ❌ **DON'T** use `System.out.println` (use logger instead) +5. ❌ **DON'T** forget Apache 2.0 license headers +6. ❌ **DON'T** use tabs (use 4 spaces for Java, 2 for frontend) +7. ❌ **DON'T** exceed 100 character line length +8. ❌ **DON'T** commit code that fails CI checks + +## Getting Help + +- **Documentation**: https://hugegraph.apache.org/docs/ +- **Issues**: https://github.com/apache/hugegraph-toolchain/issues +- **Mailing List**: [email protected] +- **Memory Files**: Check `.serena/memories/` directory + +## Memory Statistics + +- **Total Memory Files**: 7 (including this index) +- **Total Lines**: ~2,900+ +- **Total Size**: ~85KB +- **Coverage Areas**: + - Project overview and structure + - Code style and conventions + - Architecture and design patterns + - Testing infrastructure + - Development workflows + - Task completion checklists + +## Last Updated + +Onboarding completed: 2025-11-05 + +--- + +**Note**: All memories are stored in `.serena/memories/` directory and can be read using Serena MCP tools. \ No newline at end of file diff --git a/.serena/memories/architecture_and_design_patterns.md b/.serena/memories/architecture_and_design_patterns.md new file mode 100644 index 00000000..6a397912 --- /dev/null +++ b/.serena/memories/architecture_and_design_patterns.md @@ -0,0 +1,572 @@ +# Architecture and Design Patterns - HugeGraph Toolchain + +## Overall Architecture + +### System Context +``` +┌─────────────────────────────────────────────────────────────┐ +│ HugeGraph Ecosystem │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ ┌──────────────┐ ┌──────────────────────────────┐ │ +│ │ HugeGraph │◄─────┤ HugeGraph Toolchain │ │ +│ │ Server │ │ │ │ +│ │ (Core) │ │ ┌─────────────────────────┐ │ │ +│ └──────────────┘ │ │ hugegraph-client │ │ │ +│ ▲ │ │ (RESTful API wrapper) │ │ │ +│ │ │ └──────────┬──────────────┘ │ │ +│ │ │ │ │ │ +│ │ │ ┌──────▼──────────┐ │ │ +│ REST API │ │ ┌──────────────┐ │ │ +│ (HTTP/HTTPS) │ │ │ loader │ │ │ +│ │ │ │ │ tools │ │ │ +│ │ │ │ │ hubble-be │ │ │ +│ │ │ │ │ spark │ │ │ +│ └──────────────┼──────┘ │ client-go │ │ │ +│ │ └──────────────┘ │ │ +│ │ │ │ +│ ┌──────────────┐ │ ┌──────────────────┐ │ │ +│ │ External │────►│ │ hubble-fe │ │ │ +│ │ Data Sources │ │ │ (React Web UI) │ │ │ +│ │ (CSV/HDFS/ │ │ └──────────────────┘ │ │ +│ │ JDBC/Kafka) │ │ │ │ +│ └──────────────┘ └──────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Module-Specific Architectures + +## 1. hugegraph-client Architecture + +### Layered Architecture +``` +┌─────────────────────────────────────────────┐ +│ Application Layer │ +│ (User code using HugeGraph client) │ +└───────────────┬─────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────┐ +│ Manager Layer │ +│ ┌──────────────┐ ┌──────────────┐ │ +│ │SchemaManager │ │GraphManager │ │ +│ ├──────────────┤ ├──────────────┤ │ +│ │TraverserMgr │ │JobManager │ │ +│ ├──────────────┤ ├──────────────┤ │ +│ │TaskManager │ │AuthManager │ │ +│ └──────────────┘ └──────────────┘ │ +└───────────────┬─────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────┐ +│ API Layer │ +│ ┌──────────────┐ ┌──────────────┐ │ +│ │ VertexAPI │ │ EdgeAPI │ │ +│ ├──────────────┤ ├──────────────┤ │ +│ │ SchemaAPI │ │ GremlinAPI │ │ +│ ├──────────────┤ ├──────────────┤ │ +│ │ TraverserAPI │ │ JobAPI │ │ +│ └──────────────┘ └──────────────┘ │ +└───────────────┬─────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────┐ +│ REST Client Layer │ +│ ┌──────────────────────────────┐ │ +│ │ RestClient │ │ +│ │ - HTTP connection pool │ │ +│ │ - Request/Response handling │ │ +│ │ - Authentication │ │ +│ │ - Error handling │ │ +│ └──────────────────────────────┘ │ +└───────────────┬─────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────┐ +│ HugeGraph Server (REST API) │ +└─────────────────────────────────────────────┘ +``` + +### Key Components + +#### 1. RestClient (Core) +**Responsibilities**: +- HTTP/HTTPS connection management +- Request serialization (Java objects → JSON) +- Response deserialization (JSON → Java objects) +- Authentication (Basic Auth, Token) +- Error handling and retry logic +- Connection pooling + +**Key Methods**: +```java +// Generic request methods +public <T> T get(String path, Class<T> responseClass) +public <T> T post(String path, Object request, Class<T> responseClass) +public <T> T put(String path, Object request, Class<T> responseClass) +public <T> T delete(String path, Class<T> responseClass) + +// With custom headers +public <T> T request(HttpMethod method, String path, Object request, + Map<String, String> headers, Class<T> responseClass) +``` + +#### 2. Manager Pattern +Each manager handles a specific domain: + +**SchemaManager**: Schema CRUD operations +```java +// Get manager +SchemaManager schema = hugegraph.schema(); + +// Operations +schema.propertyKey("name").asText().create(); +schema.vertexLabel("person").properties("name", "age").create(); +schema.edgeLabel("knows").link("person", "person").create(); +schema.indexLabel("personByName").onV("person").by("name").create(); +``` + +**GraphManager**: Vertex/Edge operations +```java +GraphManager graph = hugegraph.graph(); + +// CRUD +Vertex v = graph.addVertex("person", "name", "Alice", "age", 30); +Edge e = v.addEdge("knows", target, "date", "2023-01-01"); +Vertex retrieved = graph.getVertex(id); +graph.removeVertex(id); +``` + +**TraverserManager**: Graph algorithms +```java +TraverserManager traverser = hugegraph.traverser(); + +// Algorithms +Path shortestPath = traverser.shortestPath(sourceId, targetId, direction, maxDepth); +List<Vertex> kHop = traverser.kHop(sourceId, direction, depth); +List<Path> kShortestPaths = traverser.kShortestPaths(sourceId, targetId, k); +``` + +#### 3. Builder Pattern (Fluent API) +```java +// PropertyKey builder +PropertyKey age = schema.propertyKey("age") + .asInt() + .valueSingle() // Single value (not set) + .ifNotExist() // Create only if not exists + .create(); + +// VertexLabel builder +VertexLabel person = schema.vertexLabel("person") + .properties("name", "age", "city") + .primaryKeys("name") // Required fields + .nullableKeys("city") // Optional fields + .ifNotExist() + .create(); + +// EdgeLabel builder +EdgeLabel knows = schema.edgeLabel("knows") + .sourceLabel("person") + .targetLabel("person") + .properties("date", "weight") + .frequency(Frequency.SINGLE) // One edge per (source,target) pair + .ifNotExist() + .create(); +``` + +### Serialization Layer +**Purpose**: Convert between Java objects and JSON + +**Key Classes**: +- `VertexSerializer`: Serialize/deserialize vertices +- `EdgeSerializer`: Serialize/deserialize edges +- `PathSerializer`: Serialize/deserialize paths +- `ResultDeserializer`: Generic result parsing + +## 2. hugegraph-loader Architecture + +### Pipeline Architecture +``` +┌──────────────────────────────────────────────────────────┐ +│ Data Loading Pipeline │ +└──────────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────┐ +│ Phase 1: Data Source Connection │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ Source Factory (based on SourceType) │ │ +│ │ - FileSource (CSV, JSON, TXT) │ │ +│ │ - HDFSSource (HDFS files) │ │ +│ │ - JDBCSource (MySQL, PostgreSQL, Oracle) │ │ +│ │ - KafkaSource (Kafka topics) │ │ +│ └─────────────────────────────────────────────────┘ │ +└──────────────────┬───────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────┐ +│ Phase 2: Data Reading & Parsing │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ Fetcher (source-specific) │ │ +│ │ - FileFetcher: Read file line-by-line │ │ +│ │ - JDBCFetcher: Execute SQL query │ │ +│ │ - KafkaFetcher: Consume messages │ │ +│ └────────────────┬────────────────────────────────┘ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ Reader (format-specific) │ │ +│ │ - CSVReader: Parse CSV records │ │ +│ │ - JSONReader: Parse JSON objects │ │ +│ │ - TextReader: Parse text lines │ │ +│ └────────────────┬────────────────────────────────┘ │ +└───────────────────┼──────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────┐ +│ Phase 3: Element Building │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ Mapping Config (struct.json) │ │ +│ │ - Field mappings: source → graph property │ │ +│ │ - ID generation strategies │ │ +│ │ - Value conversions │ │ +│ └────────────────┬────────────────────────────────┘ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ ElementBuilder │ │ +│ │ - Build Vertex from row/record │ │ +│ │ - Build Edge from row/record │ │ +│ │ - Apply transformations │ │ +│ └────────────────┬────────────────────────────────┘ │ +└───────────────────┼──────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────┐ +│ Phase 4: Batch Insertion │ +│ ┌─────────────────────────────────────────────────┐ │ +│ │ InsertTask (multi-threaded) │ │ +│ │ - Buffer elements (batch size: 500 default) │ │ +│ │ - Bulk insert via hugegraph-client API │ │ +│ │ - Error handling & retry logic │ │ +│ └────────────────┬────────────────────────────────┘ │ +└───────────────────┼──────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────┐ +│ HugeGraph Server │ +└──────────────────────────────────────────────────────────┘ +``` + +### Key Design Patterns + +#### 1. Factory Pattern (Source Creation) +```java +public interface Source { + Fetcher createFetcher(); +} + +// Factory method +public static Source create(SourceType type, SourceConfig config) { + switch (type) { + case FILE: + return new FileSource(config); + case HDFS: + return new HDFSSource(config); + case JDBC: + return new JDBCSource(config); + case KAFKA: + return new KafkaSource(config); + default: + throw new IllegalArgumentException(); + } +} +``` + +#### 2. Strategy Pattern (ID Generation) +Different strategies for generating vertex/edge IDs: +- `PrimaryKeyIdStrategy`: Use primary key fields +- `CustomIdStrategy`: User-defined ID field +- `AutomaticIdStrategy`: Server-generated IDs + +#### 3. Template Method Pattern (Parsing) +```java +abstract class AbstractReader { + // Template method + public final List<Record> read() { + open(); + List<Record> records = parseRecords(); + close(); + return records; + } + + protected abstract void open(); + protected abstract List<Record> parseRecords(); + protected abstract void close(); +} +``` + +## 3. hugegraph-hubble Architecture + +### Frontend Architecture (React + MobX) +``` +┌──────────────────────────────────────────────────────────┐ +│ Hubble Frontend │ +├──────────────────────────────────────────────────────────┤ +│ │ +│ ┌────────────────────────────────────────────────┐ │ +│ │ Presentation Layer (React Components) │ │ +│ │ │ │ +│ │ ┌──────────────┐ ┌──────────────┐ │ │ +│ │ │ GraphManager │ │ DataAnalyze │ │ │ +│ │ │ Pages │ │ Pages │ │ │ +│ │ └──────────────┘ └──────────────┘ │ │ +│ │ ┌──────────────┐ ┌──────────────┐ │ │ +│ │ │ SchemaManage │ │ DataImport │ │ │ +│ │ │ Pages │ │ Pages │ │ │ +│ │ └──────────────┘ └──────────────┘ │ │ +│ └────────────────┬───────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌────────────────────────────────────────────────┐ │ +│ │ State Management Layer (MobX Stores) │ │ +│ │ │ │ +│ │ ┌──────────────────────┐ │ │ +│ │ │ GraphManagementStore │ (graph connections) │ │ +│ │ ├──────────────────────┤ │ │ +│ │ │ DataAnalyzeStore │ (query & analysis) │ │ +│ │ ├──────────────────────┤ │ │ +│ │ │ SchemaStore │ (schema operations) │ │ +│ │ ├──────────────────────┤ │ │ +│ │ │ DataImportStore │ (data loading) │ │ +│ │ └──────────────────────┘ │ │ +│ └────────────────┬───────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌────────────────────────────────────────────────┐ │ +│ │ API Service Layer │ │ +│ │ (HTTP requests to backend) │ │ +│ └────────────────┬───────────────────────────────┘ │ +└───────────────────┼──────────────────────────────────────┘ + │ + ▼ HTTP/REST API +┌──────────────────────────────────────────────────────────┐ +│ Hubble Backend (Spring Boot) │ +├──────────────────────────────────────────────────────────┤ +│ │ +│ ┌────────────────────────────────────────────────┐ │ +│ │ Controller Layer │ │ +│ │ ┌──────────────┐ ┌──────────────┐ │ │ +│ │ │ GraphConn │ │ Schema │ │ │ +│ │ │ Controller │ │ Controller │ │ │ +│ │ ├──────────────┤ ├──────────────┤ │ │ +│ │ │ Query │ │ DataImport │ │ │ +│ │ │ Controller │ │ Controller │ │ │ +│ │ └──────────────┘ └──────────────┘ │ │ +│ └────────────────┬───────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌────────────────────────────────────────────────┐ │ +│ │ Service Layer (Business Logic) │ │ +│ │ ┌──────────────┐ ┌──────────────┐ │ │ +│ │ │ GraphConn │ │ Schema │ │ │ +│ │ │ Service │ │ Service │ │ │ +│ │ ├──────────────┤ ├──────────────┤ │ │ +│ │ │ Query │ │ DataImport │ │ │ +│ │ │ Service │ │ Service │ │ │ +│ │ └──────────────┘ └──────────────┘ │ │ +│ └────────────────┬───────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌────────────────────────────────────────────────┐ │ +│ │ Repository Layer (Data Persistence) │ │ +│ │ - File-based storage (local disk) │ │ +│ │ - Graph connection metadata │ │ +│ └────────────────┬───────────────────────────────┘ │ +└───────────────────┼──────────────────────────────────────┘ + │ + ▼ REST API (via hugegraph-client) +┌──────────────────────────────────────────────────────────┐ +│ HugeGraph Server │ +└──────────────────────────────────────────────────────────┘ +``` + +### Key Design Patterns (Hubble) + +#### 1. Observer Pattern (MobX) +```typescript +// Store definition +class GraphManagementStore { + @observable currentGraph: GraphConnection | null = null; + @observable graphList: GraphConnection[] = []; + + @action + async loadGraphs() { + const response = await api.getGraphs(); + this.graphList = response.data; + } + + @computed + get activeGraphName() { + return this.currentGraph?.name || 'None'; + } +} + +// Component observing store +@observer +class GraphSelector extends React.Component { + render() { + const { graphStore } = this.props; + return <div>{graphStore.activeGraphName}</div>; + } +} +``` + +#### 2. Repository Pattern (Backend) +```java +// Entity +@Entity +public class GraphConnection { + @Id + private Long id; + private String name; + private String host; + private Integer port; + // ... +} + +// Repository interface +public interface GraphConnectionRepository { + GraphConnection save(GraphConnection connection); + GraphConnection findById(Long id); + List<GraphConnection> findAll(); + void deleteById(Long id); +} + +// Service using repository +@Service +public class GraphConnectionService { + @Autowired + private GraphConnectionRepository repository; + + public GraphConnection create(GraphConnection connection) { + return repository.save(connection); + } +} +``` + +## 4. hugegraph-tools Architecture + +### Command Pattern +``` +┌─────────────────────────────────────────┐ +│ CLI Entry Point │ +└───────────────┬─────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────┐ +│ Command Router │ +│ (parse args, dispatch command) │ +└───────────────┬─────────────────────────┘ + │ + ├─► backup (GraphBackupCommand) + ├─► restore (GraphRestoreCommand) + ├─► deploy (DeployCommand) + ├─► graph-list (GraphListCommand) + ├─► graph-clear (GraphClearCommand) + └─► graph-mode-set (GraphModeCommand) +``` + +**Command Interface**: +```java +public interface Command { + String name(); + void execute(String[] args); +} + +// Example implementation +public class BackupCommand implements Command { + public String name() { return "backup"; } + + public void execute(String[] args) { + // Parse options + String graph = parseGraphOption(args); + String directory = parseDirectoryOption(args); + + // Execute backup via client API + HugeClient client = createClient(); + client.graphs().backup(graph, directory); + } +} +``` + +## Cross-Cutting Concerns + +### Error Handling Strategy + +**Client/Loader/Tools**: +```java +try { + // Operation +} catch (ServerException e) { + // Server-side error (4xx, 5xx) + log.error("Server error: {}", e.getMessage()); + throw new LoaderException("Failed to load data", e); +} catch (ClientException e) { + // Client-side error (network, serialization) + log.error("Client error: {}", e.getMessage()); + throw new LoaderException("Client communication failed", e); +} +``` + +### Logging Strategy + +**All modules use Log4j2**: +```java +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; + +public class MyClass { + private static final Logger LOG = LogManager.getLogger(MyClass.class); + + public void method() { + LOG.debug("Debug message"); + LOG.info("Info message"); + LOG.warn("Warning message"); + LOG.error("Error message", exception); + } +} +``` + +### Configuration Management + +**Loader** uses JSON structure files: +```json +{ + "version": "2.0", + "vertices": [ + { + "label": "person", + "input": { + "type": "file", + "path": "data/persons.csv", + "format": "CSV" + }, + "mapping": { + "id": "id", + "properties": { + "name": "name", + "age": "age" + } + } + } + ] +} +``` + +**Hubble** uses Spring properties: +```properties +server.port=8088 +spring.application.name=hugegraph-hubble +graph.server.host=localhost +graph.server.port=8080 +``` \ No newline at end of file diff --git a/.serena/memories/code_style_and_conventions.md b/.serena/memories/code_style_and_conventions.md new file mode 100644 index 00000000..6f296a7e --- /dev/null +++ b/.serena/memories/code_style_and_conventions.md @@ -0,0 +1,312 @@ +# Code Style and Conventions for HugeGraph Toolchain + +## General Principles +- **Language**: English for all code, comments, and documentation +- **License**: All source files require Apache 2.0 license headers +- **Encoding**: UTF-8 for all files +- **Line Endings**: LF (Unix-style) +- **Final Newline**: Always insert final newline + +## Java Code Style + +### Basic Formatting +- **Indentation**: 4 spaces (NO TABS) +- **Continuation Indent**: 8 spaces +- **Line Length**: Maximum 100 characters +- **Line Wrapping**: Enabled for long lines +- **Blank Lines**: + - Keep max 1 blank line in declarations + - Keep max 1 blank line in code + - 1 blank line around classes + - 1 blank line after class header + +### Naming Conventions +- **Package Names**: `^[a-z]+(\.[a-z][a-z0-9]*)*$` + - Example: `org.apache.hugegraph.client` +- **Class Names**: `PascalCase` (e.g., `RestClient`, `GraphManager`) +- **Type Parameters**: `^[A-Z][a-zA-Z0-9]*$` (e.g., `T`, `K`, `V`) +- **Constants**: `UPPER_SNAKE_CASE` (e.g., `DEFAULT_TIMEOUT`, `MAX_RETRIES`) +- **Variables**: `camelCase` starting with lowercase (e.g., `vertexId`, `edgeLabel`) +- **Methods**: `camelCase` starting with lowercase, must have 2+ chars + - Pattern: `^[a-z][a-z0-9][a-zA-Z0-9_]*$` + - Example: `getVertexById()`, `createEdge()` +- **Parameters**: `camelCase` (e.g., `userId`, `timeout`) + +### Import Rules +- NO star imports (`import org.apache.*` forbidden) +- Remove unused imports +- Remove redundant imports +- Import order (configured in .editorconfig): + 1. Static imports + 2. `java.**` + 3. `javax.**` + 4. `org.**` + 5. `com.**` + 6. All others + +### Prohibited Imports (Checkstyle) +- `java.util.logging.Logging` +- `sun.misc.BASE64Encoder/Decoder` +- Shaded/internal packages from Hadoop, HBase, Netty, etc. +- `org.codehaus.jackson` (use `com.fasterxml.jackson` instead) +- `org.jetbrains.annotations` + +### Code Structure +- **Braces**: + - Always use braces for if/while/for (multi-line) + - `do-while` always requires braces + - Opening brace on same line (K&R style) +- **Whitespace**: + - No whitespace before: `,`, `;`, `.`, post-increment/decrement + - Whitespace around operators: `=`, `+`, `-`, `*`, `/`, etc. + - Proper padding in parentheses +- **Empty Blocks**: Only `{}` allowed (not `{ }`) + +### Java-Specific Rules +- **Array Style**: `String[] args` (NOT `String args[]`) +- **Generic Whitespace**: Follow standard Java conventions +- **Equals/HashCode**: Must implement both or neither +- **Switch Statement**: Must have `default` case +- **Finalize**: No finalizers allowed +- **System.out.println**: PROHIBITED in source code (use logger) + +### Comments and JavaDoc +- **Line Comments**: Not at first column, use proper indentation +- **JavaDoc**: + - Add `<p>` tag on empty lines + - Do not wrap if one line + - Comment indentation: 4 spaces + +### Annotations +- Each annotation on separate line (for methods/constructors) +- Single parameterless annotation allowed on same line (other contexts) + +## Maven POM Style + +### XML Formatting +- **Indentation**: 4 spaces +- **Line Length**: Maximum 120 characters +- **Text Wrap**: Off for XML +- **Empty Tags**: Space inside (`<tag />`) + +### POM Organization +```xml +<project> + <modelVersion/> + <parent/> + <groupId/> + <artifactId/> + <version/> + <packaging/> + + <name/> + <description/> + <url/> + + <licenses/> + <developers/> + <scm/> + + <properties/> + <dependencyManagement/> + <dependencies/> + + <build/> + <profiles/> +</project> +``` + +## Frontend Code Style (Hubble) + +### TypeScript/JavaScript +- **Formatter**: Prettier (configured in `.prettierrc`) +- **Linter**: ESLint/TSLint +- **Naming**: + - Components: PascalCase (`GraphViewer.tsx`) + - Files: kebab-case or PascalCase + - Variables: camelCase + +### CSS/Less +- **Linter**: Stylelint (configured in `.stylelintrc`) +- **Naming**: kebab-case for class names +- **Indentation**: 2 spaces + +### Pre-commit Hooks +- **Husky**: Runs on git commit +- **lint-staged**: Auto-format staged files +- Configuration: `.lintstagedrc.yml` + +## Go Code Style (client-go) + +### Standard Go Conventions +- Follow official Go formatting (`gofmt`) +- Use `go vet` for static analysis +- Run tests with race detector: `go test -race` + +### Naming +- Exported names: Start with uppercase +- Unexported names: Start with lowercase +- Package names: Short, lowercase, single word + +## Design Patterns and Architecture + +### hugegraph-client Patterns + +#### Manager Pattern +Separate managers for different API domains: +```java +// Schema operations +SchemaManager schemaManager = hugegraph.schema(); + +// Graph operations +GraphManager graphManager = hugegraph.graph(); + +// Traversal algorithms +TraverserManager traverser = hugegraph.traverser(); + +// Async jobs +JobManager jobManager = hugegraph.job(); + +// Authentication +AuthManager authManager = hugegraph.auth(); +``` + +#### Builder Pattern +Fluent API for constructing schema elements: +```java +VertexLabel person = schema.vertexLabel("person") + .properties("name", "age", "city") + .primaryKeys("name") + .nullableKeys("city") + .create(); +``` + +#### RESTful Wrapper +- `RestClient`: Base HTTP communication layer +- All API classes extend or use `RestClient` +- Consistent error handling with custom exceptions + +### hugegraph-loader Patterns + +#### Pipeline Architecture +``` +Source → Parser → Transformer → Builder → BatchInserter → HugeGraph +``` + +- **ParseTask**: Read and parse data from sources +- **InsertTask**: Batch insert into HugeGraph +- **ElementBuilder**: Construct vertices/edges from raw data + +#### Source Abstraction +Unified interface for different data sources: +```java +interface Source { + Fetcher createFetcher(); +} + +// Implementations: +- FileSource (CSV, JSON, TXT) +- HDFSSource +- JDBCSource +- KafkaSource +``` + +### hugegraph-hubble Patterns + +#### Frontend Architecture +- **Store Pattern**: MobX stores for state management + - `GraphManagementStore`: Graph connection management + - `DataAnalyzeStore`: Query and analysis state + - `SchemaStore`: Schema management state +- **Component Hierarchy**: Container → Component → Sub-component + +#### Backend Architecture (Spring Boot) +- **Controller**: HTTP request handling +- **Service**: Business logic layer +- **Repository**: Data persistence (local file-based) +- **DTO/Entity**: Data transfer and domain objects + +## File Organization + +### Java Package Structure +``` +org.apache.hugegraph/ +├── api/ # RESTful API implementations +├── client/ # Client interfaces and implementations +├── driver/ # Driver layer +├── structure/ # Graph structure elements (Vertex, Edge, etc.) +├── exception/ # Custom exceptions +├── serializer/ # JSON serialization/deserialization +├── util/ # Utility classes +└── version/ # Version information +``` + +### Test Organization +``` +src/test/java/ +├── unit/ # Unit tests (no external dependencies) +├── api/ # API integration tests (require server) +└── functional/ # End-to-end functional tests +``` + +## Version Control Practices + +### Commit Messages +- Format: `type(scope): subject` +- Types: `feat`, `fix`, `docs`, `style`, `refactor`, `test`, `chore` +- Examples: + - `feat(client): add batch vertex query API` + - `fix(loader): handle empty CSV files correctly` + - `chore(hubble): update Node.js version to 18.20.8` + +### Branch Naming +- `master`: Main development branch +- `release-*`: Release branches +- `feature/*`: Feature branches +- `fix/*`: Bug fix branches + +## Testing Conventions + +### Test Class Naming +- Unit tests: `*Test.java` (e.g., `RestClientTest.java`) +- Test suites: `*TestSuite.java` (e.g., `UnitTestSuite.java`) + +### Test Method Naming +- Descriptive names: `testGetVertexById()`, `testCreateEdgeWithInvalidLabel()` +- Use `@Test` annotation (JUnit 4) + +### Test Organization +- Group tests into suites: + - `UnitTestSuite`: No external dependencies + - `ApiTestSuite`: API integration tests + - `FuncTestSuite`: Functional/E2E tests + +## Documentation Standards + +### JavaDoc Requirements +- All public APIs must have JavaDoc +- Include `@param`, `@return`, `@throws` tags +- Example usage in class-level JavaDoc + +### README Structure +```markdown +# Module Name + +## Features +## Quick Start +## Usage +## Doc +## License +``` + +## Error Handling + +### Java Exceptions +- Use custom exceptions: `HugeException`, `ServerException`, `ClientException` +- Proper exception chaining with causes +- Meaningful error messages + +### Go Error Handling +- Return errors explicitly: `func() (result, error)` +- Handle errors at call site +- Wrap errors with context: `fmt.Errorf("context: %w", err)` \ No newline at end of file diff --git a/.serena/memories/common_development_workflows.md b/.serena/memories/common_development_workflows.md new file mode 100644 index 00000000..ccb12bd8 --- /dev/null +++ b/.serena/memories/common_development_workflows.md @@ -0,0 +1,658 @@ +# Common Development Workflows - HugeGraph Toolchain + +## Daily Development Workflows + +### 1. Starting a New Feature + +**Step 1: Create Feature Branch** +```bash +# Update master +git checkout master +git pull origin master + +# Create feature branch +git checkout -b feature/add-batch-query-api +``` + +**Step 2: Make Changes** +```bash +# Edit code in your IDE +# Follow code style guidelines (see code_style_and_conventions.md) +``` + +**Step 3: Local Testing** +```bash +# Run unit tests +cd hugegraph-client +mvn test -Dtest=UnitTestSuite -ntp + +# Run checkstyle +mvn checkstyle:check +``` + +**Step 4: Commit Changes** +```bash +git add . +git commit -m "feat(client): add batch query API for vertices" +``` + +**Step 5: Push and Create PR** +```bash +git push origin feature/add-batch-query-api +# Open PR on GitHub +``` + +### 2. Fixing a Bug + +**Step 1: Reproduce the Bug** +```bash +# Write a failing test first (TDD approach) +cd hugegraph-loader +vim src/test/java/org/apache/hugegraph/loader/test/functional/CSVLoadTest.java + +# Add test case +@Test +public void testHandleEmptyCSVFile() { + // Test that reproduces the bug +} + +# Run test - should fail +mvn test -Dtest=CSVLoadTest#testHandleEmptyCSVFile -ntp +``` + +**Step 2: Fix the Bug** +```bash +# Edit source code to fix the issue +vim src/main/java/org/apache/hugegraph/loader/reader/CSVReader.java +``` + +**Step 3: Verify Fix** +```bash +# Run test again - should pass +mvn test -Dtest=CSVLoadTest#testHandleEmptyCSVFile -ntp + +# Run all related tests +mvn test -P file +``` + +**Step 4: Commit with Issue Reference** +```bash +git add . +git commit -m "fix(loader): handle empty CSV files correctly + +Fixes #123 + +Previously, the loader would throw NullPointerException when +encountering empty CSV files. Now it gracefully skips empty files +and logs a warning." +``` + +### 3. Adding Tests for Existing Code + +**Step 1: Identify Coverage Gaps** +```bash +# Generate coverage report +mvn test jacoco:report + +# Open report +open target/site/jacoco/index.html + +# Find classes with low coverage +``` + +**Step 2: Write Tests** +```bash +# Create test class if doesn't exist +vim src/test/java/org/apache/hugegraph/client/RestClientTest.java +``` + +```java +public class RestClientTest { + @Test + public void testConnectionTimeout() { + // Test timeout handling + } + + @Test + public void testRetryOnNetworkError() { + // Test retry logic + } +} +``` + +**Step 3: Add to Test Suite** +```java +@RunWith(Suite.class) [email protected]({ + // ... existing tests + RestClientTest.class // Add new test +}) +public class UnitTestSuite {} +``` + +### 4. Refactoring Code + +**Step 1: Ensure Tests Pass** +```bash +# Run all tests before refactoring +mvn test +``` + +**Step 2: Make Changes Incrementally** +```bash +# Small, focused changes +# Run tests after each change +mvn test -Dtest=RelevantTestClass -ntp +``` + +**Step 3: Verify All Tests Still Pass** +```bash +# Run full test suite +mvn test + +# Check code style +mvn checkstyle:check +``` + +**Step 4: Commit** +```bash +git commit -m "refactor(client): extract common HTTP logic to base class + +No functional changes, just code organization improvement." +``` + +## Module-Specific Workflows + +### Working on hugegraph-client + +**Setup Development Environment** +```bash +# Build client only +mvn clean install -pl hugegraph-client -am -DskipTests -ntp + +# Start HugeGraph server for integration tests +# Option 1: Docker +docker run -d --name hugegraph -p 8080:8080 hugegraph/hugegraph + +# Option 2: From source +./hugegraph-client/assembly/travis/install-hugegraph-from-source.sh b7998c1 +``` + +**Development Cycle** +```bash +# 1. Edit code +vim src/main/java/org/apache/hugegraph/api/VertexAPI.java + +# 2. Quick compile check +mvn compile -pl hugegraph-client -ntp + +# 3. Run relevant tests +mvn test -Dtest=VertexApiTest -ntp + +# 4. Full test suite (before commit) +mvn test -Dtest=UnitTestSuite -ntp +mvn test -Dtest=ApiTestSuite +``` + +### Working on hugegraph-loader + +**Setup Development Environment** +```bash +# Build loader with dependencies +mvn install -pl hugegraph-client,hugegraph-loader -am -DskipTests -ntp + +# Setup test environment +cd hugegraph-loader/assembly/travis +./install-hugegraph-from-source.sh 5b3d295 + +# For JDBC tests +./install-mysql.sh load_test root + +# For HDFS tests +./install-hadoop.sh +``` + +**Testing New Data Source** +```bash +# 1. Create test data files +mkdir -p src/test/resources/my-test +echo "id,name,age" > src/test/resources/my-test/data.csv +echo "1,Alice,30" >> src/test/resources/my-test/data.csv + +# 2. Create mapping config +vim src/test/resources/struct/my-test.json + +# 3. Write test +vim src/test/java/org/apache/hugegraph/loader/test/functional/MySourceTest.java + +# 4. Run test +mvn test -Dtest=MySourceTest -ntp +``` + +### Working on hugegraph-hubble + +**Setup Development Environment** +```bash +# Ensure Node.js 18.20.8 +node -v # Must be 18.20.8 + +# Install dependencies +npm install -g yarn +cd hugegraph-hubble/hubble-fe +yarn install + +# Install Python requirements (for build) +pip install -r ../hubble-dist/assembly/travis/requirements.txt +``` + +**Frontend Development Cycle** +```bash +cd hugegraph-hubble/hubble-fe + +# 1. Edit code +vim src/components/GraphViewer.tsx + +# 2. Run linter +yarn lint + +# 3. Auto-fix formatting +npx prettier --write src/components/GraphViewer.tsx + +# 4. Run tests +yarn test GraphViewer.test.tsx + +# 5. Start dev server (optional) +yarn start +``` + +**Backend Development Cycle** +```bash +cd hugegraph-hubble/hubble-be + +# 1. Edit code +vim src/main/java/org/apache/hugegraph/hubble/controller/GraphController.java + +# 2. Run tests +mvn test -Dtest=GraphControllerTest -ntp + +# 3. Build and run +mvn spring-boot:run +``` + +**Full Hubble Build** +```bash +# Build dependencies first +mvn install -pl hugegraph-client,hugegraph-loader -am -DskipTests -ntp + +# Build hubble +cd hugegraph-hubble +mvn -e compile package -Dmaven.test.skip=true -ntp + +# Start hubble +cd apache-hugegraph-hubble-incubating-*/ +bin/start-hubble.sh -d + +# Check logs +tail -f logs/hugegraph-hubble.log + +# Access UI +open http://localhost:8088 +``` + +### Working on hugegraph-client-go + +**Setup Development Environment** +```bash +cd hugegraph-client-go + +# Download dependencies +make prepare + +# Setup Go environment +go env -w GO111MODULE=on +``` + +**Development Cycle** +```bash +# 1. Edit code +vim client.go + +# 2. Format code +go fmt ./... + +# 3. Vet code +go vet ./... + +# 4. Run tests with race detector +make test + +# 5. Build binary +make compile + +# 6. Run binary +./hugegraph-client-go +``` + +## Troubleshooting Common Issues + +### Issue: Maven Build Fails with Dependency Errors + +**Solution 1: Clear Local Cache** +```bash +rm -rf ~/.m2/repository/org/apache/hugegraph +mvn clean install -U +``` + +**Solution 2: Use Stage Repository** +```bash +mvn clean install -P stage +``` + +### Issue: Tests Fail with "Connection Refused" + +**Problem**: HugeGraph server not running + +**Solution**: +```bash +# Check if server is running +curl http://localhost:8080/versions + +# If not, start it +cd apache-hugegraph-* +bin/start-hugegraph.sh + +# Wait for startup (check logs) +tail -f logs/hugegraph-server.log +``` + +### Issue: Checkstyle Violations + +**Common Fixes**: +```bash +# Line too long (max 100 chars) +# Solution: Break into multiple lines + +# Star imports +# Solution: Expand imports in IDE (IntelliJ: Ctrl+Alt+O) + +# Wrong indentation +# Solution: Use 4 spaces, not tabs +# IntelliJ: Settings → Editor → Code Style → Java → Indent: 4 + +# Missing whitespace +# Solution: Add space around operators +# Before: if(x==5) +# After: if (x == 5) +``` + +### Issue: Frontend Build Fails + +**Solution 1: Node.js Version** +```bash +# Check version +node -v + +# If wrong version, use nvm +nvm install 18.20.8 +nvm use 18.20.8 +``` + +**Solution 2: Clear Cache** +```bash +cd hugegraph-hubble/hubble-fe +rm -rf node_modules yarn.lock +yarn install +``` + +**Solution 3: Memory Limit** +```bash +# Increase Node.js memory +export NODE_OPTIONS="--max-old-space-size=4096" +mvn clean package +``` + +### Issue: HDFS Tests Fail + +**Solution**: Check Hadoop setup +```bash +# Verify Hadoop is running +jps | grep -E 'NameNode|DataNode' + +# Check HDFS status +hadoop fs -ls / + +# If issues, reinstall +./assembly/travis/install-hadoop.sh +``` + +### Issue: JDBC Tests Fail + +**Solution**: Check MySQL +```bash +# Check MySQL is running +mysql -u root -proot -e "SHOW DATABASES;" + +# Verify test database exists +mysql -u root -proot -e "USE load_test; SHOW TABLES;" + +# If issues, reinstall +./assembly/travis/install-mysql.sh load_test root +``` + +## Release Workflow + +### Preparing a Release + +**Step 1: Update Version** +```bash +# Update root pom.xml +vim pom.xml +# Change <revision>1.7.0</revision> to <revision>1.8.0</revision> + +# Update frontend version +vim hugegraph-hubble/hubble-fe/package.json +# Change "version": "1.7.0" to "version": "1.8.0" +``` + +**Step 2: Update CHANGELOG** +```bash +vim CHANGELOG.md +# Add release notes: +# ## [1.8.0] - 2025-02-01 +# ### Added +# - New batch query API +# ### Fixed +# - CSV loading bug +``` + +**Step 3: Run Full Test Suite** +```bash +# Run all tests +mvn clean verify + +# Run integration tests +cd hugegraph-client && mvn test -Dtest=ApiTestSuite +cd hugegraph-loader && mvn test -P file,hdfs,jdbc +``` + +**Step 4: Build Release Artifacts** +```bash +# Build with Apache release profile +mvn clean package -P apache-release -DskipTests + +# Artifacts in hugegraph-dist/target/ +ls hugegraph-dist/target/*.tar.gz +``` + +**Step 5: Create Release Tag** +```bash +git tag -a v1.8.0 -m "Release version 1.8.0" +git push origin v1.8.0 +``` + +## Useful Development Commands + +### Quick Checks +```bash +# Check what you've changed +git --no-pager diff +git --no-pager diff --staged + +# Check recent commits +git --no-pager log --oneline -5 + +# Find files by name +find . -name "*Test.java" -type f + +# Search in code +grep -r "RestClient" --include="*.java" . +``` + +### Clean Everything +```bash +# Clean Maven build +mvn clean + +# Deep clean +find . -name target -type d -exec rm -rf {} + +find . -name .flattened-pom.xml -delete + +# Clean frontend +cd hugegraph-hubble/hubble-fe +rm -rf node_modules build + +# Clean Go +cd hugegraph-client-go +make clean +``` + +### Performance Profiling +```bash +# Maven build with timing +mvn clean install -Dorg.slf4j.simpleLogger.showDateTime=true + +# Java heap dump on OutOfMemoryError +export MAVEN_OPTS="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp" + +# Go benchmarks +cd hugegraph-client-go +go test -bench=. -benchmem +``` + +## Git Hooks + +### Pre-commit Hook (Optional) +```bash +vim .git/hooks/pre-commit +``` + +```bash +#!/bin/bash +# Pre-commit hook for HugeGraph Toolchain + +# Run checkstyle +echo "Running checkstyle..." +mvn checkstyle:check -q +if [ $? -ne 0 ]; then + echo "Checkstyle failed. Please fix violations." + exit 1 +fi + +# Run license check +echo "Checking licenses..." +mvn apache-rat:check -q +if [ $? -ne 0 ]; then + echo "License check failed. Please add Apache 2.0 headers." + exit 1 +fi + +echo "Pre-commit checks passed." +exit 0 +``` + +```bash +chmod +x .git/hooks/pre-commit +``` + +## IDE Configuration + +### IntelliJ IDEA Setup + +**Import Project**: +1. File → Open → Select `pom.xml` +2. Import as Maven project +3. Wait for dependency resolution + +**Configure Code Style**: +1. Settings → Editor → Code Style → Java +2. Import Scheme → IntelliJ IDEA code style XML +3. Load from: `.editorconfig` + +**Configure Checkstyle Plugin**: +1. Install Checkstyle-IDEA plugin +2. Settings → Tools → Checkstyle +3. Add configuration file: `tools/checkstyle.xml` + +**Run Configurations**: +```xml +<!-- Client Tests --> +<configuration name="Client-UnitTests" type="JUnit"> + <module name="hugegraph-client" /> + <option name="TEST_OBJECT" value="class" /> + <option name="MAIN_CLASS_NAME" value="org.apache.hugegraph.unit.UnitTestSuite" /> +</configuration> + +<!-- Loader Tests --> +<configuration name="Loader-FileTests" type="JUnit"> + <module name="hugegraph-loader" /> + <option name="VM_PARAMETERS" value="-Pfile" /> +</configuration> +``` + +### VS Code Setup + +**Extensions**: +- Java Extension Pack +- Prettier (for Hubble frontend) +- ESLint +- Go (for client-go) + +**Settings** (`.vscode/settings.json`): +```json +{ + "java.configuration.updateBuildConfiguration": "automatic", + "editor.formatOnSave": true, + "editor.tabSize": 4, + "editor.insertSpaces": true, + "[typescript]": { + "editor.tabSize": 2, + "editor.defaultFormatter": "esbenp.prettier-vscode" + }, + "[javascript]": { + "editor.tabSize": 2, + "editor.defaultFormatter": "esbenp.prettier-vscode" + } +} +``` + +## Continuous Learning + +### Understanding the Codebase + +**Start Here**: +1. Read module READMEs +2. Check `example/` directories for usage examples +3. Read test cases to understand expected behavior +4. Follow imports to understand dependencies + +**Key Files to Understand**: +- `hugegraph-client/src/main/java/org/apache/hugegraph/driver/HugeClient.java` +- `hugegraph-loader/src/main/java/org/apache/hugegraph/loader/HugeGraphLoader.java` +- `hugegraph-hubble/hubble-fe/src/stores/` (MobX stores) +- `hugegraph-client-go/client.go` + +### Documentation Resources +- Project Docs: https://hugegraph.apache.org/docs/ +- API Docs: https://hugegraph.apache.org/docs/clients/restful-api/ +- GitHub Issues: https://github.com/apache/hugegraph-toolchain/issues +- Mailing List: [email protected] \ No newline at end of file diff --git a/.serena/memories/project_overview.md b/.serena/memories/project_overview.md new file mode 100644 index 00000000..55e9a94b --- /dev/null +++ b/.serena/memories/project_overview.md @@ -0,0 +1,126 @@ +# HugeGraph Toolchain Project Overview + +## Project Purpose +Apache HugeGraph Toolchain is an integration project containing a series of utilities for [Apache HugeGraph](https://github.com/apache/hugegraph), a distributed graph database. The toolchain provides essential tools for data loading, management, visualization, and client access. + +## Version Information +- Current Version: 1.7.0 +- License: Apache 2.0 +- Repository: https://github.com/apache/hugegraph-toolchain +- Project Status: Apache Incubator + +## Main Modules (6 Total) + +### 1. hugegraph-client (Java) +**Purpose**: Java RESTful API client for HugeGraph +**Language**: Java 8 +**Key Features**: +- RESTful APIs for accessing graph vertex/edge/schema operations +- Gremlin query support +- Graph traversal algorithms (shortest path, k-hop, etc.) +- Authentication and authorization support + +### 2. hugegraph-loader +**Purpose**: Data loading utility from multiple sources into HugeGraph +**Language**: Java 8 +**Supported Sources**: +- File sources: CSV, JSON, TXT (local files) +- HDFS sources +- JDBC sources: MySQL, PostgreSQL, Oracle, SQL Server +- Kafka streaming sources + +### 3. hugegraph-hubble +**Purpose**: Web-based graph management and analysis dashboard +**Tech Stack**: +- **Backend**: Spring Boot (Java 8) +- **Frontend**: React + TypeScript + MobX (Node.js 18.20.8 required) +**Features**: +- Data loading interface +- Schema management +- Graph visualization +- Query builder (Gremlin and algorithm-based) + +### 4. hugegraph-tools +**Purpose**: Command-line tools for deployment and management +**Language**: Java 8 +**Features**: +- Deployment management +- Backup and restore operations +- Administrative tasks + +### 5. hugegraph-client-go (WIP) +**Purpose**: Go client library for HugeGraph +**Language**: Go +**Status**: Work In Progress + +### 6. hugegraph-spark-connector +**Purpose**: Spark connector for HugeGraph data I/O +**Language**: Java 8 + Scala 2.12 +**Spark Version**: 3.2.2 + +## Module Dependencies +``` +hugegraph-dist (assembly) + └── hugegraph-hubble + └── hugegraph-loader + └── hugegraph-client + └── hugegraph-common (external: v1.5.0) + +hugegraph-tools + └── hugegraph-client + +hugegraph-spark-connector + └── hugegraph-client + +hugegraph-client-go (independent) +``` + +## Technology Stack + +### Java Ecosystem +- **Java Version**: 1.8 (source/target) +- **Build Tool**: Maven 3.x +- **Test Framework**: JUnit 4 + Mockito 2.25.1 +- **Common Libraries**: + - Apache Commons (IO, Lang3, Compress, CLI, Text, Codec) + - Jackson 2.12.3 (JSON processing) + - Log4j2 2.18.0 (Logging) + - Netty 4.1.65.Final + - Lombok 1.18.8 + +### Frontend (Hubble) +- **Node.js**: 18.20.8 (required exact version) +- **Package Manager**: yarn (not npm) +- **Framework**: React +- **Language**: TypeScript +- **State Management**: MobX +- **Code Quality**: Prettier + Stylelint + Husky + +### Go (Client-Go) +- **Build Tool**: Makefile +- **Testing**: Built-in Go test with race detector + +## Key External Dependencies +- HugeGraph Server (required for testing) +- HugeGraph Common library v1.5.0 +- Spark 3.2.2 (for connector) +- Flink 1.13.5 (for stream processing) + +## Project Structure +``` +toolchain/ +├── hugegraph-client/ # Java client library +├── hugegraph-loader/ # Data loading tool +├── hugegraph-hubble/ # Web dashboard +│ ├── hubble-be/ # Backend (Spring Boot) +│ ├── hubble-fe/ # Frontend (React) +│ └── hubble-dist/ # Distribution files +├── hugegraph-tools/ # CLI tools +├── hugegraph-client-go/ # Go client (WIP) +├── hugegraph-spark-connector/# Spark connector +├── hugegraph-dist/ # Assembly module +├── assembly/ # Build descriptors +├── tools/ # Checkstyle, suppressions +├── .github/workflows/ # CI/CD pipelines +└── pom.xml # Root Maven config +``` \ No newline at end of file diff --git a/.serena/memories/task_completion_checklist.md b/.serena/memories/task_completion_checklist.md new file mode 100644 index 00000000..9be1d343 --- /dev/null +++ b/.serena/memories/task_completion_checklist.md @@ -0,0 +1,373 @@ +# Task Completion Checklist for HugeGraph Toolchain + +## Before Committing Code + +### 1. Code Quality Checks + +#### Java Modules (client, loader, tools, hubble-be, spark-connector) + +**A. Checkstyle Validation** +```bash +# Run checkstyle on affected modules +mvn checkstyle:check + +# Or for specific module +mvn checkstyle:check -pl hugegraph-client +``` +**Must Pass**: No checkstyle violations allowed + +**B. License Header Check** +```bash +# Verify all files have Apache 2.0 license headers +mvn apache-rat:check +``` +**Must Pass**: All source files must have proper license headers + +**C. EditorConfig Validation** +```bash +# Verify file formatting (indentation, line endings, etc.) +mvn editorconfig:check +``` +**Must Pass**: All files must conform to .editorconfig rules + +**D. Compilation** +```bash +# Ensure code compiles without errors +mvn clean compile -pl <module> -am -Dmaven.javadoc.skip=true -ntp +``` +**Must Pass**: No compilation errors + +#### Frontend Module (hubble-fe) + +**A. Prettier Formatting** +```bash +cd hugegraph-hubble/hubble-fe + +# Check formatting +npx prettier --check . + +# Auto-fix if needed +npx prettier --write . +``` +**Must Pass**: All files properly formatted + +**B. Stylelint (CSS/Less)** +```bash +# Check CSS/Less files +npx stylelint "**/*.{css,less}" + +# Auto-fix if needed +npx stylelint "**/*.{css,less}" --fix +``` +**Must Pass**: No linting errors + +**C. TypeScript/JavaScript Linting** +```bash +# Run yarn lint +yarn lint +``` +**Must Pass**: No linting errors + +#### Go Module (client-go) + +**A. Go Formatting** +```bash +cd hugegraph-client-go + +# Format code +go fmt ./... + +# Vet code +go vet ./... +``` +**Must Pass**: No formatting or vet issues + +### 2. Run Tests + +#### Java Tests + +**A. Unit Tests** (Always run) +```bash +# For hugegraph-client +cd hugegraph-client +mvn test -Dtest=UnitTestSuite -ntp + +# For hugegraph-loader +cd hugegraph-loader +mvn test -P unit -ntp +``` +**Must Pass**: All unit tests passing + +**B. Integration/API Tests** (If API changes made) +```bash +# For hugegraph-client (requires HugeGraph server) +mvn test -Dtest=ApiTestSuite +mvn test -Dtest=FuncTestSuite + +# For hugegraph-loader (requires HugeGraph server + data sources) +mvn test -P file +mvn test -P hdfs # If HDFS changes +mvn test -P jdbc # If JDBC changes +mvn test -P kafka # If Kafka changes +``` +**Required**: If you modified API/integration code + +#### Frontend Tests +```bash +cd hugegraph-hubble/hubble-fe +yarn test +``` +**Must Pass**: All frontend tests passing + +#### Go Tests +```bash +cd hugegraph-client-go +make test # Runs with race detector +``` +**Must Pass**: All tests passing with no race conditions + +### 3. Build Verification + +#### Full Module Build +```bash +# Build the module(s) you changed +mvn clean install -pl <module> -am -DskipTests -Dmaven.javadoc.skip=true -ntp +``` +**Must Pass**: Build succeeds without errors + +#### Hubble Build (if frontend/backend changed) +```bash +# Build dependencies +mvn install -pl hugegraph-client,hugegraph-loader -am -DskipTests -ntp + +# Build hubble +cd hugegraph-hubble +mvn -e compile package -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -ntp +``` +**Must Pass**: Hubble builds successfully + +### 4. Documentation Updates + +**Check if any of these need updating:** +- [ ] Module README.md +- [ ] API documentation (JavaDoc) +- [ ] Code comments +- [ ] CHANGELOG (if applicable) +- [ ] Version numbers (if release) + +### 5. Git Pre-commit + +**A. Verify Changes** +```bash +# Check what you're committing +git status +git --no-pager diff --staged +``` + +**B. Ensure Proper Commit Message** +Format: `type(scope): subject` + +Examples: +- `feat(client): add batch query support for vertices` +- `fix(loader): handle null values in CSV parsing` +- `docs(hubble): update installation instructions` +- `chore(deps): upgrade jackson to 2.12.3` +- `refactor(tools): simplify backup command logic` +- `test(loader): add HDFS connection retry tests` + +**C. Verify No Unintended Files** +```bash +# Check .gitignore is working +git status --ignored +``` +Do NOT commit: +- `target/` directories +- `*.iml`, `.idea/` files (IDE specific) +- `node_modules/` +- `.flattened-pom.xml` +- Log files +- Build artifacts + +## Pull Request Checklist + +Before opening a PR: + +- [ ] All tests passing locally +- [ ] Code style checks passing (checkstyle, prettier, etc.) +- [ ] No merge conflicts with target branch +- [ ] PR description clearly explains: + - What changed + - Why it changed + - How to test it +- [ ] Reference issue number (if applicable): `Fixes #123` +- [ ] Updated documentation (if applicable) +- [ ] Added tests for new functionality +- [ ] CI builds passing on GitHub Actions + +## CI/CD Pipeline Checks + +The following will be automatically checked by GitHub Actions: + +### Java Client CI (`client-ci.yml`) +1. Compile hugegraph-client +2. Run UnitTestSuite +3. Run ApiTestSuite (requires HugeGraph server) +4. Run FuncTestSuite +5. Upload coverage to Codecov + +### Loader CI (`loader-ci.yml`) +1. Install dependencies (Hadoop, MySQL, HugeGraph) +2. Compile client + loader +3. Run unit tests (`-P unit`) +4. Run file tests (`-P file`) +5. Run HDFS tests (`-P hdfs`) +6. Run JDBC tests (`-P jdbc`) +7. Run Kafka tests (`-P kafka`) +8. Upload coverage to Codecov + +### Hubble CI (`hubble-ci.yml`) +1. Setup Node.js 18.20.8 +2. Install frontend dependencies (yarn) +3. Build frontend (React + TypeScript) +4. Compile backend (Spring Boot) +5. Run tests +6. Package distribution + +### Go Client CI (`client-go-ci.yml`) +1. Setup Go environment +2. Download dependencies +3. Run `make test` (with race detector) +4. Build binary + +### Tools CI (`tools-ci.yml`) +1. Compile hugegraph-tools +2. Run tests +3. Package distribution + +### Spark Connector CI (`spark-connector-ci.yml`) +1. Setup Scala environment +2. Compile spark-connector +3. Run tests + +### CodeQL Analysis (`codeql-analysis.yml`) +- Security vulnerability scanning +- Code quality analysis + +### License Checker (`license-checker.yml`) +- Verify Apache 2.0 license headers +- Check dependency licenses + +## Common Issues and Solutions + +### Issue: Checkstyle Failures +**Solution**: +1. Check error message for specific rule violation +2. Fix manually or use IDE auto-format (IntelliJ IDEA) +3. Common issues: + - Line too long (max 100 chars) + - Star imports + - Missing whitespace + - Wrong indentation (use 4 spaces) + +### Issue: Test Failures +**Solution**: +1. Check if HugeGraph server is running (for API/Func tests) +2. Verify dependencies are installed (HDFS, MySQL, Kafka) +3. Check test logs for specific error +4. Run single test for debugging: + ```bash + mvn test -Dtest=ClassName#methodName -ntp + ``` + +### Issue: Hubble Build Failures +**Solution**: +1. Verify Node.js version: `node -v` (must be 18.20.8) +2. Clear cache and reinstall: + ```bash + rm -rf node_modules yarn.lock + yarn install + ``` +3. Check for frontend errors in build output + +### Issue: Maven Build Hangs +**Solution**: +1. Kill stuck maven process: `pkill -9 -f maven` +2. Clear local repository cache: + ```bash + rm -rf ~/.m2/repository/org/apache/hugegraph + ``` +3. Retry with `-X` for debug output: + ```bash + mvn clean install -X + ``` + +## Release-Specific Tasks + +When preparing a release: + +1. **Update Version Numbers** + - Root `pom.xml`: `<revision>` property + - Frontend: `package.json` version + - Go: Version constants + +2. **Update CHANGELOG** + - Document new features + - List bug fixes + - Note breaking changes + +3. **Run Full Test Suite** + ```bash + mvn clean verify -P apache-release + ``` + +4. **Generate Distribution** + ```bash + mvn clean package -DskipTests + ``` + +5. **Sign Artifacts** (for Apache release) + ```bash + mvn clean install -P apache-release + ``` + +## Summary - Minimum Required Checks + +**For any code change, ALWAYS run:** + +```bash +# 1. Checkstyle (Java) +mvn checkstyle:check + +# 2. License check +mvn apache-rat:check + +# 3. EditorConfig +mvn editorconfig:check + +# 4. Unit tests +mvn test -Dtest=UnitTestSuite -ntp # or appropriate suite + +# 5. Build +mvn clean install -DskipTests -ntp +``` + +**For frontend changes, ALSO run:** +```bash +cd hugegraph-hubble/hubble-fe +npx prettier --check . +npx stylelint "**/*.{css,less}" +yarn lint +yarn test +``` + +**For Go changes, ALSO run:** +```bash +cd hugegraph-client-go +go fmt ./... +go vet ./... +make test +``` + +--- + +**CRITICAL**: Do NOT commit code that fails any of the required checks. CI will fail and PR will be blocked. \ No newline at end of file diff --git a/.serena/memories/testing_infrastructure.md b/.serena/memories/testing_infrastructure.md new file mode 100644 index 00000000..915c0d30 --- /dev/null +++ b/.serena/memories/testing_infrastructure.md @@ -0,0 +1,635 @@ +# Testing Infrastructure - HugeGraph Toolchain + +## Testing Philosophy + +- **Unit Tests**: Test individual components in isolation, no external dependencies +- **Integration Tests**: Test interactions with HugeGraph server and external systems +- **Functional Tests**: End-to-end workflows testing complete features + +## Test Organization + +### Test Suite Structure (Java Modules) + +All Java modules use **JUnit 4** with test suites: + +``` +src/test/java/ +├── unit/ +│ ├── *Test.java # Individual unit tests +│ └── UnitTestSuite.java # Suite aggregator +├── api/ +│ ├── *ApiTest.java # API integration tests +│ └── ApiTestSuite.java +└── functional/ + ├── *FuncTest.java # Functional tests + └── FuncTestSuite.java +``` + +### Test Naming Conventions + +**Class Names**: +- Unit tests: `ClassNameTest.java` +- Integration tests: `ClassNameApiTest.java` or `ClassNameIntegrationTest.java` +- Test suites: `UnitTestSuite.java`, `ApiTestSuite.java`, `FuncTestSuite.java` + +**Method Names**: +- Descriptive: `testGetVertexById()`, `testCreateEdgeWithInvalidLabel()` +- Pattern: `test<MethodUnderTest><Condition>()` + +## Module-Specific Testing + +## 1. hugegraph-client Tests + +### Test Suites + +#### UnitTestSuite +**Purpose**: Test serialization, utilities, and internal logic +**No External Dependencies**: Can run without HugeGraph server + +**Example Tests**: +```java +@RunWith(Suite.class) [email protected]({ + VertexSerializerTest.class, + PathSerializerTest.class, + RestResultTest.class, + BatchElementRequestTest.class, + PropertyKeyTest.class, + IndexLabelTest.class, + CommonUtilTest.class, + IdUtilTest.class, + SplicingIdGeneratorTest.class +}) +public class UnitTestSuite {} +``` + +**Run Command**: +```bash +cd hugegraph-client +mvn test -Dtest=UnitTestSuite -ntp +``` + +#### ApiTestSuite +**Purpose**: Test REST API interactions +**Requires**: HugeGraph server running on localhost:8080 + +**Example Tests**: +- `VertexApiTest`: Test vertex CRUD operations +- `EdgeApiTest`: Test edge CRUD operations +- `SchemaApiTest`: Test schema management +- `TraverserApiTest`: Test graph traversal algorithms +- `GremlinApiTest`: Test Gremlin query execution + +**Run Command**: +```bash +cd hugegraph-client +mvn test -Dtest=ApiTestSuite +``` + +#### FuncTestSuite +**Purpose**: End-to-end functional scenarios +**Requires**: HugeGraph server + complete setup + +**Run Command**: +```bash +cd hugegraph-client +mvn test -Dtest=FuncTestSuite +``` + +### Test Setup/Teardown Pattern + +```java +public class VertexApiTest extends BaseApiTest { + private static HugeClient client; + private static GraphManager graph; + + @BeforeClass + public static void setup() { + client = new HugeClient("http://localhost:8080";, "hugegraph"); + graph = client.graph(); + + // Setup schema + setupSchema(); + } + + @AfterClass + public static void teardown() { + client.close(); + } + + @Before + public void prepare() { + // Clear data before each test + graph.clearVertices(); + } + + @Test + public void testAddVertex() { + Vertex vertex = graph.addVertex("person", "name", "Alice"); + assertNotNull(vertex.id()); + assertEquals("Alice", vertex.property("name")); + } +} +``` + +### Mocking (Unit Tests) + +```java +public class RestClientTest { + @Mock + private RestClient mockClient; + + @Before + public void setup() { + MockitoAnnotations.initMocks(this); + } + + @Test + public void testGetVertex() { + // Mock response + Vertex expectedVertex = new Vertex("person"); + when(mockClient.get("/vertices/1", Vertex.class)) + .thenReturn(expectedVertex); + + // Test + Vertex result = mockClient.get("/vertices/1", Vertex.class); + assertEquals(expectedVertex, result); + + // Verify + verify(mockClient).get("/vertices/1", Vertex.class); + } +} +``` + +## 2. hugegraph-loader Tests + +### Test Profiles (Maven) + +#### Profile: unit +**Purpose**: Unit tests only +**Run Command**: +```bash +cd hugegraph-loader +mvn test -P unit -ntp +``` + +**Tests**: Parser, mapper, builder unit tests + +#### Profile: file +**Purpose**: File source loading tests +**Requires**: Test data files (CSV, JSON, TXT) +**Run Command**: +```bash +mvn test -P file +``` + +**Test Resources**: +``` +src/test/resources/ +├── file/ +│ ├── persons.csv +│ ├── knows.json +│ └── struct.json # Mapping configuration +``` + +#### Profile: hdfs +**Purpose**: HDFS source loading tests +**Requires**: Hadoop HDFS cluster (local or remote) +**Setup**: CI installs Hadoop via `install-hadoop.sh` +**Run Command**: +```bash +mvn test -P hdfs +``` + +#### Profile: jdbc +**Purpose**: Database source loading tests +**Requires**: MySQL running (CI uses Docker) +**Setup**: CI installs MySQL via `install-mysql.sh` +**Run Command**: +```bash +mvn test -P jdbc +``` + +**Test Databases**: MySQL, PostgreSQL, Oracle (if driver available) + +#### Profile: kafka +**Purpose**: Kafka streaming source tests +**Requires**: Kafka broker running +**Run Command**: +```bash +mvn test -P kafka +``` + +### Test Data Management + +**Test Resources Structure**: +``` +src/test/resources/ +├── struct/ +│ ├── vertices.json # Vertex mapping configs +│ └── edges.json # Edge mapping configs +├── file/ +│ ├── vertex_person.csv +│ ├── edge_knows.csv +│ └── example.json +├── jdbc/ +│ └── init.sql # Database init script +└── log4j2.xml # Test logging config +``` + +### Integration Test Pattern (Loader) + +```java +public class FileLoadTest extends BaseLoadTest { + private static LoadContext context; + private static HugeClient client; + + @BeforeClass + public static void setup() { + // Start HugeGraph server (CI does this) + client = new HugeClient("http://localhost:8080";, "hugegraph"); + + // Create schema + createSchema(client); + + // Prepare load context + context = new LoadContext(); + context.setStructPath("src/test/resources/struct/vertices.json"); + } + + @Test + public void testLoadCSV() { + // Load data + LoadOptions options = new LoadOptions(); + options.file = "src/test/resources/file/vertex_person.csv"; + + HugeGraphLoader loader = new HugeGraphLoader(context, options); + loader.load(); + + // Verify + List<Vertex> vertices = client.graph().listVertices("person"); + assertEquals(100, vertices.size()); + } +} +``` + +## 3. hugegraph-hubble Tests + +### Backend Tests (Java/Spring Boot) + +**Test Framework**: JUnit 4 + Spring Test + MockMvc + +**Example Controller Test**: +```java +@RunWith(SpringRunner.class) +@WebMvcTest(GraphConnectionController.class) +public class GraphConnectionControllerTest { + @Autowired + private MockMvc mockMvc; + + @MockBean + private GraphConnectionService service; + + @Test + public void testCreateConnection() throws Exception { + GraphConnection connection = new GraphConnection(); + connection.setName("test-graph"); + connection.setHost("localhost"); + connection.setPort(8080); + + when(service.create(any())).thenReturn(connection); + + mockMvc.perform(post("/api/graph-connections") + .contentType(MediaType.APPLICATION_JSON) + .content(toJson(connection))) + .andExpect(status().isOk()) + .andExpect(jsonPath("$.name").value("test-graph")); + } +} +``` + +### Frontend Tests (React/TypeScript) + +**Test Framework**: Jest + React Testing Library + +**Run Command**: +```bash +cd hugegraph-hubble/hubble-fe +yarn test +``` + +**Example Component Test**: +```typescript +import { render, screen, fireEvent } from '@testing-library/react'; +import GraphSelector from '../GraphSelector'; + +describe('GraphSelector', () => { + it('renders graph list', () => { + const graphs = [ + { id: 1, name: 'graph1' }, + { id: 2, name: 'graph2' } + ]; + + render(<GraphSelector graphs={graphs} />); + + expect(screen.getByText('graph1')).toBeInTheDocument(); + expect(screen.getByText('graph2')).toBeInTheDocument(); + }); + + it('calls onSelect when graph clicked', () => { + const onSelect = jest.fn(); + const graphs = [{ id: 1, name: 'graph1' }]; + + render(<GraphSelector graphs={graphs} onSelect={onSelect} />); + + fireEvent.click(screen.getByText('graph1')); + expect(onSelect).toHaveBeenCalledWith(graphs[0]); + }); +}); +``` + +**Store Test (MobX)**: +```typescript +import GraphManagementStore from '../stores/GraphManagementStore'; + +describe('GraphManagementStore', () => { + let store: GraphManagementStore; + + beforeEach(() => { + store = new GraphManagementStore(); + }); + + it('loads graphs from API', async () => { + // Mock API + jest.spyOn(api, 'getGraphs').mockResolvedValue([ + { id: 1, name: 'graph1' } + ]); + + await store.loadGraphs(); + + expect(store.graphList).toHaveLength(1); + expect(store.graphList[0].name).toBe('graph1'); + }); +}); +``` + +## 4. hugegraph-client-go Tests + +**Test Framework**: Go standard testing + testify + +**Run Command**: +```bash +cd hugegraph-client-go +make test # Runs: go test -race -timeout 30s +``` + +**Test Structure**: +``` +. +├── client_test.go +├── graph_test.go +├── schema_test.go +└── traverser_test.go +``` + +**Example Test**: +```go +package hugegraph + +import ( + "testing" + "github.com/stretchr/testify/assert" +) + +func TestCreateVertex(t *testing.T) { + client := NewClient("http://localhost:8080";, "hugegraph") + defer client.Close() + + vertex := Vertex{ + Label: "person", + Properties: map[string]interface{}{ + "name": "Alice", + "age": 30, + }, + } + + created, err := client.Graph().AddVertex(vertex) + assert.NoError(t, err) + assert.NotEmpty(t, created.ID) + assert.Equal(t, "Alice", created.Properties["name"]) +} + +func TestGetVertexNotFound(t *testing.T) { + client := NewClient("http://localhost:8080";, "hugegraph") + defer client.Close() + + _, err := client.Graph().GetVertex("non-existent-id") + assert.Error(t, err) +} +``` + +**Benchmark Tests**: +```go +func BenchmarkAddVertex(b *testing.B) { + client := NewClient("http://localhost:8080";, "hugegraph") + defer client.Close() + + b.ResetTimer() + for i := 0; i < b.N; i++ { + client.Graph().AddVertex(Vertex{ + Label: "person", + Properties: map[string]interface{}{"name": "test"}, + }) + } +} +``` + +## CI/CD Testing Pipeline + +### GitHub Actions Workflow + +Each module has its own CI workflow: + +#### client-ci.yml +```yaml +steps: + - name: Install HugeGraph Server + run: ./assembly/travis/install-hugegraph-from-source.sh + + - name: Compile + run: mvn compile -pl hugegraph-client -ntp + + - name: Run Unit Tests + run: mvn test -Dtest=UnitTestSuite -ntp + + - name: Run API Tests + run: mvn test -Dtest=ApiTestSuite + + - name: Run Func Tests + run: mvn test -Dtest=FuncTestSuite + + - name: Upload Coverage + uses: codecov/codecov-action@v3 +``` + +#### loader-ci.yml +```yaml +steps: + - name: Install Dependencies + run: | + ./assembly/travis/install-hadoop.sh + ./assembly/travis/install-mysql.sh + ./assembly/travis/install-hugegraph-from-source.sh + + - name: Run Tests + run: | + mvn test -P unit + mvn test -P file + mvn test -P hdfs + mvn test -P jdbc + mvn test -P kafka +``` + +### Test Utilities + +#### CI Setup Scripts +```bash +# Install HugeGraph server from source +./assembly/travis/install-hugegraph-from-source.sh <commit-id> + +# Install Hadoop for HDFS tests +./assembly/travis/install-hadoop.sh + +# Install MySQL for JDBC tests +./assembly/travis/install-mysql.sh <db-name> <password> +``` + +## Test Coverage + +### Coverage Tools +- **Java**: JaCoCo Maven plugin +- **JavaScript/TypeScript**: Jest built-in coverage +- **Go**: go test -cover + +### Generating Coverage Reports + +**Java (JaCoCo)**: +```bash +mvn test jacoco:report +# Report: target/site/jacoco/index.html +``` + +**Frontend (Jest)**: +```bash +cd hugegraph-hubble/hubble-fe +yarn test --coverage +# Report: coverage/lcov-report/index.html +``` + +**Go**: +```bash +cd hugegraph-client-go +go test -coverprofile=coverage.out ./... +go tool cover -html=coverage.out +``` + +### Coverage Targets +- Unit tests: Aim for 80%+ coverage +- Integration tests: Cover critical paths +- Functional tests: Cover end-to-end scenarios + +## Common Testing Patterns + +### Test Data Builders +```java +public class TestDataBuilder { + public static Vertex createPersonVertex(String name, int age) { + return new Vertex("person") + .property("name", name) + .property("age", age); + } + + public static Edge createKnowsEdge(Vertex source, Vertex target) { + return source.addEdge("knows", target) + .property("date", "2023-01-01"); + } +} +``` + +### Test Assertions (Custom) +```java +public class GraphAssertions { + public static void assertVertexExists(HugeClient client, Object id) { + Vertex vertex = client.graph().getVertex(id); + assertNotNull("Vertex should exist", vertex); + } + + public static void assertEdgeCount(HugeClient client, + String label, int expected) { + List<Edge> edges = client.graph().listEdges(label); + assertEquals("Edge count mismatch", expected, edges.size()); + } +} +``` + +### Parameterized Tests (JUnit 4) +```java +@RunWith(Parameterized.class) +public class IdGeneratorTest { + @Parameters + public static Collection<Object[]> data() { + return Arrays.asList(new Object[][] { + { "alice", "person:alice" }, + { "bob", "person:bob" }, + { "charlie", "person:charlie" } + }); + } + + private String input; + private String expected; + + public IdGeneratorTest(String input, String expected) { + this.input = input; + this.expected = expected; + } + + @Test + public void testGenerateId() { + String result = IdGenerator.generate("person", input); + assertEquals(expected, result); + } +} +``` + +## Debugging Tests + +### Running Single Test +```bash +# Java +mvn test -Dtest=ClassName#methodName -ntp + +# Go +go test -run TestFunctionName -v + +# Frontend +yarn test ComponentName.test.tsx +``` + +### Debug Mode (Java) +```bash +# Run with remote debugging enabled +mvnDebug test -Dtest=ClassName +# Then attach debugger to port 8000 +``` + +### Verbose Output +```bash +# Maven verbose +mvn test -X + +# Go verbose +go test -v + +# Frontend verbose +yarn test --verbose +``` \ No newline at end of file diff --git a/.serena/project.yml b/.serena/project.yml new file mode 100644 index 00000000..bd75e5b3 --- /dev/null +++ b/.serena/project.yml @@ -0,0 +1,84 @@ +# list of languages for which language servers are started; choose from: +# al bash clojure cpp csharp csharp_omnisharp +# dart elixir elm erlang fortran go +# haskell java julia kotlin lua markdown +# nix perl php python python_jedi r +# rego ruby ruby_solargraph rust scala swift +# terraform typescript typescript_vts zig +# Note: +# - For C, use cpp +# - For JavaScript, use typescript +# Special requirements: +# - csharp: Requires the presence of a .sln file in the project folder. +# When using multiple languages, the first language server that supports a given file will be used for that file. +# The first language is the default language and the respective language server will be used as a fallback. +# Note that when using the JetBrains backend, language servers are not used and this list is correspondingly ignored. +languages: +- java + +# the encoding used by text files in the project +# For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings +encoding: "utf-8" + +# whether to use the project's gitignore file to ignore files +# Added on 2025-04-07 +ignore_all_files_in_gitignore: true + +# list of additional paths to ignore +# same syntax as gitignore, so you can use * and ** +# Was previously called `ignored_dirs`, please update your config if you are using that. +# Added (renamed) on 2025-04-07 +ignored_paths: [] + +# whether the project is in read-only mode +# If set to true, all editing tools will be disabled and attempts to use them will result in an error +# Added on 2025-04-18 +read_only: false + +# list of tool names to exclude. We recommend not excluding any tools, see the readme for more details. +# Below is the complete list of tools for convenience. +# To make sure you have the latest list of tools, and to view their descriptions, +# execute `uv run scripts/print_tool_overview.py`. +# +# * `activate_project`: Activates a project by name. +# * `check_onboarding_performed`: Checks whether project onboarding was already performed. +# * `create_text_file`: Creates/overwrites a file in the project directory. +# * `delete_lines`: Deletes a range of lines within a file. +# * `delete_memory`: Deletes a memory from Serena's project-specific memory store. +# * `execute_shell_command`: Executes a shell command. +# * `find_referencing_code_snippets`: Finds code snippets in which the symbol at the given location is referenced. +# * `find_referencing_symbols`: Finds symbols that reference the symbol at the given location (optionally filtered by type). +# * `find_symbol`: Performs a global (or local) search for symbols with/containing a given name/substring (optionally filtered by type). +# * `get_current_config`: Prints the current configuration of the agent, including the active and available projects, tools, contexts, and modes. +# * `get_symbols_overview`: Gets an overview of the top-level symbols defined in a given file. +# * `initial_instructions`: Gets the initial instructions for the current project. +# Should only be used in settings where the system prompt cannot be set, +# e.g. in clients you have no control over, like Claude Desktop. +# * `insert_after_symbol`: Inserts content after the end of the definition of a given symbol. +# * `insert_at_line`: Inserts content at a given line in a file. +# * `insert_before_symbol`: Inserts content before the beginning of the definition of a given symbol. +# * `list_dir`: Lists files and directories in the given directory (optionally with recursion). +# * `list_memories`: Lists memories in Serena's project-specific memory store. +# * `onboarding`: Performs onboarding (identifying the project structure and essential tasks, e.g. for testing or building). +# * `prepare_for_new_conversation`: Provides instructions for preparing for a new conversation (in order to continue with the necessary context). +# * `read_file`: Reads a file within the project directory. +# * `read_memory`: Reads the memory with the given name from Serena's project-specific memory store. +# * `remove_project`: Removes a project from the Serena configuration. +# * `replace_lines`: Replaces a range of lines within a file with new content. +# * `replace_symbol_body`: Replaces the full definition of a symbol. +# * `restart_language_server`: Restarts the language server, may be necessary when edits not through Serena happen. +# * `search_for_pattern`: Performs a search for a pattern in the project. +# * `summarize_changes`: Provides instructions for summarizing the changes made to the codebase. +# * `switch_modes`: Activates modes by providing a list of their names +# * `think_about_collected_information`: Thinking tool for pondering the completeness of collected information. +# * `think_about_task_adherence`: Thinking tool for determining whether the agent is still on track with the current task. +# * `think_about_whether_you_are_done`: Thinking tool for determining whether the task is truly completed. +# * `write_memory`: Writes a named memory (for future reference) to Serena's project-specific memory store. +excluded_tools: [] + +# initial prompt for the project. It will always be given to the LLM upon activating the project +# (contrary to the memories, which are loaded on demand). +initial_prompt: "" + +project_name: "toolchain" +included_optional_tools: []
