Hi everyone, I would like to propose OpenCrawling as a new project for incubation within the Apache Software Foundation. Currently the project is hosted on GitHub [1].
The official proposal is currently available in the OpenCrawling Wiki in markdown format [2]. I tried to share the proposal in our Confluence but it seems that I don't have permission to create the new page under the Proposals page. Anyway if someone can guide me on resolving this issue it would be great! Once space permissions are granted on cwiki.apache.org, I will also mirror the proposal on the Incubator CWIKI proposals page. Below you also find the same proposal ready to be copy-pasted into our Confluence. We welcome feedback, questions and discussion from the Incubator community! Best regards, Piergiorgio On behalf of the OpenCrawling Core Team [1] - https://github.com/opencrawling/opencrawling [2] - https://github.com/opencrawling/opencrawling/wiki/Apache-Incubator-Proposal -------------------------------------------------------------- h1. Apache Incubator Proposal: OpenCrawling h2. Abstract *OpenCrawling* is an open-source, enterprise-grade, high-performance data crawling, content ingestion, and security-aware vector search platform. Built on modern Java 25 (leveraging Virtual Threads and Structured Concurrency), Spring Boot 4, and Spring AI, OpenCrawling serves as the reference implementation of the *Open Ingestion Standard (OIS)* and provides a secure *Model Context Protocol (MCP)* server interface. It orchestrates scalable data flows from heterogeneous enterprise repositories (e.g., SharePoint, S3, CMIS/Alfresco, BPMN engines like Camunda and Flowable, Apache Iceberg, Apache Ozone) to downstream vector databases (e.g., Milvus, Qdrant, OpenSearch, Vespa, pgvector) with source-level Access Control List (ACL) security enforcement. ---- h2. Proposal The OpenCrawling community proposes to incubate *OpenCrawling* as a new project within the Apache Software Foundation (ASF). OpenCrawling provides a decoupled, vendor-neutral enterprise data integration framework that bridges the gap between traditional enterprise content management (ECM) repositories and modern Large Language Model (LLM) / Retrieval-Augmented Generation (RAG) architectures. The project encompasses: # *Core Ingestion Runtime ({{oc-core}}, {{oc-runtime}})*: A distributed, asynchronous engine built with virtual threads, claim-check metadata patterns, and Apache Kafka event streams. # *Repository & Vector Connectors*: Standardized connectors for scanning source systems and indexing vector embeddings into major vector stores. # *Open Ingestion Standard (OIS)*: The formal JSON/YAML schema specifications defining unified document payloads, ACL security SIDs, and crawler job configurations. # *Secure Model Context Protocol (MCP) Server*: A Zero-Trust context retrieval server enforcing document-level permissions (Active Directory SIDs, LDAP groups, user principals) at query time. # *Observability & Developer Tooling*: AI-Powered Observability (AIOps) over OpenTelemetry traces, Auto-Narrativization Copilot, Java Client SDK, and Maven Archetypes for custom connector development. ---- h2. Background In enterprise AI and RAG architectures, LLM agents require seamless access to unstructured content stored across legacy and cloud repositories. However, traditional ingestion pipelines often strip out or ignore source-level security metadata (ACLs), leading to context leakage where an AI model synthesizes responses using confidential documents that the requesting user does not have permissions to view. Furthermore, legacy crawling tools (such as Apache ManifoldCF or Apache Nutch) were architected over a decade ago prior to the emergence of vector databases, LLMs, Model Context Protocol (MCP), and modern Java features like Virtual Threads (JEP 444) and Structured Concurrency. OpenCrawling was created to address this modern ingestion crisis by providing a native Java 25/Spring AI implementation engineered specifically for LLM search scenarios, zero-trust context retrieval, and high-throughput asynchronous processing. ---- h2. Rationale The Apache Software Foundation is the natural home for OpenCrawling. ASF has long been the center of innovation for enterprise search and big data infrastructure, hosting cornerstone projects such as Apache Lucene, Apache Solr, Apache Tika, Apache Kafka, Apache Iceberg, Apache Ozone, Apache ManifoldCF, and Apache Nutch. Bringing OpenCrawling to the ASF offers multiple mutual benefits: * *Ecosystem Integration*: OpenCrawling directly integrates with and builds upon existing Apache projects, including *Apache Tika* (text extraction), *Apache Kafka* (event-driven pipeline), *Apache Iceberg* (lakehouse connector), *Apache Ozone* (claim-check object storage), and *Apache Maven* (connector archetype distribution). * *Vendor-Neutral Governance*: Neutral governance under the Apache Way is vital to establishing OpenCrawling and OIS as industry-wide, vendor-agnostic ingestion standards. * *Community Sustainability*: Operating as an Apache project will attract a broader community of enterprise adopters, cloud providers, AI framework developers, and search engine vendors. ---- h2. Initial Goals During incubation, the OpenCrawling project will focus on the following milestones: # *ASF Migration & Infrastructure*: ** Transfer codebases ({{opencrawling}}, {{open-ingestion-standard}}, {{ opencrawling.github.io}}) to Apache infrastructure ({{ github.com/apache/incubator-opencrawling}}). ** Rebrand build artifacts to {{org.apache.opencrawling}}. ** Setup ASF-compliant CI/CD pipelines using GitHub Actions. # *Community & Governance*: ** Adopt the Apache Way for all decisions, roadmap discussions, and release voting. ** Expand the contributor base across independent developers, enterprise search users, and corporate contributors. # *Ecosystem & Connector Expansion*: ** Release additional output connectors (Elasticsearch, Apache Solr, RESTHeart). ** Add native integration for fine-grained authorization frameworks (e.g., OpenFGA). ** Enhance gRPC support for high-efficiency inter-microservice communication. ** Standardize OIS specification drafts under ASF governance. # *Compliance & Licensing*: ** Complete IP clearance and execute software grant agreements. ** Ensure all third-party dependencies strictly conform to Apache License Category A policies. ---- h2. Current Status h3. Meritocracy The OpenCrawling project was established with meritocratic principles from day one. Design decisions, architecture changes, issue tracking, and roadmap discussions take place openly on GitHub through RFCs, Pull Requests, and public wiki pages. h3. Community The OpenCrawling community includes developers and architects from enterprise search, ECM, and AI background. Community channels include GitHub Discussions, Slack, and social media announcements. The project actively encourages external contributions via Maven archetypes and modular connector development. h3. Core Developers The initial core developers are experienced software architects and open-source veterans with extensive experience in enterprise search, content management, and ASF governance: * *Piergiorgio Lucidi* ({{[email protected]}}) – Founder, Lead Architect. ASF Member and PMC Member/Committer on multiple Apache projects (including Apache ManifoldCF and Apache Chemistry). * *Michael Cizmar* ({{[email protected]}}) – Lead Architect & Developer. Specialist in enterprise search and cloud infrastructure. * *Luis Cabaceira* ({{[email protected]}}) – Lead Architect & Developer. Specialist in document processing and AI integration. ---- h2. Known Risks h3. Orphaned Products The risk of OpenCrawling becoming orphaned is low. The project solves an active, urgent security and performance problem in enterprise AI adoption (RAG ACL context leakage). The core maintainers are committed to its long-term evolution and actively use it in production environments. h3. Inexperience with Open Source The project leadership has deep experience with open-source communities. Piergiorgio Lucidi is an active ASF Member and PMC member with over a decade of experience guiding projects through the Apache Way. h3. Homogenous Developers The initial committers come from diverse geographical locations (Italy, United States, Portugal) and distinct organizations/consultancies. Incubating at Apache will further diversify the developer base by encouraging contributions from enterprise organizations and search vendors. h3. Reliance on Third-Party Products OpenCrawling is designed to be vendor-neutral. Core dependencies are open-source libraries under permissive licenses (Apache 2.0, MIT, BSD): * Spring Boot & Spring AI (Apache 2.0) * Apache Tika (Apache 2.0) * Apache Kafka (Apache 2.0) * PostgreSQL / pgvector (PostgreSQL License / MIT) * Docker & OpenTelemetry (Apache 2.0) There are no GPL/AGPL dependencies in the runtime core. h3. Relationship with Sponsored Products / Brand OpenCrawling is an independent project. The name "OpenCrawling" has been used for the open-source codebase. The trademark will be transferred to the Apache Software Foundation upon incubation acceptance. ---- h2. Documentation & Existing Artifacts * *GitHub Organization*: [https://github.com/opencrawling] * *Main Code Base*: {{opencrawling/opencrawling}} * *Specification Repo*: {{opencrawling/open-ingestion-standard}} * *Documentation & Wiki*: [https://github.com/opencrawling/opencrawling/wiki ] * *Java Client SDK*: {{oc-java-client-sdk}} ([Sonatype Central - org.opencrawling:oc-java-client-sdk| https://central.sonatype.com/artifact/org.opencrawling/oc-java-client-sdk]) * *Maven Archetypes*: [Sonatype Central - org.opencrawling.archetypes| https://central.sonatype.com/artifact/org.opencrawling.archetypes/opencrawling-connector-archetypes ] ---- h2. Initial Source & Intellectual Property Submission h3. Initial Source Code The initial codebase to be granted to the ASF resides in the following GitHub repositories: * {{opencrawling/opencrawling}} (Core engine, microservices, UI, connectors, MCP server) * {{opencrawling/open-ingestion-standard}} (JSON schemas, whitepaper, specifications) * {{opencrawling/opencrawling.github.io}} (Project web site and documentation source) All source code is currently licensed under the *Apache License, Version 2.0*. h3. Software Grant / ICLA / CCLA All core contributors will submit Individual Contributor License Agreements (ICLAs) and corporate software grants will be executed upon acceptance into the Incubator. ---- h2. External Dependencies All major external dependencies of OpenCrawling use Apache-compatible licenses (Category A): || Dependency || License || | *Java Development Kit (JDK 25)* | GPLv2 + Classpath Exception | | *Spring Boot / Spring AI* | Apache License 2.0 | | *Apache Tika* | Apache License 2.0 | | *Apache Kafka Clients* | Apache License 2.0 | | *Apache Iceberg SDK* | Apache License 2.0 | | *Apache Ozone Client* | Apache License 2.0 | | *Jackson / Slf4j / Logback* | Apache 2.0 / MIT / EPL 1.0 | | *Milvus / Qdrant Java SDKs* | Apache License 2.0 | | *OpenTelemetry Java SDK* | Apache License 2.0 | | *React / Vite / Tailwind (Admin UI)* | MIT | ---- h2. Cryptography OpenCrawling uses standard TLS/HTTPS protocols and hashing routines provided by the standard Java Virtual Machine (JDK) and Spring Security framework for secure transport. It does not include custom cryptographic algorithms or controlled export software. ---- h2. Required Resources h3. Mailing Lists * {{[email protected]}} * {{[email protected]}} * {{[email protected]}} (PPMC) h3. Git Repositories * {{https://github.com/apache/incubator-opencrawling}} * {{https://github.com/apache/incubator-opencrawling-site}} h3. Issue Tracking * GitHub Issues on {{apache/incubator-opencrawling}} (or ASF Jira project {{OPENCRAWLING}}) h3. CI/CD Infrastructure * GitHub Actions workflows for automated build, test, multi-arch Docker image generation, and Sonar/Scorecard quality checks. ---- h2. Initial Committers & PPMC Members * *Piergiorgio Lucidi* ({{[email protected]}}) – Initial Committer & PPMC * *Michael Cizmar* ({{[email protected]}}) – Initial Committer & PPMC * *Luis Cabaceira* ({{[email protected]}}) – Initial Committer & PPMC _(Note: Additional mentors and committers will be welcomed during the discussion period on {{[email protected]}}.)_ ---- h2. Champions & Mentors * *Champion*: Piergiorgio Lucidi ({{[email protected]}}) – ASF Member * *Mentors*: ** _(TBD - Interested ASF Members/Incubator PMC members invited to step forward during proposal discussion)_ ---- h2. Sponsoring Entity The *Apache Incubator PMC* is requested to be the sponsoring entity for this project.
