Hi tison, Yes and my account is [email protected]
Please let me know. Thanks. Cheers, PG On 2026/08/11 14:31:40 tison wrote: > Hi Piergiorgio, > > Do you have an account of the cwiki space (ASF Confluence instance)? > > Best, > tison. > > > Piergiorgio Lucidi <[email protected]> 于2026年8月11日周二 22:28写道： > > > Hi everyone, > > > > I would like to propose OpenCrawling as a new project for incubation within > > the Apache Software Foundation. Currently the project is hosted on GitHub > > [1]. > > > > The official proposal is currently available in the OpenCrawling Wiki in > > markdown format [2]. > > > > I tried to share the proposal in our Confluence but it seems that I don't > > have permission to create the new page under the Proposals page. Anyway if > > someone can guide me on resolving this issue it would be great! > > > > Once space permissions are granted on cwiki.apache.org, I will also mirror > > the proposal on the Incubator CWIKI proposals page. > > Below you also find the same proposal ready to be copy-pasted into our > > Confluence. > > > > We welcome feedback, questions and discussion from the Incubator community! > > > > Best regards, > > Piergiorgio > > On behalf of the OpenCrawling Core Team > > > > [1] - https://github.com/opencrawling/opencrawling > > [2] - > > https://github.com/opencrawling/opencrawling/wiki/Apache-Incubator-Proposal > > > > -------------------------------------------------------------- > > > > h1. Apache Incubator Proposal: OpenCrawling > > > > h2. Abstract > > > > *OpenCrawling* is an open-source, enterprise-grade, high-performance data > > crawling, content ingestion, and security-aware vector search platform. > > Built on modern Java 25 (leveraging Virtual Threads and Structured > > Concurrency), Spring Boot 4, and Spring AI, OpenCrawling serves as the > > reference implementation of the *Open Ingestion Standard (OIS)* and > > provides a secure *Model Context Protocol (MCP)* server interface. It > > orchestrates scalable data flows from heterogeneous enterprise repositories > > (e.g., SharePoint, S3, CMIS/Alfresco, BPMN engines like Camunda and > > Flowable, Apache Iceberg, Apache Ozone) to downstream vector databases > > (e.g., Milvus, Qdrant, OpenSearch, Vespa, pgvector) with source-level > > Access Control List (ACL) security enforcement. > > > > ---- > > > > h2. Proposal > > > > The OpenCrawling community proposes to incubate *OpenCrawling* as a new > > project within the Apache Software Foundation (ASF). OpenCrawling provides > > a decoupled, vendor-neutral enterprise data integration framework that > > bridges the gap between traditional enterprise content management (ECM) > > repositories and modern Large Language Model (LLM) / Retrieval-Augmented > > Generation (RAG) architectures. > > > > The project encompasses: > > # *Core Ingestion Runtime ({{oc-core}}, {{oc-runtime}})*: A distributed, > > asynchronous engine built with virtual threads, claim-check metadata > > patterns, and Apache Kafka event streams. > > # *Repository & Vector Connectors*: Standardized connectors for scanning > > source systems and indexing vector embeddings into major vector stores. > > # *Open Ingestion Standard (OIS)*: The formal JSON/YAML schema > > specifications defining unified document payloads, ACL security SIDs, and > > crawler job configurations. > > # *Secure Model Context Protocol (MCP) Server*: A Zero-Trust context > > retrieval server enforcing document-level permissions (Active Directory > > SIDs, LDAP groups, user principals) at query time. > > # *Observability & Developer Tooling*: AI-Powered Observability (AIOps) > > over OpenTelemetry traces, Auto-Narrativization Copilot, Java Client SDK, > > and Maven Archetypes for custom connector development. > > > > ---- > > > > h2. Background > > > > In enterprise AI and RAG architectures, LLM agents require seamless access > > to unstructured content stored across legacy and cloud repositories. > > However, traditional ingestion pipelines often strip out or ignore > > source-level security metadata (ACLs), leading to context leakage where an > > AI model synthesizes responses using confidential documents that the > > requesting user does not have permissions to view. > > > > Furthermore, legacy crawling tools (such as Apache ManifoldCF or Apache > > Nutch) were architected over a decade ago prior to the emergence of vector > > databases, LLMs, Model Context Protocol (MCP), and modern Java features > > like Virtual Threads (JEP 444) and Structured Concurrency. > > > > OpenCrawling was created to address this modern ingestion crisis by > > providing a native Java 25/Spring AI implementation engineered specifically > > for LLM search scenarios, zero-trust context retrieval, and high-throughput > > asynchronous processing. > > > > ---- > > > > h2. Rationale > > > > The Apache Software Foundation is the natural home for OpenCrawling. ASF > > has long been the center of innovation for enterprise search and big data > > infrastructure, hosting cornerstone projects such as Apache Lucene, Apache > > Solr, Apache Tika, Apache Kafka, Apache Iceberg, Apache Ozone, Apache > > ManifoldCF, and Apache Nutch. > > > > Bringing OpenCrawling to the ASF offers multiple mutual benefits: > > * *Ecosystem Integration*: OpenCrawling directly integrates with and builds > > upon existing Apache projects, including *Apache Tika* (text extraction), > > *Apache Kafka* (event-driven pipeline), *Apache Iceberg* (lakehouse > > connector), *Apache Ozone* (claim-check object storage), and *Apache Maven* > > (connector archetype distribution). > > * *Vendor-Neutral Governance*: Neutral governance under the Apache Way is > > vital to establishing OpenCrawling and OIS as industry-wide, > > vendor-agnostic ingestion standards. > > * *Community Sustainability*: Operating as an Apache project will attract a > > broader community of enterprise adopters, cloud providers, AI framework > > developers, and search engine vendors. > > > > ---- > > > > h2. Initial Goals > > > > During incubation, the OpenCrawling project will focus on the following > > milestones: > > > > # *ASF Migration & Infrastructure*: > > ** Transfer codebases ({{opencrawling}}, {{open-ingestion-standard}}, {{ > > opencrawling.github.io}}) to Apache infrastructure ({{ > > github.com/apache/incubator-opencrawling}} > > <http://github.com/apache/incubator-opencrawling%7D%7D>). > > ** Rebrand build artifacts to {{org.apache.opencrawling}}. > > ** Setup ASF-compliant CI/CD pipelines using GitHub Actions. > > # *Community & Governance*: > > ** Adopt the Apache Way for all decisions, roadmap discussions, and release > > voting. > > ** Expand the contributor base across independent developers, enterprise > > search users, and corporate contributors. > > # *Ecosystem & Connector Expansion*: > > ** Release additional output connectors (Elasticsearch, Apache Solr, > > RESTHeart). > > ** Add native integration for fine-grained authorization frameworks (e.g., > > OpenFGA). > > ** Enhance gRPC support for high-efficiency inter-microservice > > communication. > > ** Standardize OIS specification drafts under ASF governance. > > # *Compliance & Licensing*: > > ** Complete IP clearance and execute software grant agreements. > > ** Ensure all third-party dependencies strictly conform to Apache License > > Category A policies. > > > > ---- > > > > h2. Current Status > > > > h3. Meritocracy > > The OpenCrawling project was established with meritocratic principles from > > day one. Design decisions, architecture changes, issue tracking, and > > roadmap discussions take place openly on GitHub through RFCs, Pull > > Requests, and public wiki pages. > > > > h3. Community > > The OpenCrawling community includes developers and architects from > > enterprise search, ECM, and AI background. Community channels include > > GitHub Discussions, Slack, and social media announcements. The project > > actively encourages external contributions via Maven archetypes and modular > > connector development. > > > > h3. Core Developers > > The initial core developers are experienced software architects and > > open-source veterans with extensive experience in enterprise search, > > content management, and ASF governance: > > > > * *Piergiorgio Lucidi* ({{[email protected]}}) – Founder, Lead > > Architect. ASF Member and PMC Member/Committer on multiple Apache projects > > (including Apache ManifoldCF and Apache Chemistry). > > * *Michael Cizmar* ({{[email protected]}}) – Lead Architect & > > Developer. Specialist in enterprise search and cloud infrastructure. > > * *Luis Cabaceira* ({{[email protected]}}) – Lead Architect & > > Developer. Specialist in document processing and AI integration. > > > > ---- > > > > h2. Known Risks > > > > h3. Orphaned Products > > The risk of OpenCrawling becoming orphaned is low. The project solves an > > active, urgent security and performance problem in enterprise AI adoption > > (RAG ACL context leakage). The core maintainers are committed to its > > long-term evolution and actively use it in production environments. > > > > h3. Inexperience with Open Source > > The project leadership has deep experience with open-source communities. > > Piergiorgio Lucidi is an active ASF Member and PMC member with over a > > decade of experience guiding projects through the Apache Way. > > > > h3. Homogenous Developers > > The initial committers come from diverse geographical locations (Italy, > > United States, Portugal) and distinct organizations/consultancies. > > Incubating at Apache will further diversify the developer base by > > encouraging contributions from enterprise organizations and search vendors. > > > > h3. Reliance on Third-Party Products > > OpenCrawling is designed to be vendor-neutral. Core dependencies are > > open-source libraries under permissive licenses (Apache 2.0, MIT, BSD): > > * Spring Boot & Spring AI (Apache 2.0) > > * Apache Tika (Apache 2.0) > > * Apache Kafka (Apache 2.0) > > * PostgreSQL / pgvector (PostgreSQL License / MIT) > > * Docker & OpenTelemetry (Apache 2.0) > > > > There are no GPL/AGPL dependencies in the runtime core. > > > > h3. Relationship with Sponsored Products / Brand > > OpenCrawling is an independent project. The name "OpenCrawling" has been > > used for the open-source codebase. The trademark will be transferred to the > > Apache Software Foundation upon incubation acceptance. > > > > ---- > > > > h2. Documentation & Existing Artifacts > > > > * *GitHub Organization*: [https://github.com/opencrawling] > > * *Main Code Base*: {{opencrawling/opencrawling}} > > * *Specification Repo*: {{opencrawling/open-ingestion-standard}} > > * *Documentation & Wiki*: [ > > https://github.com/opencrawling/opencrawling/wiki > > ] > > * *Java Client SDK*: {{oc-java-client-sdk}} ([Sonatype Central - > > org.opencrawling:oc-java-client-sdk| > > https://central.sonatype.com/artifact/org.opencrawling/oc-java-client-sdk > > ]) > > * *Maven Archetypes*: [Sonatype Central - org.opencrawling.archetypes| > > > > https://central.sonatype.com/artifact/org.opencrawling.archetypes/opencrawling-connector-archetypes > > ] > > > > ---- > > > > h2. Initial Source & Intellectual Property Submission > > > > h3. Initial Source Code > > The initial codebase to be granted to the ASF resides in the following > > GitHub repositories: > > * {{opencrawling/opencrawling}} (Core engine, microservices, UI, > > connectors, MCP server) > > * {{opencrawling/open-ingestion-standard}} (JSON schemas, whitepaper, > > specifications) > > * {{opencrawling/opencrawling.github.io}} (Project web site and > > documentation source) > > > > All source code is currently licensed under the *Apache License, Version > > 2.0*. > > > > h3. Software Grant / ICLA / CCLA > > All core contributors will submit Individual Contributor License Agreements > > (ICLAs) and corporate software grants will be executed upon acceptance into > > the Incubator. > > > > ---- > > > > h2. External Dependencies > > > > All major external dependencies of OpenCrawling use Apache-compatible > > licenses (Category A): > > > > || Dependency || License || > > | *Java Development Kit (JDK 25)* | GPLv2 + Classpath Exception | > > | *Spring Boot / Spring AI* | Apache License 2.0 | > > | *Apache Tika* | Apache License 2.0 | > > | *Apache Kafka Clients* | Apache License 2.0 | > > | *Apache Iceberg SDK* | Apache License 2.0 | > > | *Apache Ozone Client* | Apache License 2.0 | > > | *Jackson / Slf4j / Logback* | Apache 2.0 / MIT / EPL 1.0 | > > | *Milvus / Qdrant Java SDKs* | Apache License 2.0 | > > | *OpenTelemetry Java SDK* | Apache License 2.0 | > > | *React / Vite / Tailwind (Admin UI)* | MIT | > > > > ---- > > > > h2. Cryptography > > > > OpenCrawling uses standard TLS/HTTPS protocols and hashing routines > > provided by the standard Java Virtual Machine (JDK) and Spring Security > > framework for secure transport. It does not include custom cryptographic > > algorithms or controlled export software. > > > > ---- > > > > h2. Required Resources > > > > h3. Mailing Lists > > * {{[email protected]}} > > * {{[email protected]}} > > * {{[email protected]}} (PPMC) > > > > h3. Git Repositories > > * {{https://github.com/apache/incubator-opencrawling}} > > * {{https://github.com/apache/incubator-opencrawling-site}} > > > > h3. Issue Tracking > > * GitHub Issues on {{apache/incubator-opencrawling}} (or ASF Jira project > > {{OPENCRAWLING}}) > > > > h3. CI/CD Infrastructure > > * GitHub Actions workflows for automated build, test, multi-arch Docker > > image generation, and Sonar/Scorecard quality checks. > > > > ---- > > > > h2. Initial Committers & PPMC Members > > > > * *Piergiorgio Lucidi* ({{[email protected]}}) – Initial Committer & > > PPMC > > * *Michael Cizmar* ({{[email protected]}}) – Initial Committer & > > PPMC > > * *Luis Cabaceira* ({{[email protected]}}) – Initial Committer & > > PPMC > > > > _(Note: Additional mentors and committers will be welcomed during the > > discussion period on {{[email protected]}}.)_ > > > > ---- > > > > h2. Champions & Mentors > > > > * *Champion*: Piergiorgio Lucidi ({{[email protected]}}) – ASF Member > > * *Mentors*: > > ** _(TBD - Interested ASF Members/Incubator PMC members invited to step > > forward during proposal discussion)_ > > > > ---- > > > > h2. Sponsoring Entity > > > > The *Apache Incubator PMC* is requested to be the sponsoring entity for > > this project. > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
