Hi Piergiorgio, I've invited you to the page. But I'm unfamiliar with Confluence, so it may not be what you're looking for.
Alternatively, you may find https://selfserve.apache.org/confluence-account.html helps. Anyway, I created the proposal page [1] with the wiki page content you shared. [1] https://cwiki.apache.org/confluence/spaces/INCUBATOR/pages/446071456/OpenCrawling+Proposal Best, tison. Piergiorgio Lucidi <[email protected]> 于2026年8月12日周三 03:36写道： > Hi PJ, > > Thank you for the feedback and for highlighting Apache StormCrawler. > > I completely agree that exploring collaboration between the two projects is > a great idea. > > While StormCrawler is an incredibly powerful tool for large scale web > crawling and processing, OpenCrawling was built specifically to tackle the > enterprise content and process automation ecosystem. > > Our core focus is bridging platforms like Alfresco, Flowable, and Camunda > directly into modern LLM and RAG architectures, heavily utilizing Spring > Boot and Spring AI. > > Despite the distinct use cases, web data versus enterprise repositories, > there is absolutely a shared interest in robust data ingestion, document > parsing and vector database integration strategies. > > OpenCrawling can become the perfect home for any crawling strategy. > > I would be thrilled to connect with the StormCrawler community to see how > our projects might complement each other and share best practices moving > forward. > > We could propose to implement a brand new OpenCrawling Storm Bolt. > > In Apache Storm topology, data flows from *Spouts* (URL queues) to *Bolts* > (fetchers, parsers and indexers). The most seamless integration is to build > an opencrawling-storm-bolt. > > StormCrawler handles the heavy lifting of recursive web crawling, > politeness and HTML parsing. > > Instead of using StormCrawler's native OpenSearch/Elasticsearch indexer > bolt, the topology passes the parsed document to the new custom > OpenCrawling Storm Bolt. > > The Bolt acts as an OpenCrawling Repository Connector. It takes the raw > HTML, wraps it in the Open Ingestion Standard (OIS) format, attaches any > relevant baseline metadata and pushes it through OpenCrawling’s secure > pipeline (Java 25 / Spring Boot 4) for chunking, embedding, and Vector DB > ingestion. > > Cheers, > > PG > > Il Mar 11 Ago 2026, 18:44 PJ Fanning <[email protected]> ha scritto: > > > This doesn't block OpenCrawling joining as an ASF Incubator podling > > but I just want to highlight that there is already Apache > > StormCrawler. > > > > https://stormcrawler.apache.org/ > > > > It would be great if these projects could collaborate in areas of > > shared interest. > > > > On Tue, 11 Aug 2026 at 15:28, Piergiorgio Lucidi <[email protected] > > > > wrote: > > > > > > Hi everyone, > > > > > > I would like to propose OpenCrawling as a new project for incubation > > within > > > the Apache Software Foundation. Currently the project is hosted on > GitHub > > > [1]. > > > > > > The official proposal is currently available in the OpenCrawling Wiki > in > > > markdown format [2]. > > > > > > I tried to share the proposal in our Confluence but it seems that I > don't > > > have permission to create the new page under the Proposals page. Anyway > > if > > > someone can guide me on resolving this issue it would be great! > > > > > > Once space permissions are granted on cwiki.apache.org, I will also > > mirror > > > the proposal on the Incubator CWIKI proposals page. > > > Below you also find the same proposal ready to be copy-pasted into our > > > Confluence. > > > > > > We welcome feedback, questions and discussion from the Incubator > > community! > > > > > > Best regards, > > > Piergiorgio > > > On behalf of the OpenCrawling Core Team > > > > > > [1] - https://github.com/opencrawling/opencrawling > > > [2] - > > > > > > https://github.com/opencrawling/opencrawling/wiki/Apache-Incubator-Proposal > > > > > > -------------------------------------------------------------- > > > > > > h1. Apache Incubator Proposal: OpenCrawling > > > > > > h2. Abstract > > > > > > *OpenCrawling* is an open-source, enterprise-grade, high-performance > data > > > crawling, content ingestion, and security-aware vector search platform. > > > Built on modern Java 25 (leveraging Virtual Threads and Structured > > > Concurrency), Spring Boot 4, and Spring AI, OpenCrawling serves as the > > > reference implementation of the *Open Ingestion Standard (OIS)* and > > > provides a secure *Model Context Protocol (MCP)* server interface. It > > > orchestrates scalable data flows from heterogeneous enterprise > > repositories > > > (e.g., SharePoint, S3, CMIS/Alfresco, BPMN engines like Camunda and > > > Flowable, Apache Iceberg, Apache Ozone) to downstream vector databases > > > (e.g., Milvus, Qdrant, OpenSearch, Vespa, pgvector) with source-level > > > Access Control List (ACL) security enforcement. > > > > > > ---- > > > > > > h2. Proposal > > > > > > The OpenCrawling community proposes to incubate *OpenCrawling* as a new > > > project within the Apache Software Foundation (ASF). OpenCrawling > > provides > > > a decoupled, vendor-neutral enterprise data integration framework that > > > bridges the gap between traditional enterprise content management (ECM) > > > repositories and modern Large Language Model (LLM) / > Retrieval-Augmented > > > Generation (RAG) architectures. > > > > > > The project encompasses: > > > # *Core Ingestion Runtime ({{oc-core}}, {{oc-runtime}})*: A > distributed, > > > asynchronous engine built with virtual threads, claim-check metadata > > > patterns, and Apache Kafka event streams. > > > # *Repository & Vector Connectors*: Standardized connectors for > scanning > > > source systems and indexing vector embeddings into major vector stores. > > > # *Open Ingestion Standard (OIS)*: The formal JSON/YAML schema > > > specifications defining unified document payloads, ACL security SIDs, > and > > > crawler job configurations. > > > # *Secure Model Context Protocol (MCP) Server*: A Zero-Trust context > > > retrieval server enforcing document-level permissions (Active Directory > > > SIDs, LDAP groups, user principals) at query time. > > > # *Observability & Developer Tooling*: AI-Powered Observability (AIOps) > > > over OpenTelemetry traces, Auto-Narrativization Copilot, Java Client > SDK, > > > and Maven Archetypes for custom connector development. > > > > > > ---- > > > > > > h2. Background > > > > > > In enterprise AI and RAG architectures, LLM agents require seamless > > access > > > to unstructured content stored across legacy and cloud repositories. > > > However, traditional ingestion pipelines often strip out or ignore > > > source-level security metadata (ACLs), leading to context leakage where > > an > > > AI model synthesizes responses using confidential documents that the > > > requesting user does not have permissions to view. > > > > > > Furthermore, legacy crawling tools (such as Apache ManifoldCF or Apache > > > Nutch) were architected over a decade ago prior to the emergence of > > vector > > > databases, LLMs, Model Context Protocol (MCP), and modern Java features > > > like Virtual Threads (JEP 444) and Structured Concurrency. > > > > > > OpenCrawling was created to address this modern ingestion crisis by > > > providing a native Java 25/Spring AI implementation engineered > > specifically > > > for LLM search scenarios, zero-trust context retrieval, and > > high-throughput > > > asynchronous processing. > > > > > > ---- > > > > > > h2. Rationale > > > > > > The Apache Software Foundation is the natural home for OpenCrawling. > ASF > > > has long been the center of innovation for enterprise search and big > data > > > infrastructure, hosting cornerstone projects such as Apache Lucene, > > Apache > > > Solr, Apache Tika, Apache Kafka, Apache Iceberg, Apache Ozone, Apache > > > ManifoldCF, and Apache Nutch. > > > > > > Bringing OpenCrawling to the ASF offers multiple mutual benefits: > > > * *Ecosystem Integration*: OpenCrawling directly integrates with and > > builds > > > upon existing Apache projects, including *Apache Tika* (text > extraction), > > > *Apache Kafka* (event-driven pipeline), *Apache Iceberg* (lakehouse > > > connector), *Apache Ozone* (claim-check object storage), and *Apache > > Maven* > > > (connector archetype distribution). > > > * *Vendor-Neutral Governance*: Neutral governance under the Apache Way > is > > > vital to establishing OpenCrawling and OIS as industry-wide, > > > vendor-agnostic ingestion standards. > > > * *Community Sustainability*: Operating as an Apache project will > > attract a > > > broader community of enterprise adopters, cloud providers, AI framework > > > developers, and search engine vendors. > > > > > > ---- > > > > > > h2. Initial Goals > > > > > > During incubation, the OpenCrawling project will focus on the following > > > milestones: > > > > > > # *ASF Migration & Infrastructure*: > > > ** Transfer codebases ({{opencrawling}}, {{open-ingestion-standard}}, > {{ > > > opencrawling.github.io}}) to Apache infrastructure ({{ > > > github.com/apache/incubator-opencrawling}} > > <http://github.com/apache/incubator-opencrawling%7D%7D>). > > > ** Rebrand build artifacts to {{org.apache.opencrawling}}. > > > ** Setup ASF-compliant CI/CD pipelines using GitHub Actions. > > > # *Community & Governance*: > > > ** Adopt the Apache Way for all decisions, roadmap discussions, and > > release > > > voting. > > > ** Expand the contributor base across independent developers, > enterprise > > > search users, and corporate contributors. > > > # *Ecosystem & Connector Expansion*: > > > ** Release additional output connectors (Elasticsearch, Apache Solr, > > > RESTHeart). > > > ** Add native integration for fine-grained authorization frameworks > > (e.g., > > > OpenFGA). > > > ** Enhance gRPC support for high-efficiency inter-microservice > > > communication. > > > ** Standardize OIS specification drafts under ASF governance. > > > # *Compliance & Licensing*: > > > ** Complete IP clearance and execute software grant agreements. > > > ** Ensure all third-party dependencies strictly conform to Apache > License > > > Category A policies. > > > > > > ---- > > > > > > h2. Current Status > > > > > > h3. Meritocracy > > > The OpenCrawling project was established with meritocratic principles > > from > > > day one. Design decisions, architecture changes, issue tracking, and > > > roadmap discussions take place openly on GitHub through RFCs, Pull > > > Requests, and public wiki pages. > > > > > > h3. Community > > > The OpenCrawling community includes developers and architects from > > > enterprise search, ECM, and AI background. Community channels include > > > GitHub Discussions, Slack, and social media announcements. The project > > > actively encourages external contributions via Maven archetypes and > > modular > > > connector development. > > > > > > h3. Core Developers > > > The initial core developers are experienced software architects and > > > open-source veterans with extensive experience in enterprise search, > > > content management, and ASF governance: > > > > > > * *Piergiorgio Lucidi* ({{[email protected]}}) – Founder, Lead > > > Architect. ASF Member and PMC Member/Committer on multiple Apache > > projects > > > (including Apache ManifoldCF and Apache Chemistry). > > > * *Michael Cizmar* ({{[email protected]}}) – Lead Architect & > > > Developer. Specialist in enterprise search and cloud infrastructure. > > > * *Luis Cabaceira* ({{[email protected]}}) – Lead Architect & > > > Developer. Specialist in document processing and AI integration. > > > > > > ---- > > > > > > h2. Known Risks > > > > > > h3. Orphaned Products > > > The risk of OpenCrawling becoming orphaned is low. The project solves > an > > > active, urgent security and performance problem in enterprise AI > adoption > > > (RAG ACL context leakage). The core maintainers are committed to its > > > long-term evolution and actively use it in production environments. > > > > > > h3. Inexperience with Open Source > > > The project leadership has deep experience with open-source > communities. > > > Piergiorgio Lucidi is an active ASF Member and PMC member with over a > > > decade of experience guiding projects through the Apache Way. > > > > > > h3. Homogenous Developers > > > The initial committers come from diverse geographical locations (Italy, > > > United States, Portugal) and distinct organizations/consultancies. > > > Incubating at Apache will further diversify the developer base by > > > encouraging contributions from enterprise organizations and search > > vendors. > > > > > > h3. Reliance on Third-Party Products > > > OpenCrawling is designed to be vendor-neutral. Core dependencies are > > > open-source libraries under permissive licenses (Apache 2.0, MIT, BSD): > > > * Spring Boot & Spring AI (Apache 2.0) > > > * Apache Tika (Apache 2.0) > > > * Apache Kafka (Apache 2.0) > > > * PostgreSQL / pgvector (PostgreSQL License / MIT) > > > * Docker & OpenTelemetry (Apache 2.0) > > > > > > There are no GPL/AGPL dependencies in the runtime core. > > > > > > h3. Relationship with Sponsored Products / Brand > > > OpenCrawling is an independent project. The name "OpenCrawling" has > been > > > used for the open-source codebase. The trademark will be transferred to > > the > > > Apache Software Foundation upon incubation acceptance. > > > > > > ---- > > > > > > h2. Documentation & Existing Artifacts > > > > > > * *GitHub Organization*: [https://github.com/opencrawling] > > > * *Main Code Base*: {{opencrawling/opencrawling}} > > > * *Specification Repo*: {{opencrawling/open-ingestion-standard}} > > > * *Documentation & Wiki*: [ > > https://github.com/opencrawling/opencrawling/wiki > > > ] > > > * *Java Client SDK*: {{oc-java-client-sdk}} ([Sonatype Central - > > > org.opencrawling:oc-java-client-sdk| > > > > > > https://central.sonatype.com/artifact/org.opencrawling/oc-java-client-sdk > > ]) > > > * *Maven Archetypes*: [Sonatype Central - org.opencrawling.archetypes| > > > > > > https://central.sonatype.com/artifact/org.opencrawling.archetypes/opencrawling-connector-archetypes > > > ] > > > > > > ---- > > > > > > h2. Initial Source & Intellectual Property Submission > > > > > > h3. Initial Source Code > > > The initial codebase to be granted to the ASF resides in the following > > > GitHub repositories: > > > * {{opencrawling/opencrawling}} (Core engine, microservices, UI, > > > connectors, MCP server) > > > * {{opencrawling/open-ingestion-standard}} (JSON schemas, whitepaper, > > > specifications) > > > * {{opencrawling/opencrawling.github.io}} (Project web site and > > > documentation source) > > > > > > All source code is currently licensed under the *Apache License, > Version > > > 2.0*. > > > > > > h3. Software Grant / ICLA / CCLA > > > All core contributors will submit Individual Contributor License > > Agreements > > > (ICLAs) and corporate software grants will be executed upon acceptance > > into > > > the Incubator. > > > > > > ---- > > > > > > h2. External Dependencies > > > > > > All major external dependencies of OpenCrawling use Apache-compatible > > > licenses (Category A): > > > > > > || Dependency || License || > > > | *Java Development Kit (JDK 25)* | GPLv2 + Classpath Exception | > > > | *Spring Boot / Spring AI* | Apache License 2.0 | > > > | *Apache Tika* | Apache License 2.0 | > > > | *Apache Kafka Clients* | Apache License 2.0 | > > > | *Apache Iceberg SDK* | Apache License 2.0 | > > > | *Apache Ozone Client* | Apache License 2.0 | > > > | *Jackson / Slf4j / Logback* | Apache 2.0 / MIT / EPL 1.0 | > > > | *Milvus / Qdrant Java SDKs* | Apache License 2.0 | > > > | *OpenTelemetry Java SDK* | Apache License 2.0 | > > > | *React / Vite / Tailwind (Admin UI)* | MIT | > > > > > > ---- > > > > > > h2. Cryptography > > > > > > OpenCrawling uses standard TLS/HTTPS protocols and hashing routines > > > provided by the standard Java Virtual Machine (JDK) and Spring Security > > > framework for secure transport. It does not include custom > cryptographic > > > algorithms or controlled export software. > > > > > > ---- > > > > > > h2. Required Resources > > > > > > h3. Mailing Lists > > > * {{[email protected]}} > > > * {{[email protected]}} > > > * {{[email protected]}} (PPMC) > > > > > > h3. Git Repositories > > > * {{https://github.com/apache/incubator-opencrawling}} > > > * {{https://github.com/apache/incubator-opencrawling-site}} > > > > > > h3. Issue Tracking > > > * GitHub Issues on {{apache/incubator-opencrawling}} (or ASF Jira > project > > > {{OPENCRAWLING}}) > > > > > > h3. CI/CD Infrastructure > > > * GitHub Actions workflows for automated build, test, multi-arch Docker > > > image generation, and Sonar/Scorecard quality checks. > > > > > > ---- > > > > > > h2. Initial Committers & PPMC Members > > > > > > * *Piergiorgio Lucidi* ({{[email protected]}}) – Initial > Committer > > & > > > PPMC > > > * *Michael Cizmar* ({{[email protected]}}) – Initial > Committer & > > > PPMC > > > * *Luis Cabaceira* ({{[email protected]}}) – Initial Committer > & > > PPMC > > > > > > _(Note: Additional mentors and committers will be welcomed during the > > > discussion period on {{[email protected]}}.)_ > > > > > > ---- > > > > > > h2. Champions & Mentors > > > > > > * *Champion*: Piergiorgio Lucidi ({{[email protected]}}) – ASF > > Member > > > * *Mentors*: > > > ** _(TBD - Interested ASF Members/Incubator PMC members invited to step > > > forward during proposal discussion)_ > > > > > > ---- > > > > > > h2. Sponsoring Entity > > > > > > The *Apache Incubator PMC* is requested to be the sponsoring entity for > > > this project. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > > > > Piergiorgio Lucidi > Mobile: 3395381669 >
