Hi tison, Thank you so much for publishing the proposal and for trying to solve this but I'm continuing to have the same issue. I tried to create a blank page giving Proposals as the main page but I see the same permission error.
Cheers, PG Il giorno mer 12 ago 2026 alle ore 03:00 tison <[email protected]> ha scritto: > Hi Piergiorgio, > > I've invited you to the page. But I'm unfamiliar with Confluence, so it may > not be what you're looking for. > > Alternatively, you may find > https://selfserve.apache.org/confluence-account.html helps. > > Anyway, I created the proposal page [1] with the wiki page content you > shared. > > [1] > > https://cwiki.apache.org/confluence/spaces/INCUBATOR/pages/446071456/OpenCrawling+Proposal > > Best, > tison. > > > Piergiorgio Lucidi <[email protected]> 于2026年8月12日周三 03:36写道： > > > Hi PJ, > > > > Thank you for the feedback and for highlighting Apache StormCrawler. > > > > I completely agree that exploring collaboration between the two projects > is > > a great idea. > > > > While StormCrawler is an incredibly powerful tool for large scale web > > crawling and processing, OpenCrawling was built specifically to tackle > the > > enterprise content and process automation ecosystem. > > > > Our core focus is bridging platforms like Alfresco, Flowable, and Camunda > > directly into modern LLM and RAG architectures, heavily utilizing Spring > > Boot and Spring AI. > > > > Despite the distinct use cases, web data versus enterprise repositories, > > there is absolutely a shared interest in robust data ingestion, document > > parsing and vector database integration strategies. > > > > OpenCrawling can become the perfect home for any crawling strategy. > > > > I would be thrilled to connect with the StormCrawler community to see how > > our projects might complement each other and share best practices moving > > forward. > > > > We could propose to implement a brand new OpenCrawling Storm Bolt. > > > > In Apache Storm topology, data flows from *Spouts* (URL queues) to > *Bolts* > > (fetchers, parsers and indexers). The most seamless integration is to > build > > an opencrawling-storm-bolt. > > > > StormCrawler handles the heavy lifting of recursive web crawling, > > politeness and HTML parsing. > > > > Instead of using StormCrawler's native OpenSearch/Elasticsearch indexer > > bolt, the topology passes the parsed document to the new custom > > OpenCrawling Storm Bolt. > > > > The Bolt acts as an OpenCrawling Repository Connector. It takes the raw > > HTML, wraps it in the Open Ingestion Standard (OIS) format, attaches any > > relevant baseline metadata and pushes it through OpenCrawling’s secure > > pipeline (Java 25 / Spring Boot 4) for chunking, embedding, and Vector DB > > ingestion. > > > > Cheers, > > > > PG > > > > Il Mar 11 Ago 2026, 18:44 PJ Fanning <[email protected]> ha scritto: > > > > > This doesn't block OpenCrawling joining as an ASF Incubator podling > > > but I just want to highlight that there is already Apache > > > StormCrawler. > > > > > > https://stormcrawler.apache.org/ > > > > > > It would be great if these projects could collaborate in areas of > > > shared interest. > > > > > > On Tue, 11 Aug 2026 at 15:28, Piergiorgio Lucidi < > [email protected] > > > > > > wrote: > > > > > > > > Hi everyone, > > > > > > > > I would like to propose OpenCrawling as a new project for incubation > > > within > > > > the Apache Software Foundation. Currently the project is hosted on > > GitHub > > > > [1]. > > > > > > > > The official proposal is currently available in the OpenCrawling Wiki > > in > > > > markdown format [2]. > > > > > > > > I tried to share the proposal in our Confluence but it seems that I > > don't > > > > have permission to create the new page under the Proposals page. > Anyway > > > if > > > > someone can guide me on resolving this issue it would be great! > > > > > > > > Once space permissions are granted on cwiki.apache.org, I will also > > > mirror > > > > the proposal on the Incubator CWIKI proposals page. > > > > Below you also find the same proposal ready to be copy-pasted into > our > > > > Confluence. > > > > > > > > We welcome feedback, questions and discussion from the Incubator > > > community! > > > > > > > > Best regards, > > > > Piergiorgio > > > > On behalf of the OpenCrawling Core Team > > > > > > > > [1] - https://github.com/opencrawling/opencrawling > > > > [2] - > > > > > > > > > > https://github.com/opencrawling/opencrawling/wiki/Apache-Incubator-Proposal > > > > > > > > -------------------------------------------------------------- > > > > > > > > h1. Apache Incubator Proposal: OpenCrawling > > > > > > > > h2. Abstract > > > > > > > > *OpenCrawling* is an open-source, enterprise-grade, high-performance > > data > > > > crawling, content ingestion, and security-aware vector search > platform. > > > > Built on modern Java 25 (leveraging Virtual Threads and Structured > > > > Concurrency), Spring Boot 4, and Spring AI, OpenCrawling serves as > the > > > > reference implementation of the *Open Ingestion Standard (OIS)* and > > > > provides a secure *Model Context Protocol (MCP)* server interface. It > > > > orchestrates scalable data flows from heterogeneous enterprise > > > repositories > > > > (e.g., SharePoint, S3, CMIS/Alfresco, BPMN engines like Camunda and > > > > Flowable, Apache Iceberg, Apache Ozone) to downstream vector > databases > > > > (e.g., Milvus, Qdrant, OpenSearch, Vespa, pgvector) with source-level > > > > Access Control List (ACL) security enforcement. > > > > > > > > ---- > > > > > > > > h2. Proposal > > > > > > > > The OpenCrawling community proposes to incubate *OpenCrawling* as a > new > > > > project within the Apache Software Foundation (ASF). OpenCrawling > > > provides > > > > a decoupled, vendor-neutral enterprise data integration framework > that > > > > bridges the gap between traditional enterprise content management > (ECM) > > > > repositories and modern Large Language Model (LLM) / > > Retrieval-Augmented > > > > Generation (RAG) architectures. > > > > > > > > The project encompasses: > > > > # *Core Ingestion Runtime ({{oc-core}}, {{oc-runtime}})*: A > > distributed, > > > > asynchronous engine built with virtual threads, claim-check metadata > > > > patterns, and Apache Kafka event streams. > > > > # *Repository & Vector Connectors*: Standardized connectors for > > scanning > > > > source systems and indexing vector embeddings into major vector > stores. > > > > # *Open Ingestion Standard (OIS)*: The formal JSON/YAML schema > > > > specifications defining unified document payloads, ACL security SIDs, > > and > > > > crawler job configurations. > > > > # *Secure Model Context Protocol (MCP) Server*: A Zero-Trust context > > > > retrieval server enforcing document-level permissions (Active > Directory > > > > SIDs, LDAP groups, user principals) at query time. > > > > # *Observability & Developer Tooling*: AI-Powered Observability > (AIOps) > > > > over OpenTelemetry traces, Auto-Narrativization Copilot, Java Client > > SDK, > > > > and Maven Archetypes for custom connector development. > > > > > > > > ---- > > > > > > > > h2. Background > > > > > > > > In enterprise AI and RAG architectures, LLM agents require seamless > > > access > > > > to unstructured content stored across legacy and cloud repositories. > > > > However, traditional ingestion pipelines often strip out or ignore > > > > source-level security metadata (ACLs), leading to context leakage > where > > > an > > > > AI model synthesizes responses using confidential documents that the > > > > requesting user does not have permissions to view. > > > > > > > > Furthermore, legacy crawling tools (such as Apache ManifoldCF or > Apache > > > > Nutch) were architected over a decade ago prior to the emergence of > > > vector > > > > databases, LLMs, Model Context Protocol (MCP), and modern Java > features > > > > like Virtual Threads (JEP 444) and Structured Concurrency. > > > > > > > > OpenCrawling was created to address this modern ingestion crisis by > > > > providing a native Java 25/Spring AI implementation engineered > > > specifically > > > > for LLM search scenarios, zero-trust context retrieval, and > > > high-throughput > > > > asynchronous processing. > > > > > > > > ---- > > > > > > > > h2. Rationale > > > > > > > > The Apache Software Foundation is the natural home for OpenCrawling. > > ASF > > > > has long been the center of innovation for enterprise search and big > > data > > > > infrastructure, hosting cornerstone projects such as Apache Lucene, > > > Apache > > > > Solr, Apache Tika, Apache Kafka, Apache Iceberg, Apache Ozone, Apache > > > > ManifoldCF, and Apache Nutch. > > > > > > > > Bringing OpenCrawling to the ASF offers multiple mutual benefits: > > > > * *Ecosystem Integration*: OpenCrawling directly integrates with and > > > builds > > > > upon existing Apache projects, including *Apache Tika* (text > > extraction), > > > > *Apache Kafka* (event-driven pipeline), *Apache Iceberg* (lakehouse > > > > connector), *Apache Ozone* (claim-check object storage), and *Apache > > > Maven* > > > > (connector archetype distribution). > > > > * *Vendor-Neutral Governance*: Neutral governance under the Apache > Way > > is > > > > vital to establishing OpenCrawling and OIS as industry-wide, > > > > vendor-agnostic ingestion standards. > > > > * *Community Sustainability*: Operating as an Apache project will > > > attract a > > > > broader community of enterprise adopters, cloud providers, AI > framework > > > > developers, and search engine vendors. > > > > > > > > ---- > > > > > > > > h2. Initial Goals > > > > > > > > During incubation, the OpenCrawling project will focus on the > following > > > > milestones: > > > > > > > > # *ASF Migration & Infrastructure*: > > > > ** Transfer codebases ({{opencrawling}}, {{open-ingestion-standard}}, > > {{ > > > > opencrawling.github.io}}) to Apache infrastructure ({{ > > > > github.com/apache/incubator-opencrawling}} > > > <http://github.com/apache/incubator-opencrawling%7D%7D>). > > > > ** Rebrand build artifacts to {{org.apache.opencrawling}}. > > > > ** Setup ASF-compliant CI/CD pipelines using GitHub Actions. > > > > # *Community & Governance*: > > > > ** Adopt the Apache Way for all decisions, roadmap discussions, and > > > release > > > > voting. > > > > ** Expand the contributor base across independent developers, > > enterprise > > > > search users, and corporate contributors. > > > > # *Ecosystem & Connector Expansion*: > > > > ** Release additional output connectors (Elasticsearch, Apache Solr, > > > > RESTHeart). > > > > ** Add native integration for fine-grained authorization frameworks > > > (e.g., > > > > OpenFGA). > > > > ** Enhance gRPC support for high-efficiency inter-microservice > > > > communication. > > > > ** Standardize OIS specification drafts under ASF governance. > > > > # *Compliance & Licensing*: > > > > ** Complete IP clearance and execute software grant agreements. > > > > ** Ensure all third-party dependencies strictly conform to Apache > > License > > > > Category A policies. > > > > > > > > ---- > > > > > > > > h2. Current Status > > > > > > > > h3. Meritocracy > > > > The OpenCrawling project was established with meritocratic principles > > > from > > > > day one. Design decisions, architecture changes, issue tracking, and > > > > roadmap discussions take place openly on GitHub through RFCs, Pull > > > > Requests, and public wiki pages. > > > > > > > > h3. Community > > > > The OpenCrawling community includes developers and architects from > > > > enterprise search, ECM, and AI background. Community channels include > > > > GitHub Discussions, Slack, and social media announcements. The > project > > > > actively encourages external contributions via Maven archetypes and > > > modular > > > > connector development. > > > > > > > > h3. Core Developers > > > > The initial core developers are experienced software architects and > > > > open-source veterans with extensive experience in enterprise search, > > > > content management, and ASF governance: > > > > > > > > * *Piergiorgio Lucidi* ({{[email protected]}}) – Founder, Lead > > > > Architect. ASF Member and PMC Member/Committer on multiple Apache > > > projects > > > > (including Apache ManifoldCF and Apache Chemistry). > > > > * *Michael Cizmar* ({{[email protected]}}) – Lead Architect > & > > > > Developer. Specialist in enterprise search and cloud infrastructure. > > > > * *Luis Cabaceira* ({{[email protected]}}) – Lead Architect & > > > > Developer. Specialist in document processing and AI integration. > > > > > > > > ---- > > > > > > > > h2. Known Risks > > > > > > > > h3. Orphaned Products > > > > The risk of OpenCrawling becoming orphaned is low. The project solves > > an > > > > active, urgent security and performance problem in enterprise AI > > adoption > > > > (RAG ACL context leakage). The core maintainers are committed to its > > > > long-term evolution and actively use it in production environments. > > > > > > > > h3. Inexperience with Open Source > > > > The project leadership has deep experience with open-source > > communities. > > > > Piergiorgio Lucidi is an active ASF Member and PMC member with over a > > > > decade of experience guiding projects through the Apache Way. > > > > > > > > h3. Homogenous Developers > > > > The initial committers come from diverse geographical locations > (Italy, > > > > United States, Portugal) and distinct organizations/consultancies. > > > > Incubating at Apache will further diversify the developer base by > > > > encouraging contributions from enterprise organizations and search > > > vendors. > > > > > > > > h3. Reliance on Third-Party Products > > > > OpenCrawling is designed to be vendor-neutral. Core dependencies are > > > > open-source libraries under permissive licenses (Apache 2.0, MIT, > BSD): > > > > * Spring Boot & Spring AI (Apache 2.0) > > > > * Apache Tika (Apache 2.0) > > > > * Apache Kafka (Apache 2.0) > > > > * PostgreSQL / pgvector (PostgreSQL License / MIT) > > > > * Docker & OpenTelemetry (Apache 2.0) > > > > > > > > There are no GPL/AGPL dependencies in the runtime core. > > > > > > > > h3. Relationship with Sponsored Products / Brand > > > > OpenCrawling is an independent project. The name "OpenCrawling" has > > been > > > > used for the open-source codebase. The trademark will be transferred > to > > > the > > > > Apache Software Foundation upon incubation acceptance. > > > > > > > > ---- > > > > > > > > h2. Documentation & Existing Artifacts > > > > > > > > * *GitHub Organization*: [https://github.com/opencrawling] > > > > * *Main Code Base*: {{opencrawling/opencrawling}} > > > > * *Specification Repo*: {{opencrawling/open-ingestion-standard}} > > > > * *Documentation & Wiki*: [ > > > https://github.com/opencrawling/opencrawling/wiki > > > > ] > > > > * *Java Client SDK*: {{oc-java-client-sdk}} ([Sonatype Central - > > > > org.opencrawling:oc-java-client-sdk| > > > > > > > > > > https://central.sonatype.com/artifact/org.opencrawling/oc-java-client-sdk > > > ]) > > > > * *Maven Archetypes*: [Sonatype Central - > org.opencrawling.archetypes| > > > > > > > > > > https://central.sonatype.com/artifact/org.opencrawling.archetypes/opencrawling-connector-archetypes > > > > ] > > > > > > > > ---- > > > > > > > > h2. Initial Source & Intellectual Property Submission > > > > > > > > h3. Initial Source Code > > > > The initial codebase to be granted to the ASF resides in the > following > > > > GitHub repositories: > > > > * {{opencrawling/opencrawling}} (Core engine, microservices, UI, > > > > connectors, MCP server) > > > > * {{opencrawling/open-ingestion-standard}} (JSON schemas, whitepaper, > > > > specifications) > > > > * {{opencrawling/opencrawling.github.io}} (Project web site and > > > > documentation source) > > > > > > > > All source code is currently licensed under the *Apache License, > > Version > > > > 2.0*. > > > > > > > > h3. Software Grant / ICLA / CCLA > > > > All core contributors will submit Individual Contributor License > > > Agreements > > > > (ICLAs) and corporate software grants will be executed upon > acceptance > > > into > > > > the Incubator. > > > > > > > > ---- > > > > > > > > h2. External Dependencies > > > > > > > > All major external dependencies of OpenCrawling use Apache-compatible > > > > licenses (Category A): > > > > > > > > || Dependency || License || > > > > | *Java Development Kit (JDK 25)* | GPLv2 + Classpath Exception | > > > > | *Spring Boot / Spring AI* | Apache License 2.0 | > > > > | *Apache Tika* | Apache License 2.0 | > > > > | *Apache Kafka Clients* | Apache License 2.0 | > > > > | *Apache Iceberg SDK* | Apache License 2.0 | > > > > | *Apache Ozone Client* | Apache License 2.0 | > > > > | *Jackson / Slf4j / Logback* | Apache 2.0 / MIT / EPL 1.0 | > > > > | *Milvus / Qdrant Java SDKs* | Apache License 2.0 | > > > > | *OpenTelemetry Java SDK* | Apache License 2.0 | > > > > | *React / Vite / Tailwind (Admin UI)* | MIT | > > > > > > > > ---- > > > > > > > > h2. Cryptography > > > > > > > > OpenCrawling uses standard TLS/HTTPS protocols and hashing routines > > > > provided by the standard Java Virtual Machine (JDK) and Spring > Security > > > > framework for secure transport. It does not include custom > > cryptographic > > > > algorithms or controlled export software. > > > > > > > > ---- > > > > > > > > h2. Required Resources > > > > > > > > h3. Mailing Lists > > > > * {{[email protected]}} > > > > * {{[email protected]}} > > > > * {{[email protected]}} (PPMC) > > > > > > > > h3. Git Repositories > > > > * {{https://github.com/apache/incubator-opencrawling}} > > > > * {{https://github.com/apache/incubator-opencrawling-site}} > > > > > > > > h3. Issue Tracking > > > > * GitHub Issues on {{apache/incubator-opencrawling}} (or ASF Jira > > project > > > > {{OPENCRAWLING}}) > > > > > > > > h3. CI/CD Infrastructure > > > > * GitHub Actions workflows for automated build, test, multi-arch > Docker > > > > image generation, and Sonar/Scorecard quality checks. > > > > > > > > ---- > > > > > > > > h2. Initial Committers & PPMC Members > > > > > > > > * *Piergiorgio Lucidi* ({{[email protected]}}) – Initial > > Committer > > > & > > > > PPMC > > > > * *Michael Cizmar* ({{[email protected]}}) – Initial > > Committer & > > > > PPMC > > > > * *Luis Cabaceira* ({{[email protected]}}) – Initial > Committer > > & > > > PPMC > > > > > > > > _(Note: Additional mentors and committers will be welcomed during the > > > > discussion period on {{[email protected]}}.)_ > > > > > > > > ---- > > > > > > > > h2. Champions & Mentors > > > > > > > > * *Champion*: Piergiorgio Lucidi ({{[email protected]}}) – ASF > > > Member > > > > * *Mentors*: > > > > ** _(TBD - Interested ASF Members/Incubator PMC members invited to > step > > > > forward during proposal discussion)_ > > > > > > > > ---- > > > > > > > > h2. Sponsoring Entity > > > > > > > > The *Apache Incubator PMC* is requested to be the sponsoring entity > for > > > > this project. > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > Piergiorgio Lucidi > > Mobile: 3395381669 > > > -- Piergiorgio
