RobertIndie commented on code in PR #21079: URL: https://github.com/apache/pulsar/pull/21079#discussion_r1308547161
########## pip/pip-297.md: ########## @@ -0,0 +1,168 @@ +# Title: Support raising exceptions using Function & Connector context + +# Background knowledge + +The **Pulsar Function** is a serverless computing framework that runs on top of Pulsar and processes messages. + +The **Pulsar IO Connector** is a framework that allows users to easily integrate Pulsar with external systems, such as +databases, messaging systems, and data pipelines. With Pulsar IO Connector, you can create, deploy, and manage +connectors that read data from or write data to Pulsar topics. There are two types of Pulsar IO Connectors: source and +sink. A **source connector** imports data from another system to Pulsar, while a **sink connector** exports data from +Pulsar to another system. The Pulsar IO Connector is implemented based on the Pulsar Function framework. So in +the following, we treat the connector as a special kind of function. The `function` refers to both function and +connector. + +**Function Instance** is a running instance of a Pulsar IO Connector that interacts with a specific external system or a +Pulsar Function that processes messages from the topic. + +**Function Framework** is a framework for running the Function instance. + +**Function Context** is an interface that provides access to various information and resources for the connector or the +function. The function context is passed to the connector or the function when it is initialized, and then can be used +to interact with the Pulsar system. + +The current implementation of the exception handler: +**Function instance thread**: The function framework initializes a thread for each function instance to handle the +core logic of the function/connector, including consuming messages from the Pulsar topic for the sink connector, +executing the logic of the function, producing messages to the Pulsar topic for the source connector, handling the +exception, etc. + +**Exception handling logic**: The function itself can throw exceptions, and this thread will catch the exception and +then close the function. This means that the function will stop working until it is restarted manually or +automatically by the function framework. + +Even though it is not explicitly defined, there are two types of exceptions that could be handled for the function or +the framework. + +- **Recoverable exception**: This is an exception that the function can recover from by itself, such as network + connection issues, persistence issues, etc. The function can catch these exceptions and retry the operation until it + succeeds or reaches a limit. +- **Unrecoverable exception**: This is an exception that the function cannot recover from by itself and needs to notify + the framework to terminate it. These are fatal exceptions that indicate a configuration issue, a logic error, or an + incompatible system. The function framework will catch these exceptions, report them to users, and terminate the + function. Review Comment: I rethink about it. And I feel that `Fatal exception` and `Non-fatal exception` are more appropriate. In this PIP, fatal exception means an exception that could cause the function instance to be terminated. Not all the unretriable exceptions could cause the instance to be terminated. For instance, the pulsar function throws an exception. The record could be negatively ack and the function instance can continue to work. Using `fatal` and `non-fatal` seems clearer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
