Brian, Comments inline.
On Fri, Nov 1, 2019 at 8:01 PM Brian Candler <[email protected]> wrote: > A few other function and state questions. When there is existing data > in a topic: > > 1. Does a newly-deployed function start consuming from the start of a > topic, or the end? (Or is this selectable?) > A function will place a subscription on input topics it consumes from. By default, a newly-deployed function will create a subscription starting from latest. You can get around by using pulsar-admin to reset the cursor. There is a pull request to expose a setting to specify the initial position when creating a function. https://github.com/apache/pulsar/pull/5532 > > 2. Can a newly-deployed function be told to consume the compacted > version of a topic? > In theory yes. However this is not exposed as well. Can you create a github issue for this? > > 3. Is it possible to "rewind" the subscription of a deployed function to > a previous point in time? > Yes. You can use `pulsar-admin topics reset-cursor` to rewind the subscription. > > 3b. If so, will this also reset stored state to how it was at that point > in time? > Unfortunately you can only rewind the subscription of the input topics. In order to also support rewinding stored state, we have to introduce checkpointing mechanism to Pulsar Functions. > > 4. From the command line, can I wipe *all* the state stored for a function? > The mechanism of wiping the state is in place. You can use bk tool to do so. However it might be making more sense add a command to pulsar-admin. > > 5. Can I list the stored state keys, so I can iterate over them? (Aside: > I looked in the REST API documentation, but it doesn't seem to cover > administering functions at all) > > The mechanism is also in place. We can also add that. > > AFAICS, functions have subscription(s) to their input topic(s); indeed I > can see them in the output of "pulsar-admin topics subscriptions > <topicname>" > > So for (3) I guess it's possible to use "reset-cursor" on the > subscription - but I'm not sure if that's safe/recommended with pulsar > functions. It surely wouldn't update state though (3b). > The reason for question (3b) is I wonder if it's possible to change the > logic in a function (e.g. make a logic fix) and re-run it over the last > hour/day/whatever, with the state back as it was then. That avoids > having to re-run from the beginning of time. > Thanks, > > Brian. > >
