Re: [jetty-users] Threading model (was Re: Jetty 12 schedule?_
On Thu, 10 Nov 2022 at 12:29, Cantor, Scott wrote: > >The proper way is to use the request / response objects as passed to > > you, and not hold onto them. > > That would imply that one has to immediate extract every possible bit from > them and construct a new façade for the data before one's servlet method > even calls another object. That's a pretty huge step backward, if true. > Scott, it is typical to handle requests with multi classes, potentially calling deep into a code base. Often these calls will accept the request and/or response as an argument and that is a reasonable style as it makes it very clear that the call is part of a request/response calling cycle. Problems start to occur when a request/response is wrapped in another object that abstracts away from the request response paradigm and just looks like some dataful object (which just happens to be backed by data a request). Such objects can easily be put into persistent data structures that have no lifecycle relationship to the original request. So the antipattern is to facade a request into a non-request like API. If your facades do look like requests and their lifecycle is apparent, then that is a reasonable abstraction. regards -- Greg Wilkins CTO http://webtide.com ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Threading model (was Re: Jetty 12 schedule?_
>The assumption that you'll be passed a HttpServletRequest and/or > HttpServletResponse to these actions isn't 100% true. Ok, I can see that. >Take for example the Async I/O events. Which we will never, ever call/use/allow. >There are similar things elsewhere in the spec. Most of them are in the same bucket, we don't do that kind of thing. > Keep in mind that there is no provision in the Servlet spec for notifying > of the lifecycle of the HttpServletRequest or HttpServletResponse > objects. Understood, but they have to exist for the lifetime of the call to the servlet dispatch method, absent use of other container services that interrupt the processing chain of that method, no? Otherwise it's madness since we couldn't even make it work by passing them around if we wanted to. >The proper way is to use the request / response objects as passed to > you, and not hold onto them. That would imply that one has to immediate extract every possible bit from them and construct a new façade for the data before one's servlet method even calls another object. That's a pretty huge step backward, if true. > As a servlet endpoint, you are expected to get what you need from the > request (headers, body, etc), formulate a response (status code, > headers, etc) and then produce a body. Of course, but that is never (outside trivial samples) one class. It's a chain of many objects and methods doing work. You don't pass the interfaces into every method. And even if you did pass them into every method in a synchronous call chain, you're implying that wouldn't even help. And that seems impossible since I can't even see how you would ever be in the middle of the code to interfere with it. Again, there is no use of async anything whatsoever (once we get control), and there never will be. For exactly this reason. >If you need to track things between dispatches of the same exchange, > it's usually a good idea to use the request attributes to hold onto objects > for the scope of that one request. If I have no HttpServletRequest, how would I possibly put or get them? That doesn't change the situation, really. >Don't forget about normal redispatch behavior. That's worth more study, especially the error case, but that's likely past the point we'd be doing anything unusual depending on these interfaces. But it is a point we need to consider. >Imagine if the APIs contained the request/response objects > everywhere, which ones do we give you if wrapping occurs? (eg: if a > Filter wrapped the request, and added the listener, then the Servlet > wrapped again, and then that listener needed to fire, do we give you the > servlet wrapped request or the filter wrapped request?) Yes, I can see why they had to do things the way they did for the Async APIs. Which we don't use. -- Scott ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Threading model (was Re: Jetty 12 schedule?_
> Is it fair to assume that the only way a request could possibly cross threads like that is if there's also an actually explicit entry point into one's application logic by which the container is actually invoking a standard interface that is defined to accept the HttpServletRequest/Response? The assumption that you'll be passed a HttpServletRequest and/or HttpServletResponse to these actions isn't 100% true. Take for example the Async I/O events. Those are listeners. https://javadoc.io/static/jakarta.servlet/jakarta.servlet-api/5.0.0/jakarta/servlet/ReadListener.html https://javadoc.io/static/jakarta.servlet/jakarta.servlet-api/5.0.0/jakarta/servlet/WriteListener.html You are not expected to be manipulating / accessing / referencing the request and/or response at this point in the process. That should have happened long before you even engaged these listeners. There are similar things elsewhere in the spec. Keep in mind that there is no provision in the Servlet spec for notifying of the lifecycle of the HttpServletRequest or HttpServletResponse objects. If a method existed to know when the original HttpServletRequest and HttpServletResponse was / is / has been ended it's lifecycle (and optionally been recycled by the container), you could possible write code to hold onto the request/response objects safely (but realities of concurrency programming means that this kind of event can only occur AFTER it's done, which is often too late to turn things off) > In other words, I take your point that people can build code that leverages various advanced async/etc. features in the servlet spec, and we don't necessarily know that's happening, but it has to *be* some explicit invocation of a container technology that in turn is going to call back in with the "proper" servlet request/response, right? The proper way is to use the request / response objects as passed to you, and not hold onto them. This even includes objects like HttpSession and the ServletInputStream / ServletOutputStream. As a servlet endpoint, you are expected to get what you need from the request (headers, body, etc), formulate a response (status code, headers, etc) and then produce a body. The async processing exists to delay / suspend actions to later, but that also results in a container managed thread for the AsyncContext, one that the container is aware of and knows the scope/lifecycle of. The async I/O for requests expects that you've gathered what you need from the request url line / headers / etc before you start. The async I/O for responses expects that you've decided what you are going to send, and are setting up to send the body of the response. If you need to track things between dispatches of the same exchange, it's usually a good idea to use the request attributes to hold onto objects for the scope of that one request. > So it's difficult to understand how we could possibly be passed in a different instance than the original one the servlet dispatch received since there's nowhere for it to occur. Don't forget about normal redispatch behavior. An example of this is if you use the RequestDispatcher, this will be a wrapped or altered request/response objects depending on your mode of use (include vs forward). You can also have an Error being processed via a redispatch using the DispatcherType.ERROR (which can appear to be a different request/response, but in reality is a spec mandated cleanup of the request/response before redispatch). In both of those cases, a request attribute works great to track behavior throughout the entire request lifecycle (even redispatch). > Basically it feels like for this to break, something has to call an API that also involves passing in a callback interface that itself has to receive the request/response back. As I showed above, not all listeners / interfaces have the request/response params given to you, the API was created to respect the Filter Chain, DispatcherTypes, and Request/Response Wrappers along with the normal HTTP lifecycle (eg: you can't change response status code or headers if the HttpServletResponse.isCommitted() is true) Imagine if the APIs contained the request/response objects everywhere, which ones do we give you if wrapping occurs? (eg: if a Filter wrapped the request, and added the listener, then the Servlet wrapped again, and then that listener needed to fire, do we give you the servlet wrapped request or the filter wrapped request?) Joakim Erdfelt / joa...@webtide.com On Wed, Nov 9, 2022 at 5:22 PM Cantor, Scott wrote: > >It's a general anti-pattern to hold onto, use, reference a > > HttpServletRequest or HttpServletResponse object outside of the > > dispatch from the container. > > One more question about this I guess... > > Is it fair to assume that the only way a request could possibly cross > threads like that is if there's also an actually explicit entry point into > one's application logic by which the container is actually invoking a >
[jetty-users] Threading model (was Re: Jetty 12 schedule?_
>It's a general anti-pattern to hold onto, use, reference a > HttpServletRequest or HttpServletResponse object outside of the > dispatch from the container. One more question about this I guess... Is it fair to assume that the only way a request could possibly cross threads like that is if there's also an actually explicit entry point into one's application logic by which the container is actually invoking a standard interface that is defined to accept the HttpServletRequest/Response? In other words, I take your point that people can build code that leverages various advanced async/etc. features in the servlet spec, and we don't necessarily know that's happening, but it has to *be* some explicit invocation of a container technology that in turn is going to call back in with the "proper" servlet request/response, right? Otherwise how could one ever know which instance of the request/response interfaces to act on? All we do is implement servlets (or make use of them) that accept a servlet dispatch call in and respond out. Nothing else in our design implements any servlet APIs that can accept a request/response in a different way. So it's difficult to understand how we could possibly be passed in a different instance than the original one the servlet dispatch received since there's nowhere for it to occur. That's why we've struggled to grasp what the risk is as this has come up at times. I fully appreciate that you're speaking from a container/generality perspective here and that in the general case what you're saying is obviously true. And yes, it's an anti-pattern because it makes assumptions that aren't generally true. I'm strongly suspecting they are, however, still true for us (and frankly for the vast majority of traditional apps that just implement Servlet). Basically it feels like for this to break, something has to call an API that also involves passing in a callback interface that itself has to receive the request/response back. -- Scott ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Jetty 12 schedule?
>It's a general anti-pattern to hold onto, use, reference a > HttpServletRequest or HttpServletResponse object outside of the > dispatch from the container. Unfortunately it's likely to be borderline impossible for us to avoid it in any practical way given the layering involved, but we will dig into it further. >Example of the most recent issue they are working: That's helpful, thanks. -- Scott ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Jetty 12 schedule?
> Our undersatanding has been that a given request is still thread-locked from start to finish even though there are changes to the model in how requests are dispatched, as long as one didn't call out to some kind of asynchronous API or something that could suspend the processing of the request. Incorrect assumption. Well ... it is an incomplete or partial assumption really, as it is not limited to only async processing. Example, All the way back to Servlet 3, if any servlet component declares async-supported=true (even if they didn't use async) then your thread-local won't work reliably either across vendors as that was was vaguely defined in the spec, heck the meaning of *not* declaring async-supported, (what is the default?) wasn't even defined. Some vendors took it to mean async-supported=true was default (Jetty took this approach), others took it to mean async-supported=false was default (Tomcat took this approach). It wasn't until Servlet 4 where the default was defined, let alone all of the other near constant corner cases in the async processing layer that crops up with regularity in the servlet spec (re-dispatch, error handling, wrappers, etc). Now with the changes in Servlet 5, a container (like Jetty) is free to use as many threads to service the exchange as it deems appropriate based on the technology you use, there's no longer an expectation of 1 thread per exchange (even your 3rd party libraries and web-fragments can change the behavior without your application being aware of it). This means technology choices (not configuration) within the Servlet spec will have a greater impact on how the threads are being used to service your request. (choices you won't know at initialization until an application starts to use those technologies to handle a request). Example: If a component uses AsyncContext you have multiple threads. (best choice here is to use the AsyncContext to reference the request/response) If a component decides to use Async I/O you have multiple threads as well. (don't hold onto the request/response here, the container will call into your application from different threads to satisfy the async I/O behavior) If a component uses Jakarta WebSockets you have multiple threads (depending on how you send/receive messages the thread count can be extreme and can come fast and furious. And there is an extra requirement to never hold onto the request/response objects in websocket due to http lifecycle reasons). It's a general anti-pattern to hold onto, use, reference a HttpServletRequest or HttpServletResponse object outside of the dispatch from the container. As you can easily create situations of use-after-recycle and super odd behavior (the spring folks hit this with regularity, and have taken steps to revert code that holds onto requests/responses outside of the dispatches, but haven't completed this effort). Example of the most recent issue they are working: https://github.com/spring-projects/spring-security/issues/9175 Joakim Erdfelt / joa...@webtide.com On Wed, Nov 9, 2022 at 3:29 PM Cantor, Scott wrote: > Note that our understanding of the SingleThreadedModel is NOT that it > changes how a given request is handled, just that it means a servlet is > forced to allow for multiple requests at once. > > That is not an issue for our code, we're fully thread-safe/aware and that > doesn't break thread-local storage. That's literally how you often make > things thread-safe in fact. > > -- Scott > > > ___ > jetty-users mailing list > jetty-users@eclipse.org > To unsubscribe from this list, visit > https://www.eclipse.org/mailman/listinfo/jetty-users > ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Jetty 12 schedule?
Note that our understanding of the SingleThreadedModel is NOT that it changes how a given request is handled, just that it means a servlet is forced to allow for multiple requests at once. That is not an issue for our code, we're fully thread-safe/aware and that doesn't break thread-local storage. That's literally how you often make things thread-safe in fact. -- Scott ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Jetty 12 schedule?
>You cannot use thread-local reliably starting in Jakarta Servlet 5. Our undersatanding has been that a given request is still thread-locked from start to finish even though there are changes to the model in how requests are dispatched, as long as one didn't call out to some kind of asynchronous API or something that could suspend the processing of the request. We have looked into the issue and couldn't find any statements that a given servlet request is allowed to switch threads mid-request or how that could possibly make any sense. How would the container even get a chance to do so? But to be fair, I have expressed concerns about it and we have been aware that there are changes supposedly coming aorund that. Every time we look, we can't find the clarity we need about what it actually means in practice. And if it breaks, we'll be stuck because I don't think we have any other practical options. If you have some kind of information somewhere you could point us to about what it actually means that a thread can be swapped out or how/when that could possibly happen, it would be useful. -- Scott ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Jetty 12 schedule?
> The only practical way to inject the per-request interfaces into various objects is to use thread-local storage to capture and stash the current Servlet request/response and then inject a façade that pulls the thread-local copy out and delegates the method calls to it. Jakarta Servlet 5 deprecated jakarta.servlet.SingleThreadModel, and Servlet 6 removed it. You cannot use thread-local reliably starting in Jakarta Servlet 5. Joakim Erdfelt / joa...@webtide.com On Wed, Nov 9, 2022 at 3:08 PM Cantor, Scott wrote: > >Scott, I'm not understanding how your facades work or why exactly > > they are needed? Can you give some examples? > > We have a deeply modular Spring-based system, where beans very "high" up > need access to request/response state. The only practical way to inject the > per-request interfaces into various objects is to use thread-local storage > to capture and stash the current Servlet request/response and then inject a > façade that pulls the thread-local copy out and delegates the method calls > to it. > > This is, among other things, a way to allow Spring ot instantiate > components from third-party libraries that depend on Servlet APIs. > > It's not to say we can't come up with other ways, but it's very widespread > in our current design. > > > Why can you not just deploy old javax. * code to one context and new > > jakarta. * code to a different one. > > You're focused on javax/jakarta but that's not the issue. We already > ported to jakarta. Both Servlet 5 and 6 are jakarta. The problem is they're > not the same/compatible API, there are breaking changes in 6 removing > deprecated methods that have been around a while. > > -- Scott > > > ___ > jetty-users mailing list > jetty-users@eclipse.org > To unsubscribe from this list, visit > https://www.eclipse.org/mailman/listinfo/jetty-users > ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Jetty 12 schedule?
>The best advice I can offer is that your Servlet 5 facade should not > implement a facade for any API that is deprecated. Our early attempts at that failed, but we're not done bottoming out what we might be able to get to work. I appreciate the suggestions, though. We really weren't after anything but a rough idea of whether 12 will be out soon enough to just move up to Servlet 6 and be done with it. >Once you do that, it gets much much easier. Yeah, that was the first direction we went in but it hasn't worked out quite yet. >Do you have an open source project where you have these facades > created? Yes, it's part of the libraries at the bottom layer of the Shibboleth software, underneath OpenSAML. One thing we did do was move towards a Supplier/etc. model where we don't actually try and "fool" the components into thinking we actually implement the interfaces, but that only works when we control the component signature. -- Scott ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Jetty 12 schedule?
>Scott, I'm not understanding how your facades work or why exactly > they are needed? Can you give some examples? We have a deeply modular Spring-based system, where beans very "high" up need access to request/response state. The only practical way to inject the per-request interfaces into various objects is to use thread-local storage to capture and stash the current Servlet request/response and then inject a façade that pulls the thread-local copy out and delegates the method calls to it. This is, among other things, a way to allow Spring ot instantiate components from third-party libraries that depend on Servlet APIs. It's not to say we can't come up with other ways, but it's very widespread in our current design. > Why can you not just deploy old javax. * code to one context and new > jakarta. * code to a different one. You're focused on javax/jakarta but that's not the issue. We already ported to jakarta. Both Servlet 5 and 6 are jakarta. The problem is they're not the same/compatible API, there are breaking changes in 6 removing deprecated methods that have been around a while. -- Scott ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Jetty 12 schedule?
> There are, though, two incompatible jakarta-namespaced APIs, Servlet 5 and 6. That's where we're having issues because we implement facades for the APIs that can't load on both at once. > > We can implement separate ones for 5 and 6 (and thus Jetty 11 and 12) but we're not sure how we'll manage it if we do. The best advice I can offer is that your Servlet 5 facade should not implement a facade for any API that is deprecated. Once you do that, it gets much much easier. Do you have an open source project where you have these facades created? Joakim Erdfelt / joa...@webtide.com ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Jetty 12 schedule?
Scott, I'm not understanding how your facades work or why exactly they are needed? Can you give some examples? Why can you not just deploy old javax.* code to one context and new jakarta.* code to a different one. cheers On Thu, 10 Nov 2022 at 07:58, Cantor, Scott wrote: > >jetty-12 is dependent on neither namespace, but can deploy contexts > > that are dependent on javax.* or jakarta.* > > There are, though, two incompatible jakarta-namespaced APIs, Servlet 5 and > 6. That's where we're having issues because we implement facades for the > APIs that can't load on both at once. > > We can implement separate ones for 5 and 6 (and thus Jetty 11 and 12) but > we're not sure how we'll manage it if we do. > > We haven't really looked into how 12 is going to do what it does yet, > we're just assessing options and knowing when 12 will likely come out helps > with our planning. > > Most projects are just worried about the APIs they *call* but when you > also implement HttpServletRequest/Response, it's not that simple obviously, > and we had no idea Servlet 6.0 actually removed things from 5.0 until today. > > -- Scott > > > ___ > jetty-users mailing list > jetty-users@eclipse.org > To unsubscribe from this list, visit > https://www.eclipse.org/mailman/listinfo/jetty-users > -- Greg Wilkins CTO http://webtide.com ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Jetty 12 schedule?
>jetty-12 is dependent on neither namespace, but can deploy contexts > that are dependent on javax.* or jakarta.* There are, though, two incompatible jakarta-namespaced APIs, Servlet 5 and 6. That's where we're having issues because we implement facades for the APIs that can't load on both at once. We can implement separate ones for 5 and 6 (and thus Jetty 11 and 12) but we're not sure how we'll manage it if we do. We haven't really looked into how 12 is going to do what it does yet, we're just assessing options and knowing when 12 will likely come out helps with our planning. Most projects are just worried about the APIs they *call* but when you also implement HttpServletRequest/Response, it's not that simple obviously, and we had no idea Servlet 6.0 actually removed things from 5.0 until today. -- Scott ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Jetty 12 schedule?
On Thu, 10 Nov 2022 at 07:08, Cantor, Scott wrote: > > I'm refering specifically to the fact that Servlet 5.0 and 6.0, in the > Jakarta namespace, are not compatible because they've removed some > deprecated methods and interfaces and added others. Implementing both in > some system is problematic, at minimum. > > We haven't explored every trick of course. > Scott, jetty-10 is dependent on the javax.* namespace jetty-11 is dependent on the jakarta.* namespace jetty-12 is dependent on neither namespace, but can deploy contexts that are dependent on javax.* or jakarta.* However, you cannot mix APIs within the one context. regards -- Greg Wilkins CTO http://webtide.com ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Jetty 12 schedule?
>If you are asking about supporting jakarta. * servlets, that is Jetty 11. Nope. That was our plan but we have facades that implement the servlet interfaces by delegating to the container's, and that doesn't work because... > Jetty 12 will support multiple servlet versions. I'm refering specifically to the fact that Servlet 5.0 and 6.0, in the Jakarta namespace, are not compatible because they've removed some deprecated methods and interfaces and added others. Implementing both in some system is problematic, at minimum. We haven't explored every trick of course. -- Scott ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Jetty 12 schedule?
If you are asking about supporting jakarta.* servlets, that is Jetty 11. Jetty 12 will support multiple servlet versions. cheers, Jesse -- jesse mcconnell jesse.mcconn...@gmail.com On Wed, Nov 9, 2022 at 2:00 PM Joakim Erdfelt wrote: > Jetty 12 already has Alpha quality releases out that you can use to start > experimenting with. > > We should be going to Betas (or Release candidates) in the next few weeks. > > Joakim Erdfelt / joa...@webtide.com > > > On Wed, Nov 9, 2022 at 12:30 PM Cantor, Scott wrote: > >> Please forgive this, as I despise "when's it done?" questions, but we >> just realized there are breaking changes in Servlet 6 vs 5 and we have >> expsoure to that so it's starting to become a possibility we'll have to >> move to 6 or do a lot of ugly stuff. >> >> Given Tomcat's already moved, we're just wondering how long 12's likely >> to be incubating so we can plan. >> >> Thanks and apologies, >> -- Scott >> >> >> ___ >> jetty-users mailing list >> jetty-users@eclipse.org >> To unsubscribe from this list, visit >> https://www.eclipse.org/mailman/listinfo/jetty-users >> > ___ > jetty-users mailing list > jetty-users@eclipse.org > To unsubscribe from this list, visit > https://www.eclipse.org/mailman/listinfo/jetty-users > ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Jetty 12 schedule?
Jetty 12 already has Alpha quality releases out that you can use to start experimenting with. We should be going to Betas (or Release candidates) in the next few weeks. Joakim Erdfelt / joa...@webtide.com On Wed, Nov 9, 2022 at 12:30 PM Cantor, Scott wrote: > Please forgive this, as I despise "when's it done?" questions, but we just > realized there are breaking changes in Servlet 6 vs 5 and we have expsoure > to that so it's starting to become a possibility we'll have to move to 6 or > do a lot of ugly stuff. > > Given Tomcat's already moved, we're just wondering how long 12's likely to > be incubating so we can plan. > > Thanks and apologies, > -- Scott > > > ___ > jetty-users mailing list > jetty-users@eclipse.org > To unsubscribe from this list, visit > https://www.eclipse.org/mailman/listinfo/jetty-users > ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
[jetty-users] Jetty 12 schedule?
Please forgive this, as I despise "when's it done?" questions, but we just realized there are breaking changes in Servlet 6 vs 5 and we have expsoure to that so it's starting to become a possibility we'll have to move to 6 or do a lot of ugly stuff. Given Tomcat's already moved, we're just wondering how long 12's likely to be incubating so we can plan. Thanks and apologies, -- Scott ___ jetty-users mailing list jetty-users@eclipse.org To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
