Currently there is no limit. It is basically depends on jetty's HashSessionManager. A quick glance through the HashSessionManager doesn't reveal any API to control the max number of sessions. I will keep looking. I just fixed the issue of resource leak. Resources are not released properly when they are expired. I will log a bug in few mins. I think we should get that into the 1.5 release.
Thanks Venki On Thu, Feb 4, 2016 at 1:27 PM, Jason Altekruse <altekruseja...@gmail.com> wrote: > Venki, > > Even for authenticated sessions, do we impose any kind of upper limit on > the number of concurrent connections? I seems like we should issue a > reasonable error message about an over-used Drillbit before the process > dies due to hitting the system file handle limit. > > - Jason > > On Thu, Feb 4, 2016 at 11:00 AM, Josh Schlesser <j...@spoutable.com> > wrote: > > > Thanks Venki, > > Let me know how I can help further. > > > > > On Feb 2, 2016, at 10:23 PM, Venki Korukanti < > venki.koruka...@gmail.com> > > wrote: > > > > > > I think you made valid points. It makes sense to have session less REST > > > calls in both auth enabled and disabled cases. > > > > > > In case of auth enabled: > > > 1) Session-less calls can be authenticated using Basic auth (this was > > > already asked on mailing list sometime back) as a start and move onto > > token > > > based auth later. These requests usually come from non-browsers. The > only > > > issue is setting options before the query. For this we can implement > your > > > suggestion of enhancing the query REST API to accept the options. > > > 2) Session-based call using form auth for browser based access. If we > > > enhance the UI to enter options in the query form, we don't need any > > > session on server actually. > > > > > > I will get a fix ASAP to remove the sessions in anonymous calls as they > > the > > > session are not reused in non-browser cases. > > > > > > Thanks > > > Venki > > > > > > On Tue, Feb 2, 2016 at 9:20 PM, Josh Schlesser <j...@spoutable.com> > > wrote: > > > > > >> No, it wasn’t logging out, it was just stopping, obviously that caused > > >> dangling sessions for the authenticated scenario. > > >> > > >> I don’t think that a short timeout for anonymous sessions is a good > way > > to > > >> go for anonymous api calls. Session management isn’t what anybody > would > > >> expect when using a REST api that is anonymous in a server to server > > >> context. I would expect to use a token for authorization for a > server > > to > > >> server REST api as well. I’m not saying that is what it should be > here, > > >> but that is what my general expectation is based on using other apis. > > In > > >> the case of browser to server REST apis, I have run into > authentication > > for > > >> a browser session and subsequent REST calls leaning on a browser > cookie > > for > > >> persistent authentication. > > >> > > >> Removing sessions for anonymous calls seems like the right path and > > >> possibly easy and I think would be the expected behavior from most > > >> developers. I would advocate for sessionless and token authenticated > > REST > > >> apis for when using authentication for the server to server case and > > cookie > > >> based with a session for the browser to server scenario, but its > really > > the > > >> browser that has a session, not the api per se, its just piggybacking > > on a > > >> regular authenticated web session for the REST api calls. > > >> > > >> This would actually leave me in a quandary for what I am trying to do > > >> which is set a session configuration option ’store.format', but I cant > > >> think of any reason that those types of settings shouldn’t just be set > > on a > > >> per request basis for a REST api. In a server to server context for a > > rest > > >> api, keeping it sessionless means you could front a cluster of > drillbits > > >> with a load balancer and not worry about dying nodes and sticky > sessions > > >> etc... > > >> > > >> I have to get something up and running quickly right now so im > > versioning > > >> back to 1.4 and just spinning up a separate drillbit that will have > the > > >> store.format system variable set to ‘json’ . it will be ok for me > until > > a > > >> good long term solution arrives in drill. > > >> > > >> I’ll run the test on short session_max_idle_secs to 30 seconds on > > >> 1.5.0-SNAPSHOT to see if that gets rid of the file handle starvation > > >> problem, but keep in mind that means that users of the web console > will > > >> have 30 seconds between pages or they have to authenticate again, > which > > >> will probably be very annoying. It doesnt seem like a good long term > > >> solution either. > > >> > > >> How do you think all of this should work? I look forward to staying > > >> involved. > > >> > > >> Cheers, > > >> Josh > > >> > > >>> On Feb 2, 2016, at 4:40 PM, Venki Korukanti < > venki.koruka...@gmail.com > > > > > >> wrote: > > >>> > > >>> When auth is *enabled*, is the worker process logging out after > queries > > >> are > > >>> done? When auth is *disabled* can you set session_max_idle_secs in > > >>> drill.exec.http block in drill-override.conf to something like 30 > > (secs) > > >>> and try? This way anonymous sessions are closed quickly and not kept > > for > > >>> 1hr (default value). I think we may need to avoid creating sessions > in > > >>> anonymous mode (when auth is disabled). > > >>> > > >>> Thanks > > >>> Venki > > >>> > > >>> On Tue, Feb 2, 2016 at 4:02 PM, Josh Schlesser <j...@spoutable.com> > > >> wrote: > > >>> > > >>>> I have a background worker process (on a server, not a browser) that > > >> kicks > > >>>> off every minute or so and issues some queries sequentially to the > > rest > > >>>> query endpoint. In 1.4 with no authentication this worked fine > > except > > >>>> that in 1 instance I need to issue a CTAS query with a different > > format > > >>>> (json). > > >>>> > > >>>> I upgraded to 1.5-SNAPSHOT commit > > >> bb3fc15216d9cab804fc9a6f0e5bd34597dd4394 > > >>>> > > >>>> Since the upgrade I am getting a resource starvation problem with or > > >>>> without authentication > > >>>> The drillbit process stays up for a an hour or less and then becomes > > >>>> unresponsive and eats up the cpu. > > >>>> > > >>>> It is definitely a resource starvation issue, not sure if its a > > resource > > >>>> leak. > > >>>> Below is a stack trace. > > >>>> Also when i lsof on the pid there are a lot (more than a thousand) > of > > >>>> files like this listed which are used by NIO selectors. so it > smells > > >> like > > >>>> a resource leak. > > >>>> > > >>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE > > NAME > > >>>> java 2931 root 288u 0000 0,11 0 7705 > > >>>> anon_inode > > >>>> > > >>>> 2016-02-02 21:56:26,520 [qtp1250890858-11590] ERROR > > >>>> o.a.d.e.s.r.a.AnonymousLoginService - Login failed. > > >>>> java.lang.IllegalStateException: failed to create a child event loop > > >>>> at > > >>>> > > >> > > > io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68) > > >>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final] > > >>>> at > > >>>> > > >> > > > io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49) > > >>>> ~[netty-transport-4.0.27.Final.jar:4.0.27.Final] > > >>>> at > > >>>> > > >> > > > io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61) > > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > > >>>> at > > >>>> > > >> > > > io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49) > > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > > >>>> at > > >>>> > > >> > > > org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73) > > >>>> ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT] > > >>>> at > > >>>> > > >> > > > org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239) > > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT] > > >>>> at > > >>>> > org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220) > > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT] > > >>>> at > > >>>> > org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178) > > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT] > > >>>> at > > >>>> > > >> > > > org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56) > > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT] > > >>>> at > > >>>> > > >> > > > org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47) > > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT] > > >>>> at > > >>>> > > >> > > > org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71) > > >>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503) > > >>>> [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) > > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111) > > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478) > > >>>> [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) > > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045) > > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) > > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at org.eclipse.jetty.server.Server.handle(Server.java:462) > > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) > > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) > > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534) > > >>>> [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) > > >>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) > > >>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91] > > >>>> Caused by: java.lang.RuntimeException: epoll_create1() failed: Too > > many > > >>>> open files > > >>>> at io.netty.channel.epoll.Native.epollCreate(Native Method) > > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > > >>>> at > > >>>> io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74) > > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > > >>>> at > > >>>> > > >> > > > io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76) > > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > > >>>> at > > >>>> > > >> > > > io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64) > > >>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final] > > >>>> ... 25 common frames omitted > > >>>> 2016-02-02 21:56:30,130 [qtp1250890858-11591] ERROR > > >>>> o.a.d.e.s.r.a.AnonymousLoginService - Login failed. > > >>>> java.lang.IllegalStateException: failed to create a child event loop > > >>>> at > > >>>> > > >> > > > io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68) > > >>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final] > > >>>> at > > >>>> > > >> > > > io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49) > > >>>> ~[netty-transport-4.0.27.Final.jar:4.0.27.Final] > > >>>> at > > >>>> > > >> > > > io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61) > > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > > >>>> at > > >>>> > > >> > > > io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49) > > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > > >>>> at > > >>>> > > >> > > > org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73) > > >>>> ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT] > > >>>> at > > >>>> > > >> > > > org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239) > > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT] > > >>>> at > > >>>> > org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220) > > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT] > > >>>> at > > >>>> > org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178) > > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT] > > >>>> at > > >>>> > > >> > > > org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56) > > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT] > > >>>> at > > >>>> > > >> > > > org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47) > > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT] > > >>>> at > > >>>> > > >> > > > org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71) > > >>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503) > > >>>> [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) > > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111) > > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478) > > >>>> [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) > > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045) > > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) > > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at org.eclipse.jetty.server.Server.handle(Server.java:462) > > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) > > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) > > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534) > > >>>> [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) > > >>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at > > >>>> > > >> > > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) > > >>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505] > > >>>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91] > > >>>> Caused by: java.lang.RuntimeException: epoll_create1() failed: Too > > many > > >>>> open files > > >>>> at io.netty.channel.epoll.Native.epollCreate(Native Method) > > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > > >>>> at > > >>>> io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74) > > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > > >>>> at > > >>>> > > >> > > > io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76) > > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > > >>>> at > > >>>> > > >> > > > io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64) > > >>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final] > > >>>> ... 25 common frames omitted > > >>>> > > >>>> > > >>>> > > >>>>> On Feb 2, 2016, at 7:40 AM, Venki Korukanti < > > venki.koruka...@gmail.com > > >>> > > >>>> wrote: > > >>>>> > > >>>>> Currently we keep the DrillClient per session. All the state is in > > >> Server > > >>>>> and DrillClient is the reference to reuse the state. DrillClient is > > >>>>> automatically closed when the session expires (default value is 1hr > > >> after > > >>>>> the last activity on session) or user explicitly logs out. I am > > trying > > >> to > > >>>>> understand if there is a resource leak. Do you have too many > sessions > > >>>> open > > >>>>> when the system load is max or just few sessions but you have > already > > >> ran > > >>>>> many queries using the existing sessions? If it is the former it is > > >>>>> understandable to have per connection per session life. Also are > the > > >>>>> resources not freeing up after logout? > > >>>>> > > >>>>> If you need to have multiple simultaneous sessions, it is better to > > >>>> connect > > >>>>> to different Drillbits (may be in a round-robin fashion) than > always > > >>>>> connecting to a single Drillbit. > > >>>>> > > >>>>> Thanks > > >>>>> Venki > > >>>>> > > >>>>> On Mon, Feb 1, 2016 at 11:51 PM, Josh Schlesser < > j...@spoutable.com > > >>>> <mailto:j...@spoutable.com>> wrote: > > >>>>> > > >>>>>> First: Im a total newb at contributing to apache projects so > please > > >>>> excuse > > >>>>>> any indiscretions, feel free to give comments on style or > whatever, > > i > > >>>> take > > >>>>>> feedback well. Thick skin too. > > >>>>>> > > >>>>>> > > >>>>>> Ill give some background next and then a proposal. > > >>>>>> > > >>>>>> Background: > > >>>>>> I recently changed over to using authentication in the 1.5 > snapshot > > >>>>>> because I need to have a session via the REST api so that I can > set > > >> the > > >>>>>> session storage options in an initial query for a subsequent CTAS > > >> query. > > >>>>>> Previously all rest calls seemed to be completely independent. > > >>>>>> > > >>>>>> Since the change I have started seeing ‘too many files open’ > errors > > in > > >>>> my > > >>>>>> drillbit.log and the drillbit java process becomes effectively > hung > > >>>> waiting > > >>>>>> for open file descriptor slots. When running the top command the > > >>>> machine > > >>>>>> is running at max load due to the drillbit process and the > drillbit > > >>>> becomes > > >>>>>> effectively unresponsive, even the simple pages in the web console > > >> don’t > > >>>>>> respond. Investigating further it seems that there might be a > file > > >>>> kept > > >>>>>> open per session by the drillbit process for the life of the > > session. > > >>>> I > > >>>>>> used the lsof unix command on the drillbit process and found a lot > > of > > >>>> unix > > >>>>>> pipes. Looking at the code it looks like these pipes could be for > > the > > >>>>>> communication between the web process and the rpc server, with one > > >> being > > >>>>>> allocated per session. I haven’t validated this, its just a guess > > >> after > > >>>>>> scanning the code. I had 1.4 running without this requirement > and > > >>>> without > > >>>>>> ever seeing the error. It seems without authentication the number > > of > > >>>> open > > >>>>>> files is a non-issue for me, possibly due to sessions. > > >>>>>> > > >>>>>> I'm wondering if my guess about what is causing the ‘too many open > > >>>> files’ > > >>>>>> error is plausible? Does anybody with a deeper understanding of > > the > > >>>>>> architecture have any comments on this? > > >>>>>> > > >>>>>> Proposal: > > >>>>>> Assuming sessions are the issue, I am making some changes to my > rest > > >>>>>> client so that sessions are more effectively used and I can up the > > >>>> ulimit > > >>>>>> for the drillbit process for the linux user in hopes of mitigating > > >>>> this. I > > >>>>>> am effectively creating a rest client based session pool that > resets > > >>>>>> session variables to defaults when the session gets reused. > > >> However, > > >>>> it > > >>>>>> seems hacky. > > >>>>>> > > >>>>>> Below is an idea for getting per request based settings which > seems > > >> less > > >>>>>> hacky in the long term. > > >>>>>> > > >>>>>> Can I add a new array member to the query.json REST method in a > > >>>> backwards > > >>>>>> compatible way to set session level parameters in a single > request? > > >>>>>> Currently a rest request via the api has a body like so: > > >>>>>> { “queryType”: “SQL”, “query” : “<drill query>”} > > >>>>>> > > >>>>>> id like to do the following > > >>>>>> > > >>>>>> { “queryType”: “SQL”, “query” : “<drill query>”, > “sessionSettings”: > > >>>>>> [“option_1_name”:”option_1_value”, > > “option_2_name”:”option_2_value”]} > > >>>>>> > > >>>>>> or even > > >>>>>> > > >>>>>> { “queryType”: “SQL”, “query” : “<drill query>”, > “sessionSettings”: > > >>>> [“SET > > >>>>>> `option_name` = value”, “SET `option_name1` = value1”,“SET > > >>>> `option_name2` = > > >>>>>> value2”, “SET `option_name3` = value3”]} > > >>>>>> > > >>>>>> As far as I can tell drill is essentially stateless between > queries > > >>>> right > > >>>>>> now except for session level system parameters and authentication. > > >>>> There > > >>>>>> aren’t any in memory temp tables or cursors or variables like > PL/SQL > > >> or > > >>>>>> PSQL or other SQLs that would make it stateful. > > >>>>>> > > >>>>>> Given the stateless assumption, being able to set session level > > params > > >>>> on > > >>>>>> a per request basis would cover all of the cases that I might > need. > > >> It > > >>>>>> looks relatively straight forward to add something to QueryWrapper > > to > > >>>>>> accept an optional query session settings section of the json > packet > > >> and > > >>>>>> execute those ’SET' commands before the final query. This will > > work > > >>>> for > > >>>>>> me, as I can run without authentication in an ’secure' backend > > >>>> environment > > >>>>>> which will remove sessions and hence file descriptors, assuming my > > >>>>>> assumptions about file descriptors and sessions are correct. > > >>>>>> > > >>>>>> > > >>>>>> My java is rusty (circa 2003) but some casual googling implies > that > > if > > >>>>>> this were added as a 3rd @FormParam to submitQuery in > QueryResources > > >> it > > >>>>>> would be magically be null if it werent present and could easily > be > > >>>>>> ignored. If its present then an alternative constructor of > > >> QueryWrapper > > >>>>>> could be called with the extra param and it would be easy to alter > > its > > >>>> run > > >>>>>> method to execute the SET commands. There would need to be some > > error > > >>>>>> handling of course if the SET commands were illegal or failed to > run > > >> for > > >>>>>> some reason. > > >>>>>> > > >>>>>> If this seems reasonable, how do I go about contributing? I > looked > > >>>>>> through the links in the docs to apache foundation incubator > > projects > > >>>> but > > >>>>>> the links to drill were broken :( > > http://drill.apache.org/team.html > > >> < > > >>>>>> http://drill.apache.org/team.html < > > http://drill.apache.org/team.html > > >>>> > > >>>> I read this > > >>>>>> > http://drill.apache.org/docs/apache-drill-contribution-guidelines/ > > < > > >>>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/> > < > > >>>>>> > http://drill.apache.org/docs/apache-drill-contribution-guidelines/> > > >>>> and > > >>>>>> i have subscribed to the dev mailing list (obvious since you are > > >> getting > > >>>>>> this). It said to post here before creating a JIRA. Am I > missing > > >>>>>> anything in my assumptions? Comments? Should I just submit a > JIRA > > >> and > > >>>> a > > >>>>>> patch or submit a JIRA and a comment or wait for comments before > > >> coding > > >>>>>> stuff up as an example? > > >>>>>> > > >>>>>> Thanks for taking the time to read and respond. > > >>>>>> > > >>>>>> Josh > > >>>> > > >>>> > > >> > > >> > > > > >
