Currently we keep the DrillClient per session. All the state is in Server
and DrillClient is the reference to reuse the state. DrillClient is
automatically closed when the session expires (default value is 1hr after
the last activity on session) or user explicitly logs out. I am trying to
understand if there is a resource leak. Do you have too many sessions open
when the system load is max or just few sessions but you have already ran
many queries using the existing sessions? If it is the former it is
understandable to have per connection per session life. Also are the
resources not freeing up after logout?
If you need to have multiple simultaneous sessions, it is better to connect
to different Drillbits (may be in a round-robin fashion) than always
connecting to a single Drillbit.
Thanks
Venki
On Mon, Feb 1, 2016 at 11:51 PM, Josh Schlesser <j...@spoutable.com> wrote:
> First: Im a total newb at contributing to apache projects so please excuse
> any indiscretions, feel free to give comments on style or whatever, i take
> feedback well. Thick skin too.
>
>
> Ill give some background next and then a proposal.
>
> Background:
> I recently changed over to using authentication in the 1.5 snapshot
> because I need to have a session via the REST api so that I can set the
> session storage options in an initial query for a subsequent CTAS query.
> Previously all rest calls seemed to be completely independent.
>
> Since the change I have started seeing ‘too many files open’ errors in my
> drillbit.log and the drillbit java process becomes effectively hung waiting
> for open file descriptor slots. When running the top command the machine
> is running at max load due to the drillbit process and the drillbit becomes
> effectively unresponsive, even the simple pages in the web console don’t
> respond. Investigating further it seems that there might be a file kept
> open per session by the drillbit process for the life of the session. I
> used the lsof unix command on the drillbit process and found a lot of unix
> pipes. Looking at the code it looks like these pipes could be for the
> communication between the web process and the rpc server, with one being
> allocated per session. I haven’t validated this, its just a guess after
> scanning the code. I had 1.4 running without this requirement and without
> ever seeing the error. It seems without authentication the number of open
> files is a non-issue for me, possibly due to sessions.
>
> I'm wondering if my guess about what is causing the ‘too many open files’
> error is plausible? Does anybody with a deeper understanding of the
> architecture have any comments on this?
>
> Proposal:
> Assuming sessions are the issue, I am making some changes to my rest
> client so that sessions are more effectively used and I can up the ulimit
> for the drillbit process for the linux user in hopes of mitigating this. I
> am effectively creating a rest client based session pool that resets
> session variables to defaults when the session gets reused. However, it
> seems hacky.
>
> Below is an idea for getting per request based settings which seems less
> hacky in the long term.
>
> Can I add a new array member to the query.json REST method in a backwards
> compatible way to set session level parameters in a single request?
> Currently a rest request via the api has a body like so:
> { “queryType”: “SQL”, “query” : “<drill query>”}
>
> id like to do the following
>
> { “queryType”: “SQL”, “query” : “<drill query>”, “sessionSettings”:
> [“option_1_name”:”option_1_value”, “option_2_name”:”option_2_value”]}
>
> or even
>
> { “queryType”: “SQL”, “query” : “<drill query>”, “sessionSettings”: [“SET
> `option_name` = value”, “SET `option_name1` = value1”,“SET `option_name2` =
> value2”, “SET `option_name3` = value3”]}
>
> As far as I can tell drill is essentially stateless between queries right
> now except for session level system parameters and authentication. There
> aren’t any in memory temp tables or cursors or variables like PL/SQL or
> PSQL or other SQLs that would make it stateful.
>
> Given the stateless assumption, being able to set session level params on
> a per request basis would cover all of the cases that I might need. It
> looks relatively straight forward to add something to QueryWrapper to
> accept an optional query session settings section of the json packet and
> execute those ’SET' commands before the final query. This will work for
> me, as I can run without authentication in an ’secure' backend environment
> which will remove sessions and hence file descriptors, assuming my
> assumptions about file descriptors and sessions are correct.
>
>
> My java is rusty (circa 2003) but some casual googling implies that if
> this were added as a 3rd @FormParam to submitQuery in QueryResources it
> would be magically be null if it werent present and could easily be
> ignored. If its present then an alternative constructor of QueryWrapper
> could be called with the extra param and it would be easy to alter its run
> method to execute the SET commands. There would need to be some error
> handling of course if the SET commands were illegal or failed to run for
> some reason.
>
> If this seems reasonable, how do I go about contributing? I looked
> through the links in the docs to apache foundation incubator projects but
> the links to drill were broken :( http://drill.apache.org/team.html <
> http://drill.apache.org/team.html> I read this
> http://drill.apache.org/docs/apache-drill-contribution-guidelines/ <
> http://drill.apache.org/docs/apache-drill-contribution-guidelines/> and
> i have subscribed to the dev mailing list (obvious since you are getting
> this). It said to post here before creating a JIRA. Am I missing
> anything in my assumptions? Comments? Should I just submit a JIRA and a
> patch or submit a JIRA and a comment or wait for comments before coding
> stuff up as an example?
>
> Thanks for taking the time to read and respond.
>
> Josh
