[
https://issues.apache.org/jira/browse/LENS-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14626061#comment-14626061
]
Amareshwari Sriramadasu commented on LENS-602:
----------------------------------------------
Here is the summary of the solutions thought of and the pros & cons.
Get logs for a pattern : timestamp or threadId/Name or any pattern - Can be
achieved through grepping. Not in scope now. I think Lens should not pick this
as the output is not in control and there can be other tools built on top of
logs for debuggability.
Get logs of query submission
* On ESTIMATE, EXPLAIN : Involves logs to do with estimate (i.e. query
rewriting logs) and estimate/explain on driver. These are not associated with
query handle - they are associated with request handle.
* On EXECUTE and PREPARE : Involves logs to do with estimate, execution on
driver, result formatting. They are associated with query handle.
* Approaches put up
** Grep on lens.log.dir : Works fine, but is computationally expensive. So
avoiding this approach.
** Storing logs of query in separate files : One file per logSegregationId will
be too many files. So, discarding this option.
** Storing logs of query in separate files : Use separate MDC for query handle
than logSegregationId
*** Creating them through code : Would require a full log management - with
creation and purging policy in code. Avoiding this as SiftingAppender like
features can take care of them fully.
*** Use sifting appender through log4j : I tried to use MDCSiftingAppender
looking at
http://blog.nanthrax.net/2014/08/mdc-logging-with-apache-karaf-and-camel/ from
[~jbonofre]. But it was not straight forward to use with log4j.properties. Was
facing issues in Appender. Another option is to write an Appender in code.
*** Use sifting appender from logback : Haven't tried this yet. Would require
the code to move to slf4j fully and change the default log4j to logback. Need
to check if both log4j and logback can be used in deployment - because
statistics right now use some log4j api in code for uploading stats.
*** All the above leave with purging policy at deployment to take care of not
creating more files in a log directory so that it does not hit the limit on OS.
I'm exploring more on sifting appender. The solution put above has removed the
fetching logs for each request by requestid.
The api will still look for file in ${lens.log.dir} and serve it over rest.
> Easy access to per-query lens server logs
> -----------------------------------------
>
> Key: LENS-602
> URL: https://issues.apache.org/jira/browse/LENS-602
> Project: Apache Lens
> Issue Type: New Feature
> Reporter: Angad Singh
> Assignee: Amareshwari Sriramadasu
> Labels: Hackathon-July
>
> Right now one has to have access to the lens server machine and find a lens
> query's job logs manually in lensserver.log file. This is neither scalable
> nor user-friendly for a shared multi-tenanted lens server.
> Just throwing server-exceptions to the client or showing the job ID is also
> often not enough. Even when the query succeeds, one needs to see how
> candidate fact tables and their columns, etc. were pruned and how join chains
> were resolved, for example. That is only possible by seeing the lens server
> logs.
> Instead of that, this ticket is to propose that lens store logs for each lens
> query in a different log file on the server and that there be a REST end
> point to access a query's log (by query ID). That URL can be pasted on the
> client shell when the query is launched and the user can see a tailed log of
> all that lens is doing behind the scenes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
