> On May 20, 2015, 5:23 a.m., Amareshwari Sriramadasu wrote: > > I'm thinking, shall we make --verbose at lens-cli option, which initializes > > logger to console (if verbose is on) or to a file (otherwise). That would > > make logging available for all commands and we should not require passing > > verbose flag around and check for flag for logging allover. > > Amareshwari Sriramadasu wrote: > I meant while opening > $lens-cli // can open in verbose loggind mode i.e. logging on console > $lens-cli --quiet // can open in quiet logging mode i.e. no log on > console. > > Rajat Khandelwal wrote: > +1. > > Himanshu Gahlaut wrote: > would completely agree to the fact that passing verbose flag to functions > is a less optimal approach. Will try to post a suggestion soon. > > Himanshu Gahlaut wrote: > Another approach could be to change the logging level of cliLogger > programmatically to DEBUG from INFO. This can be done at only one place in > BaseLensCommand#beforeInvocation method which intercepts all commands. We can > log everything we want to display at DEBUG Level and keep ConsoleAppender's > Threshold at INFO level in file. The file appender can have its own Threshold > for logging as per its need. > > Raju Bairishetti wrote: > Yes, passing verbose to all the fuctions is not a good idea. I have done > these changes so that log information for other ocmmands won't be display on > the cli. > > We can enable the verbose by configuring the log4j properties. Right now, > it is not displaying any logs on the cli. In that case, we do not need even > clilogger for logging. We can directly use class level loggers for logging. > Otherwise we can simply configure one more cli logger and need to change the > existing logs for all commands to use cli logger. This amy affect server side > log information like we do not know the class level log information(i.e. > everything shows clilogger info). > > @Himanshu any suggestions? > > Himanshu Gahlaut wrote: > @Raju, Need your help in defining the goal of verbose option in command > clearly. Is the goal debugging or is it to provide intermediate information > which might be used to get more information similar to a joub url when we run > pig scripts ? > > Explaining below, how a clear goal will help. > > (1) If the goal is debugging, then as you already suggested above, class > level loggers and programmatically changing the log level to debug for > ConsoleAppender would be good enough. All loggers will start logging on > console which will help in debugging. > > (2) If the goal is intermediate informational output, then a separate > logger like a cliLogger would help. Any code which wants to print some > informational output can use this logger and we can programmatically change > its level to DEBUG in command interceptor when verbose option is ON. > > More inclined towards (1), however would prefer to hear more from you. > > Rajat Khandelwal wrote: > I think we should have a debug option for cli, with which debugging can > be done. Other than that, query progress should not be part of debug session. > query progress should be shown if `--progress` is true in the command. > > Raju Bairishetti wrote: > I think initial goal is to give some information about the query status > to the users. User can get the logs on cli by having appeneders in log4j > properties. Seems that does not require any code change. > > Rajat Khandelwal wrote: > The way I see it, is that there can be an interface `QueryProgress` in > lens-server-api and `LensDriver` can have a method returning `QueryProgress` > The only thing `QueryProgress` *needs* to implement is `float/double > getProgress()`. Each driver can have its own implementation, For example, > doing a query status on hive query returns three rows, one of which is > progress. The driver can parse and return that progress. Similar thing can be > there for JDBC. CLI just needs to ask for `QueryProgress` object from server > which in turn will ask it from the driver. Driver can give it's own > implementation, which will eventually be returned by server to cli. > > Raju Bairishetti wrote: > Yes, Right now Query status contains progress information as well. @Rajat > Can we use the same for knowing the query progress information(i.e. > QueryStatus.getProgessMessage()) or do we need to introduce a separate > interface? > > Rajat Khandelwal wrote: > Let's use what we already have :). But in cli's non verbose mode let's > show the progress on cli in case of sync > queries(http://stackoverflow.com/questions/852665/command-line-progress-bar-in-java). > For async queries, `query progress` command can be there for the same > purpose. `query status` prints the whole status. > > In verbose/debug mode, sync queries are showing the whole status, rest > all remains the same.
Summarizing concerns and discussion above. We would like to have --verbose option to know more about whats happening a command, and --debug option to actually look at debug logs. Here is how it can be achiveed : - All client and CLI classes continues to use class level loggers for info, debug logging for debugging purpose. --debug option can add console appender to see the logs directly on console. If no option is available (as it is now), use can add console appendar in log4j, all the logs will start showing up now. But can be enhanced to add --debug option which sets the console appender. - For showing more information about a command, we need a separate logger than class level logger. --verbose option can add console appender for this separate verbose logger. Commands will be using this verbose logger to give intermediate information about the command. we are trying to add verbose option and verbose logger in this jira. If adding debug option has to be seperate jira, I'm fine with it. Does it make sense? - Amareshwari ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34457/#review84467 ----------------------------------------------------------- On May 20, 2015, 4:48 a.m., Raju Bairishetti wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/34457/ > ----------------------------------------------------------- > > (Updated May 20, 2015, 4:48 a.m.) > > > Review request for lens and Amareshwari Sriramadasu. > > > Bugs: LENS-25 > https://issues.apache.org/jira/browse/LENS-25 > > > Repository: lens > > > Description > ------- > > Added a cli logger for showing the query execution status > Added a verbose option > > > Diffs > ----- > > lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java > 920ba9c > lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java > a48734a > lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab > lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82 > tools/conf/client/log4j.properties 5520304 > > Diff: https://reviews.apache.org/r/34457/diff/ > > > Testing > ------- > > Tested on cli > > **Enable logging by settign verbose value to true (by default to true)** > > **lens-shell>query execute cube select product_details.category, store_sales > from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') > --verbose true** > 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query > org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver > 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - > RUNNING:Query is running in HiveServer! - > Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}] > 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - > SUCCESSFUL:Query is successful! - Result Available > product_details.category _c1 > Results of query stored at : > file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows > process in (27) seconds. > > **Disable logging by setting verbose value to false (by default to true)** > lens-shell>query execute cube select sample_dim.name, sample_dim2.name from > sample_dim where sample_dim.name != 'first' --verbose false > sample_dim.name sample_dim2.name > Results of query stored at : > file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows > process in (8) seconds. > > > Thanks, > > Raju Bairishetti > >
