> On May 20, 2015, 5:23 a.m., Amareshwari Sriramadasu wrote: > > I'm thinking, shall we make --verbose at lens-cli option, which initializes > > logger to console (if verbose is on) or to a file (otherwise). That would > > make logging available for all commands and we should not require passing > > verbose flag around and check for flag for logging allover. > > Amareshwari Sriramadasu wrote: > I meant while opening > $lens-cli // can open in verbose loggind mode i.e. logging on console > $lens-cli --quiet // can open in quiet logging mode i.e. no log on > console. > > Rajat Khandelwal wrote: > +1. > > Himanshu Gahlaut wrote: > would completely agree to the fact that passing verbose flag to functions > is a less optimal approach. Will try to post a suggestion soon. > > Himanshu Gahlaut wrote: > Another approach could be to change the logging level of cliLogger > programmatically to DEBUG from INFO. This can be done at only one place in > BaseLensCommand#beforeInvocation method which intercepts all commands. We can > log everything we want to display at DEBUG Level and keep ConsoleAppender's > Threshold at INFO level in file. The file appender can have its own Threshold > for logging as per its need. > > Raju Bairishetti wrote: > Yes, passing verbose to all the fuctions is not a good idea. I have done > these changes so that log information for other ocmmands won't be display on > the cli. > > We can enable the verbose by configuring the log4j properties. Right now, > it is not displaying any logs on the cli. In that case, we do not need even > clilogger for logging. We can directly use class level loggers for logging. > Otherwise we can simply configure one more cli logger and need to change the > existing logs for all commands to use cli logger. This amy affect server side > log information like we do not know the class level log information(i.e. > everything shows clilogger info). > > @Himanshu any suggestions? > > Himanshu Gahlaut wrote: > @Raju, Need your help in defining the goal of verbose option in command > clearly. Is the goal debugging or is it to provide intermediate information > which might be used to get more information similar to a joub url when we run > pig scripts ? > > Explaining below, how a clear goal will help. > > (1) If the goal is debugging, then as you already suggested above, class > level loggers and programmatically changing the log level to debug for > ConsoleAppender would be good enough. All loggers will start logging on > console which will help in debugging. > > (2) If the goal is intermediate informational output, then a separate > logger like a cliLogger would help. Any code which wants to print some > informational output can use this logger and we can programmatically change > its level to DEBUG in command interceptor when verbose option is ON. > > More inclined towards (1), however would prefer to hear more from you. > > Rajat Khandelwal wrote: > I think we should have a debug option for cli, with which debugging can > be done. Other than that, query progress should not be part of debug session. > query progress should be shown if `--progress` is true in the command. > > Raju Bairishetti wrote: > I think initial goal is to give some information about the query status > to the users. User can get the logs on cli by having appeneders in log4j > properties. Seems that does not require any code change. > > Rajat Khandelwal wrote: > The way I see it, is that there can be an interface `QueryProgress` in > lens-server-api and `LensDriver` can have a method returning `QueryProgress` > The only thing `QueryProgress` *needs* to implement is `float/double > getProgress()`. Each driver can have its own implementation, For example, > doing a query status on hive query returns three rows, one of which is > progress. The driver can parse and return that progress. Similar thing can be > there for JDBC. CLI just needs to ask for `QueryProgress` object from server > which in turn will ask it from the driver. Driver can give it's own > implementation, which will eventually be returned by server to cli. > > Raju Bairishetti wrote: > Yes, Right now Query status contains progress information as well. @Rajat > Can we use the same for knowing the query progress information(i.e. > QueryStatus.getProgessMessage()) or do we need to introduce a separate > interface? > > Rajat Khandelwal wrote: > Let's use what we already have :). But in cli's non verbose mode let's > show the progress on cli in case of sync > queries(http://stackoverflow.com/questions/852665/command-line-progress-bar-in-java). > For async queries, `query progress` command can be there for the same > purpose. `query status` prints the whole status. > > In verbose/debug mode, sync queries are showing the whole status, rest > all remains the same. > > Amareshwari Sriramadasu wrote: > Summarizing concerns and discussion above. > > We would like to have --verbose option to know more about whats happening > a command, and --debug option to actually look at debug logs. Here is how it > can be achiveed : > > - All client and CLI classes continues to use class level loggers for > info, debug logging for debugging purpose. --debug option can add console > appender to see the logs directly on console. If no option is available (as > it is now), use can add console appendar in log4j, all the logs will start > showing up now. But can be enhanced to add --debug option which sets the > console appender. > - For showing more information about a command, we need a separate logger > than class level logger. --verbose option can add console appender for this > separate verbose logger. Commands will be using this verbose logger to give > intermediate information about the command. > > we are trying to add verbose option and verbose logger in this jira. If > adding debug option has to be seperate jira, I'm fine with it. > > Does it make sense?
Thanks Amareshwari and Rajat for the responses. Will do the needful changes accordingly. - Raju ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34457/#review84467 ----------------------------------------------------------- On May 20, 2015, 4:48 a.m., Raju Bairishetti wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/34457/ > ----------------------------------------------------------- > > (Updated May 20, 2015, 4:48 a.m.) > > > Review request for lens and Amareshwari Sriramadasu. > > > Bugs: LENS-25 > https://issues.apache.org/jira/browse/LENS-25 > > > Repository: lens > > > Description > ------- > > Added a cli logger for showing the query execution status > Added a verbose option > > > Diffs > ----- > > lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java > 920ba9c > lens-cli/src/test/java/org/apache/lens/cli/TestLensQueryCommands.java > a48734a > lens-client/src/main/java/org/apache/lens/client/LensClient.java 016e4ab > lens-client/src/main/java/org/apache/lens/client/LensStatement.java 5ce3a82 > tools/conf/client/log4j.properties 5520304 > > Diff: https://reviews.apache.org/r/34457/diff/ > > > Testing > ------- > > Tested on cli > > **Enable logging by settign verbose value to true (by default to true)** > > **lens-shell>query execute cube select product_details.category, store_sales > from sales where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') > --verbose true** > 2015-05-20 08:42:05,563 [ LensStatement.java:Spring Shell:227] - Query > org.apache.lens.api.query.LensQuery@58dd5a16 submitted to HiveDriver > 2015-05-20 08:42:15,510 [ LensStatement.java:Spring Shell:232] - > RUNNING:Query is running in HiveServer! - > Progress:0.3333333432674408:[{"taskId":"Stage-5","type":"MAPREDLOCAL","externalHandle":"Stage-5","taskState":"FINISHED_STATE"},{"taskId":"Stage-2","type":"MAPRED","externalHandle":null,"taskState":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}] > 2015-05-20 08:42:25,561 [ LensStatement.java:Spring Shell:232] - > SUCCESSFUL:Query is successful! - Result Available > product_details.category _c1 > Results of query stored at : > file:/tmp/lensreports/hdfsout/256d4c84-496c-49e8-993c-3d8b8f0451a5 -1 rows > process in (27) seconds. > > **Disable logging by setting verbose value to false (by default to true)** > lens-shell>query execute cube select sample_dim.name, sample_dim2.name from > sample_dim where sample_dim.name != 'first' --verbose false > sample_dim.name sample_dim2.name > Results of query stored at : > file:/tmp/lensreports/hdfsout/a6abe7e3-e398-4c47-916c-80ab80c49c0b -1 rows > process in (8) seconds. > > > Thanks, > > Raju Bairishetti > >
