> On Oct. 20, 2015, 2:18 a.m., Raju Bairishetti wrote: > > lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java, > > line 128 > > <https://reviews.apache.org/r/39161/diff/8/?file=1100953#file1100953line128> > > > > Seems like there is some duplication of code. Can we change the run() > > method like this If you feel it avoids code duplication? > > > > public run() { > > try { > > //purge result set paths > > > > purgePaths(resultsetPath, true, resultsetRetention); > > > > //purge hdfsoutputpaths > > purgePaths(hdfsOutputPath,hdfsOutputRetention); > > } > > > > > > private void purgePaths(Path outputPath, boolean isFile, > > DateUtil.TimeDiff retention) { > > int counter = 0; > > FileSystem fs = outputPath.getFileSystem(conf); > > FileStatus[] fileList = fs.listStatus(resultsetPath); > > for (FileStatus f : fileList) { > > if (f.isFile() && canBePurged(f, retention)) { > > try { > > fs.delete(f.getPath()); > > counter++; > > } catch (IOException e) { > > getMetrics().incrCounter(this.getClass(), > > QUERY_RESULT_PURGER_ERROR_COUNTER); > > } > > } > > } > > log.info("Purged {} files in {}", counter, resultsetPath.toString()); > > }
Well, this code would not work. In one of the directory, in resultsetPath, we only want to purge files and in hdfsoutputPath we only want to purge directories. However, I will try to re-use the code using a purgeDirectory flag - Deepak ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/39161/#review103140 ----------------------------------------------------------- On Oct. 19, 2015, 12:22 p.m., Deepak Barr wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/39161/ > ----------------------------------------------------------- > > (Updated Oct. 19, 2015, 12:22 p.m.) > > > Review request for lens. > > > Bugs: LENS-292 > https://issues.apache.org/jira/browse/LENS-292 > > > Repository: lens > > > Description > ------- > > Added query result purge policy on both Driver persisted and Lens persisted > resultset. Default retention of 1 day. Purging can be turned off with a > setting. > > > Diffs > ----- > > lens-cube/src/main/java/org/apache/lens/cube/parse/DateUtil.java cbcc76c > > lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java > f202603 > > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java > deb8572 > > lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java > PRE-CREATION > lens-server/src/main/resources/lensserver-default.xml 1e6bd10 > > lens-server/src/test/java/org/apache/lens/server/query/TestQueryResultPurger.java > PRE-CREATION > src/site/apt/admin/config.apt 3a0678f > > Diff: https://reviews.apache.org/r/39161/diff/ > > > Testing > ------- > > Yes. > > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] Lens Checkstyle Rules .............................. SUCCESS [ 2.785 > s] > [INFO] Lens ............................................... SUCCESS [ 5.339 > s] > [INFO] Lens API ........................................... SUCCESS [ 28.950 > s] > [INFO] Lens API for server and extensions ................. SUCCESS [ 28.187 > s] > [INFO] Lens Cube .......................................... SUCCESS [07:05 > min] > [INFO] Lens DB storage .................................... SUCCESS [ 28.169 > s] > [INFO] Lens Query Library ................................. SUCCESS [ 22.031 > s] > [INFO] Lens Hive Driver ................................... SUCCESS [04:13 > min] > [INFO] Lens Driver for JDBC ............................... SUCCESS [ 49.452 > s] > [INFO] Lens Elastic Search Driver ......................... SUCCESS [ 22.865 > s] > [INFO] Lens Server ........................................ SUCCESS [10:58 > min] > [INFO] Lens client ........................................ SUCCESS [ 47.198 > s] > [INFO] Lens CLI ........................................... SUCCESS [03:44 > min] > [INFO] Lens Examples ...................................... SUCCESS [ 11.516 > s] > [INFO] Lens Distribution .................................. SUCCESS [ 12.779 > s] > [INFO] Lens ML Lib ........................................ SUCCESS [02:00 > min] > [INFO] Lens ML Ext Distribution ........................... SUCCESS [ 3.212 > s] > [INFO] Lens Regression .................................... SUCCESS [ 15.624 > s] > [INFO] Lens UI ............................................ SUCCESS [ 3.415 > s] > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 32:45 min > [INFO] Finished at: 2015-10-07T23:19:44+05:30 > [INFO] Final Memory: 218M/2795M > [INFO] > ------------------------------------------------------------------------ > > > Thanks, > > Deepak Barr > >
