> On Oct. 13, 2015, 11:23 p.m., Sowmya Ramesh wrote: > >
Thanks for the review. > On Oct. 13, 2015, 11:23 p.m., Sowmya Ramesh wrote: > > addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRTool.java, line 113 > > <https://reviews.apache.org/r/39261/diff/1/?file=1097136#file1097136line113> > > > > I don't fully understand the logic here. obtainJobCounters for HiveDR > > is called twice - once from HiveDRTool.run > > hiveReplicationCounters.obtainJobCounters(job.getConfiguration(), job, > > false) > > and > > EventUtils.invokeCopy hiveReplicationCounters.obtainJobCounters(conf, > > job, true); > > > > Why is called twice? First job includes both export handling and distcp > > - why are we recording start and end time of that job and where is it used? > > > > Also are we storing counters for the job which handled export and > > Distcp instead of only DistCP? Where are the counters got from invokeCopy > > obtainJobCounters call stored? As I mentioned in the jira issue, that approach for this to create custom mapreduce counters for the Hive replication job. If distcp jobs are invoked from Hive replication job map tasks (export), populate custom MR counters in each map task with required counters value obtained from the completed distcp jobs. Once Hive replication job completed successfully, obtain and store aggregate custom counters from Hive replication job in a counter file. Due to this reason it is called twice. start and end time of job is getting used to know how much time respective job has taken to finish. So right now we are capturing 3 parameters TIMETAKEN, COPIEDBYTES, COPY. No, we are not storing counters for the job which handled export and distcp in a file. Due to this I have used custom mapreduce counters for job and populate each map task with required counters value. > On Oct. 13, 2015, 11:23 p.m., Sowmya Ramesh wrote: > > addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRTool.java, line 114 > > <https://reviews.apache.org/r/39261/diff/1/?file=1097136#file1097136line114> > > > > Was FS/HiveDR counters tested in multi cluster setup and did it work? > > For HiveDR did you verify if counters was populated in graphDB? > > > > I can see that JobCounters.storeJobCounters gets the FS conf from > > job.getConfiguration() - does this refer to source cluster or destination > > cluster FS conf? Where does the counters.txt get created? > > > > In post processing in multi cluster setup if Falcon is running on diff > > cluster when counters.txt is accessed can it throw File not found exception > > if file is created on diff cluster? Same for FS replicaiton counters. Yes I have tested in multicluster setup and it works. Even I have checked the graphDB and it got populated with required counters. counters.txt file (new Path(job.getConfiguration().get("counterLogDir"), "counter.txt");) will get created in Falcon logDir path <arg>-counterLogDir</arg> <arg>${logDir}/job-${nominalTime}/${srcClusterName == 'NA' ? '' : srcClusterName}/</arg> . logDir is fully qualified hostname. > On Oct. 13, 2015, 11:23 p.m., Sowmya Ramesh wrote: > > addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyMapper.java, > > line 76 > > <https://reviews.apache.org/r/39261/diff/1/?file=1097137#file1097137line76> > > > > Lets say the DistCP job never ran e.g. export event list empty; > > context.getCounter will return null and increment will throw NPE, isn't? Fixed. > On Oct. 13, 2015, 11:23 p.m., Sowmya Ramesh wrote: > > addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-workflow.xml, > > line 248 > > <https://reviews.apache.org/r/39261/diff/1/?file=1097140#file1097140line248> > > > > where is srcClusterName set with actual soruce cluster name? Did you > > verify file generated has source cluster name? > > > > I can see that for FS replication its set in > > getReplicationWorkflowAction. > > > > For HiveDr it runs a sa process. Its set to "NA" in > > ProcessExecutionWorkflowBuilder.getWorkflowProperties but couldn't find > > anywhere where its set to source cluster name. > > Did you verify file name genearated has "NA" or src cluster name for > > HiveDR? Value for srcClusterName will be provided through invoked workflow. If it is "NA", then file will be created inside logDir without srcClusterName. I have already tested and it will not bring any impact. Also it will get clean up when logcleanupservice invoke. - Peeyush ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/39261/#review102553 ----------------------------------------------------------- On Oct. 13, 2015, 8:58 a.m., Peeyush Bishnoi wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/39261/ > ----------------------------------------------------------- > > (Updated Oct. 13, 2015, 8:58 a.m.) > > > Review request for Falcon. > > > Bugs: FALCON-1480 > https://issues.apache.org/jira/browse/FALCON-1480 > > > Repository: falcon-git > > > Description > ------- > > FALCON-1480: Gather data transfer details of Hive DR > > > Diffs > ----- > > addons/hivedr/pom.xml a1be288 > addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRArgs.java 574524d > addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRTool.java bebdb0b > > addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyMapper.java > 5eb8acb > addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventUtils.java > f8397ff > addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRUtils.java > d9d6ab0 > > addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-workflow.xml > 74902b4 > > addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-workflow.xml > 72d40a3 > metrics/src/main/java/org/apache/falcon/job/HiveReplicationCounters.java > PRE-CREATION > metrics/src/main/java/org/apache/falcon/job/JobCountersHandler.java e8b68ff > metrics/src/main/java/org/apache/falcon/job/JobType.java 456e57f > > Diff: https://reviews.apache.org/r/39261/diff/ > > > Testing > ------- > > Yes. Manual testing done. > > > Thanks, > > Peeyush Bishnoi > >
