> On Oct. 23, 2015, 8:34 p.m., Sowmya Ramesh wrote: > > common/src/main/java/org/apache/falcon/entity/parser/DatasourceEntityParser.java, > > line 21 > > <https://reviews.apache.org/r/38465/diff/3-5/?file=1083437#file1083437line21> > > > > use org.apache.commons.lang3.StringUtils
Switched to using org.apache.commons.lang3.StringUtils for the following files: DatasourceEntityParser FeedImportCoordinatorBuilder ImportWorkflowBuilder HdfsClassLoader However, there are other classes (especially in regression package) still use org.apache.commons.lang.StringUtils. > On Oct. 23, 2015, 8:34 p.m., Sowmya Ramesh wrote: > > oozie/src/main/java/org/apache/falcon/oozie/FeedImportCoordinatorBuilder.java, > > line 21 > > <https://reviews.apache.org/r/38465/diff/3-5/?file=1083459#file1083459line21> > > > > use org.apache.commons.lang3.StringUtils Switched to using org.apache.commons.lang3.StringUtils for the following files: DatasourceEntityParser FeedImportCoordinatorBuilder ImportWorkflowBuilder HdfsClassLoader However, there are other classes (especially in regression package) still use org.apache.commons.lang.StringUtils. > On Oct. 23, 2015, 8:34 p.m., Sowmya Ramesh wrote: > > oozie/src/main/java/org/apache/falcon/oozie/ImportWorkflowBuilder.java, > > line 31 > > <https://reviews.apache.org/r/38465/diff/3-5/?file=1083460#file1083460line31> > > > > use org.apache.commons.lang3.StringUtils. Typo? Switched to using org.apache.commons.lang3.StringUtils for the following files: DatasourceEntityParser FeedImportCoordinatorBuilder ImportWorkflowBuilder HdfsClassLoader However, there are other classes (especially in regression package) still use org.apache.commons.lang.StringUtils. > On Oct. 23, 2015, 8:34 p.m., Sowmya Ramesh wrote: > > common/src/main/java/org/apache/falcon/metadata/InstanceRelationshipGraphBuilder.java, > > line 274 > > <https://reviews.apache.org/r/38465/diff/5/?file=1102901#file1102901line274> > > > > For replicaiton and eviction there can be cases where > > getOutputFeedInstancePaths or getInputFeedNames can return "NONE" or > > "IGNORE" i.e. no input or ourput feeds. Can there be such a scenario for > > Import? Import, in its current state, should produce a output path whenever it runs as it will produce a snapshot of data. Its safe to assume that there will be an output path. > On Oct. 23, 2015, 8:34 p.m., Sowmya Ramesh wrote: > > oozie/src/main/java/org/apache/falcon/oozie/ImportWorkflowBuilder.java, > > line 69 > > <https://reviews.apache.org/r/38465/diff/3-5/?file=1083460#file1083460line69> > > > > why was srcClusterName changed from cluster.getName() to "NA"? > > Do we req both srcClusterName and WorkflowExecutionArgs.CLUSTER_NAME? > > Balu Vellanki wrote: > In all workflows that do not do replication, the srcClusterName is set to > "NA". So it makes sense for the import process to set srcClusterName to "NA". Correct. The post processing xml has an argument srcClusterName that needs to be set to NA in case of flows other than replication. WorkflowExecutionArgs.CLUSTER_NAME is also used in the post processing and so it needs to be passed on. > On Oct. 23, 2015, 8:34 p.m., Sowmya Ramesh wrote: > > oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java, > > line 133 > > <https://reviews.apache.org/r/38465/diff/5/?file=1102932#file1102932line133> > > > > I see that WorkflowExecutionArgs.DATASOURCE_NAME.getName is set with > > "NONE" or "IGNORE". What is the logic here? When should be None and what > > cases it will be IGNORE? > > Balu Vellanki wrote: > @Sowmya - Look at Jira https://issues.apache.org/jira/browse/FALCON-1401. > NONE is the right value to use when there is no input/output exists. IGNORE > is supported for backwards compatibility, but should not be used going > forward. > > Sowmya Ramesh wrote: > @Balu - I know about that jira. I am asking in this jira why different > values are used and what is the logic behind that if I am missing something. > Also that is for feed names, I am not sure if we check > DATASOURCE_NAME.getName() for NONE or IGNORE anywhere. Backward compatibilty > makes sense in that jira as it spans multiple apache version. This is new > code and hence not required to use both. > > Balu Vellanki wrote: > Agree with you. IGNORE should not be used here. NONE is used. The post process xml has the following for datasource argument <arg>-datasource</arg> <arg>${datasource == 'NA' ? 'NONE' : datasource}</arg> - Venkatesan ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/38465/#review103847 ----------------------------------------------------------- On Oct. 28, 2015, 9:55 p.m., Venkatesan Ramachandran wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/38465/ > ----------------------------------------------------------- > > (Updated Oct. 28, 2015, 9:55 p.m.) > > > Review request for Falcon, Ajay Yadava, Balu Vellanki, Peeyush Bishnoi, > Sowmya Ramesh, and Venkat Ranganathan. > > > Repository: falcon-git > > > Description > ------- > > FALCON-1459 : Ability to import from database > > > Diffs > ----- > > client/src/main/java/org/apache/falcon/LifeCycle.java 58a2a6c > client/src/main/java/org/apache/falcon/Tag.java beeb812 > client/src/main/java/org/apache/falcon/entity/v0/EntityType.java 0657124 > client/src/main/java/org/apache/falcon/metadata/RelationshipType.java > f034772 > client/src/main/resources/datasource-0.1.xsd PRE-CREATION > client/src/main/resources/feed-0.1.xsd 77b8f4b > client/src/main/resources/jaxb-binding.xjb 6f1d6c7 > client/src/main/resources/mysql_database.xml PRE-CREATION > common/src/main/java/org/apache/falcon/entity/DatasourceHelper.java > PRE-CREATION > common/src/main/java/org/apache/falcon/entity/EntityUtil.java ceefb17 > common/src/main/java/org/apache/falcon/entity/FeedHelper.java 5c252a8 > > common/src/main/java/org/apache/falcon/entity/parser/DatasourceEntityParser.java > PRE-CREATION > > common/src/main/java/org/apache/falcon/entity/parser/EntityParserFactory.java > 5a33201 > common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java > c5cfdd2 > common/src/main/java/org/apache/falcon/entity/store/ConfigurationStore.java > 4dd1c68 > common/src/main/java/org/apache/falcon/entity/v0/EntityGraph.java bd4c6cf > > common/src/main/java/org/apache/falcon/entity/v0/EntityIntegrityChecker.java > bd32852 > > common/src/main/java/org/apache/falcon/metadata/EntityRelationshipGraphBuilder.java > 8c3876c > > common/src/main/java/org/apache/falcon/metadata/InstanceRelationshipGraphBuilder.java > f485764 > common/src/main/java/org/apache/falcon/metadata/MetadataMappingService.java > 56fbde0 > common/src/main/java/org/apache/falcon/metadata/RelationshipLabel.java > 5b312da > common/src/main/java/org/apache/falcon/util/HdfsClassLoader.java > PRE-CREATION > common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionArgs.java > ac7140c > > common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java > 9bfc51b > common/src/test/java/org/apache/falcon/entity/AbstractTestBase.java aab9cee > common/src/test/java/org/apache/falcon/entity/EntityTypeTest.java 640e87d > common/src/test/java/org/apache/falcon/entity/FeedHelperTest.java 4020d36 > > common/src/test/java/org/apache/falcon/entity/parser/DatasourceEntityParserTest.java > PRE-CREATION > > common/src/test/java/org/apache/falcon/entity/parser/FeedEntityParserTest.java > 905be68 > common/src/test/java/org/apache/falcon/entity/v0/EntityGraphTest.java > 3863b11 > common/src/test/resources/config/datasource/datasource-0.1.xml PRE-CREATION > common/src/test/resources/config/datasource/datasource-file-0.1.xml > PRE-CREATION > common/src/test/resources/config/datasource/datasource-invalid-0.1.xml > PRE-CREATION > common/src/test/resources/config/feed/feed-import-0.1.xml PRE-CREATION > common/src/test/resources/config/feed/feed-import-exclude-fields-0.1.xml > PRE-CREATION > common/src/test/resources/config/feed/feed-import-invalid-0.1.xml > PRE-CREATION > common/src/test/resources/config/feed/feed-import-noargs-0.1.xml > PRE-CREATION > docs/src/site/twiki/EntitySpecification.twiki e07fe12 > docs/src/site/twiki/FalconCLI.twiki d994866 > messaging/src/main/java/org/apache/falcon/messaging/JMSMessageProducer.java > 515562a > > oozie/src/main/java/org/apache/falcon/oozie/DatabaseImportWorkflowBuilder.java > PRE-CREATION > > oozie/src/main/java/org/apache/falcon/oozie/FeedImportCoordinatorBuilder.java > PRE-CREATION > oozie/src/main/java/org/apache/falcon/oozie/ImportWorkflowBuilder.java > PRE-CREATION > oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java > a04ae95 > > oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java > 3213a70 > > oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java > 0dc09ee > oozie/src/main/java/org/apache/falcon/oozie/feed/FeedBundleBuilder.java > 9e55edf > > oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java > f953557 > > oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java > 6e2a631 > > oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java > 3f49adb > oozie/src/main/resources/action/feed/import-sqoop-database-action.xml > PRE-CREATION > oozie/src/main/resources/action/post-process.xml df0d286 > pom.xml 87c55e3 > webapp/pom.xml 9e4dc8f > webapp/src/test/java/org/apache/falcon/lifecycle/FeedImportIT.java > PRE-CREATION > webapp/src/test/java/org/apache/falcon/resource/TestContext.java d067dee > webapp/src/test/java/org/apache/falcon/util/HsqldbTestUtils.java > PRE-CREATION > webapp/src/test/resources/datasource-template.xml PRE-CREATION > webapp/src/test/resources/feed-template3.xml PRE-CREATION > > Diff: https://reviews.apache.org/r/38465/diff/ > > > Testing > ------- > > * Unit tests > * Integration tests > * Manual tests > * Setup MySQL, create table and populate > * Create datasource and feed entity with import policy in Falcon > * Made sure the data lands up in the HDFS. > > > Thanks, > > Venkatesan Ramachandran > >
