> On Sept. 25, 2015, 1:07 a.m., Sowmya Ramesh wrote: > > common/src/main/java/org/apache/falcon/entity/DatasourceHelper.java, line 55 > > <https://reviews.apache.org/r/38465/diff/3/?file=1083434#file1083434line55> > > > > feedCluster.getImport() can return null as import minoccurs is 0?
IMPORT is optional in XSD, but if IMPORT is present, everthing underneath is mandatory. Also, The ImportDatasourceType() is called during IMPORT lifecycle operation, which gets invoked only if IMPORT tag is present. > On Sept. 25, 2015, 1:07 a.m., Sowmya Ramesh wrote: > > common/src/main/java/org/apache/falcon/util/HdfsClassLoader.java, line 134 > > <https://reviews.apache.org/r/38465/diff/3/?file=1083444#file1083444line134> > > > > can fileURL be null? No. Checked in the XSD during submit. > On Sept. 25, 2015, 1:07 a.m., Sowmya Ramesh wrote: > > common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java, > > line 76 > > <https://reviews.apache.org/r/38465/diff/3/?file=1083445#file1083445line76> > > > > Instead of adding new type cant we reuse GENEARTE as import also > > generates feeds unless we save the operation type somewhere and can be > > useful to deintify the operation. IMPORT is different from GENERATE. IMPORT brings data from outside of Hadoop ecosystem while GENERATE creates a derived data using an IMPORTED dataset. > On Sept. 25, 2015, 1:07 a.m., Sowmya Ramesh wrote: > > oozie/src/main/java/org/apache/falcon/oozie/ImportWorkflowBuilder.java, > > line 63 > > <https://reviews.apache.org/r/38465/diff/3/?file=1083460#file1083460line63> > > > > As import geanerate feed should this be set to Ignore? The Feed output is set to feedInstancePaths now. > On Sept. 25, 2015, 1:07 a.m., Sowmya Ramesh wrote: > > oozie/src/main/java/org/apache/falcon/oozie/DatabaseImportWorkflowBuilder.java, > > line 147 > > <https://reviews.apache.org/r/38465/diff/3/?file=1083458#file1083458line147> > > > > can exrtaArgs be null? FeedHelper.getImportArguments() checks for null and returns an map appropriately. > On Sept. 25, 2015, 1:07 a.m., Sowmya Ramesh wrote: > > oozie/src/main/java/org/apache/falcon/oozie/DatabaseImportWorkflowBuilder.java, > > line 57 > > <https://reviews.apache.org/r/38465/diff/3/?file=1083458#file1083458line57> > > > > Have we tested this on secure cluster setup? For cases when workflow > > runs on diff m/c and hdfs is on diff node, do we have to pass any creds? Tested on a secure cluster. - Venkatesan ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/38465/#review100491 ----------------------------------------------------------- On Sept. 23, 2015, 11:58 p.m., Venkatesan Ramachandran wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/38465/ > ----------------------------------------------------------- > > (Updated Sept. 23, 2015, 11:58 p.m.) > > > Review request for Falcon, Ajay Yadava, Balu Vellanki, Peeyush Bishnoi, > Sowmya Ramesh, and Venkat Ranganathan. > > > Repository: falcon-git > > > Description > ------- > > FALCON-1459 : Ability to import from database > > > Diffs > ----- > > client/src/main/java/org/apache/falcon/LifeCycle.java 58a2a6c > client/src/main/java/org/apache/falcon/Tag.java beeb812 > client/src/main/java/org/apache/falcon/entity/v0/EntityType.java 0657124 > client/src/main/java/org/apache/falcon/metadata/RelationshipType.java > f034772 > client/src/main/resources/datasource-0.1.xsd PRE-CREATION > client/src/main/resources/feed-0.1.xsd 2af28d2 > client/src/main/resources/jaxb-binding.xjb 6f1d6c7 > client/src/main/resources/mysql_database.xml PRE-CREATION > common/src/main/java/org/apache/falcon/entity/DatasourceHelper.java > PRE-CREATION > common/src/main/java/org/apache/falcon/entity/EntityUtil.java ad41674 > common/src/main/java/org/apache/falcon/entity/FeedHelper.java 572923b > > common/src/main/java/org/apache/falcon/entity/parser/DatasourceEntityParser.java > PRE-CREATION > > common/src/main/java/org/apache/falcon/entity/parser/EntityParserFactory.java > 5a33201 > common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java > 4f5599e > common/src/main/java/org/apache/falcon/entity/store/ConfigurationStore.java > e27187b > common/src/main/java/org/apache/falcon/entity/v0/EntityGraph.java bd4c6cf > > common/src/main/java/org/apache/falcon/entity/v0/EntityIntegrityChecker.java > bd32852 > > common/src/main/java/org/apache/falcon/metadata/EntityRelationshipGraphBuilder.java > 8c3876c > common/src/main/java/org/apache/falcon/util/HdfsClassLoader.java > PRE-CREATION > > common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java > 4454239 > common/src/test/java/org/apache/falcon/entity/AbstractTestBase.java e9946c4 > common/src/test/java/org/apache/falcon/entity/EntityTypeTest.java 640e87d > common/src/test/java/org/apache/falcon/entity/FeedHelperTest.java c70cfcc > > common/src/test/java/org/apache/falcon/entity/parser/DatasourceEntityParserTest.java > PRE-CREATION > > common/src/test/java/org/apache/falcon/entity/parser/FeedEntityParserTest.java > b6fdb13 > common/src/test/java/org/apache/falcon/entity/v0/EntityGraphTest.java > 3863b11 > common/src/test/resources/config/datasource/datasource-0.1.xml PRE-CREATION > common/src/test/resources/config/datasource/datasource-invalid-0.1.xml > PRE-CREATION > common/src/test/resources/config/feed/feed-import-0.1.xml PRE-CREATION > common/src/test/resources/config/feed/feed-import-invalid-0.1.xml > PRE-CREATION > docs/src/site/twiki/EntitySpecification.twiki d4f4140 > > falcon-regression/merlin-core/src/main/java/org/apache/falcon/regression/core/util/HiveAssert.java > 2a934b5 > > oozie/src/main/java/org/apache/falcon/oozie/DatabaseImportWorkflowBuilder.java > PRE-CREATION > > oozie/src/main/java/org/apache/falcon/oozie/FeedImportCoordinatorBuilder.java > PRE-CREATION > oozie/src/main/java/org/apache/falcon/oozie/ImportWorkflowBuilder.java > PRE-CREATION > oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java > a04ae95 > > oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java > 3213a70 > oozie/src/main/java/org/apache/falcon/oozie/feed/FeedBundleBuilder.java > b819dee > oozie/src/main/resources/action/feed/import-sqoop-database-action.xml > PRE-CREATION > pom.xml 646de69 > webapp/pom.xml 828f7f5 > webapp/src/test/java/org/apache/falcon/lifecycle/FeedImportIT.java > PRE-CREATION > webapp/src/test/java/org/apache/falcon/resource/TestContext.java d067dee > webapp/src/test/java/org/apache/falcon/util/HsqldbTestUtils.java > PRE-CREATION > webapp/src/test/resources/datasource-template.xml PRE-CREATION > webapp/src/test/resources/feed-template3.xml PRE-CREATION > > Diff: https://reviews.apache.org/r/38465/diff/ > > > Testing > ------- > > * Unit tests > * Integration tests > * Manual tests > * Setup MySQL, create table and populate > * Create datasource and feed entity with import policy in Falcon > * Made sure the data lands up in the HDFS. > > > Thanks, > > Venkatesan Ramachandran > >
