> On May 16, 2016, 6:59 a.m., Hemanth Yamijala wrote: > > catalog/src/main/java/org/apache/atlas/catalog/query/AtlasEntityQuery.java, > > line 37 > > <https://reviews.apache.org/r/47382/diff/1/?file=1383456#file1383456line37> > > > > If I understood this correctly, the basepipe for the Gremlin search for > > entities begins by trying to identify all entity vertices (by virtue of > > using the full text search property that is common to all vertices). This > > would still be a huge result set to begin with, no? > > John Speidel wrote: > This is currently the only way to identify an entity vertex with the > current database model. It is a requirement to be able to search all entity > types and to be able to apply tags to entities across types. So, we need to > be able to get all of the entity vertices, which is the set that would be > returned from the base query. The biggest issue with using this field to > determine the entity set is that it isn't indexed. Also, the query starts > with V(), which is all vertices in the graph but this doesn't result in all > vertices being returned but instead adds them to the pipeline interator. > > That being said, there are lots of improvements that can be made to > improve query performance. This would include, changes to the database model > so that we knew the propery names ahead of time so that we could use an index > instead of a filter. Also, one optimization that I was going to add which > didn't make it in was to use the type if it was provided so that indexes > could be used as mentioned above. For now, I don't think that we can improve > on this.
I thought we do index the ENTITY_TEXT_PROPERTY_KEY. This is done in GraphBackedSearchIndexer.createFullTextIndex. Please see comment below on search and we can probably get this in and open follow-up JIRAs to fix. - Hemanth ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47382/#review133332 ----------------------------------------------------------- On May 14, 2016, 12:58 a.m., John Speidel wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47382/ > ----------------------------------------------------------- > > (Updated May 14, 2016, 12:58 a.m.) > > > Review request for atlas and Hemanth Yamijala. > > > Bugs: ATLAS-491 > https://issues.apache.org/jira/browse/ATLAS-491 > > > Repository: atlas > > > Description > ------- > > Initial implementation of Catalog API. > > Known Issues/Limitations: > - No update/delete support for taxonomies/terms > -- this will be added next week > - Some gaps in test coverage > - Need more in-depth test coverage that exercise the generated gremlin > pipeline > - some refactoring should still be done espcecially in the 'query' package > - need to add more logging > - need to finish some javadoc > > > Diffs > ----- > > catalog/pom.xml PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/AtlasTypeSystem.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/BaseRequest.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/BaseResourceProvider.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/CollectionRequest.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/DefaultDateFormatter.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/DefaultPropertyMapper.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/DefaultTypeSystem.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/EntityResourceProvider.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/EntityTagResourceProvider.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/InstanceRequest.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/JsonSerializer.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/PropertyMapper.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/PropertyValueFormatter.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/Request.java PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/ResourceComparator.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/ResourceProvider.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/Result.java PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/TaxonomyResourceProvider.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/TermPath.java PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/TermResourceProvider.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/TermVertexWrapper.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/VertexWrapper.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/definition/BaseResourceDefinition.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/definition/EntityResourceDefinition.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/definition/EntityTagResourceDefinition.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/definition/ResourceDefinition.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/definition/TaxonomyResourceDefinition.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/definition/TermResourceDefinition.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/exception/CatalogException.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/exception/CatalogRuntimeException.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/exception/InvalidPayloadException.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/exception/InvalidQueryException.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/exception/ResourceAlreadyExistsException.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/exception/ResourceNotFoundException.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/projection/GenericRelation.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/projection/Projection.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/projection/ProjectionResult.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/projection/Relation.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/projection/RelationProjection.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/projection/RelationSet.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/projection/TagRelation.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/projection/TraitRelation.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/query/AlwaysQueryExpression.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/query/AtlasEntityQuery.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/query/AtlasEntityTagQuery.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/query/AtlasQuery.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/query/AtlasTaxonomyQuery.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/query/AtlasTermQuery.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/query/BaseQueryExpression.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/query/BooleanQueryExpression.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/query/PrefixQueryExpression.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/query/ProjectionQueryExpression.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/query/QueryExpression.java > PRE-CREATION > catalog/src/main/java/org/apache/atlas/catalog/query/QueryFactory.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/query/RegexQueryExpression.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/query/TermQueryExpression.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/query/TermRangeQueryExpression.java > PRE-CREATION > > catalog/src/main/java/org/apache/atlas/catalog/query/WildcardQueryExpression.java > PRE-CREATION > catalog/src/test/java/org/apache/atlas/catalog/CollectionRequestTest.java > PRE-CREATION > > catalog/src/test/java/org/apache/atlas/catalog/DefaultDateFormatterTest.java > PRE-CREATION > > catalog/src/test/java/org/apache/atlas/catalog/DefaultPropertyMapperTest.java > PRE-CREATION > > catalog/src/test/java/org/apache/atlas/catalog/EntityResourceProviderTest.java > PRE-CREATION > > catalog/src/test/java/org/apache/atlas/catalog/EntityTagResourceProviderTest.java > PRE-CREATION > catalog/src/test/java/org/apache/atlas/catalog/InstanceRequestTest.java > PRE-CREATION > catalog/src/test/java/org/apache/atlas/catalog/JsonSerializerTest.java > PRE-CREATION > catalog/src/test/java/org/apache/atlas/catalog/ResourceComparatorTest.java > PRE-CREATION > > catalog/src/test/java/org/apache/atlas/catalog/TaxonomyResourceProviderTest.java > PRE-CREATION > > catalog/src/test/java/org/apache/atlas/catalog/TermResourceProviderTest.java > PRE-CREATION > catalog/src/test/java/org/apache/atlas/catalog/VertexWrapperTest.java > PRE-CREATION > > catalog/src/test/java/org/apache/atlas/catalog/definition/EntityResourceDefinitionTest.java > PRE-CREATION > > catalog/src/test/java/org/apache/atlas/catalog/definition/EntityTagResourceDefinitionTest.java > PRE-CREATION > > catalog/src/test/java/org/apache/atlas/catalog/definition/TaxonomyResourceDefinitionTest.java > PRE-CREATION > > catalog/src/test/java/org/apache/atlas/catalog/definition/TermResourceDefinitionTest.java > PRE-CREATION > > catalog/src/test/java/org/apache/atlas/catalog/query/AlwaysQueryExpressionTest.java > PRE-CREATION > catalog/src/test/java/org/apache/atlas/catalog/query/QueryFactoryTest.java > PRE-CREATION > pom.xml 5e2871e > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > cccafc2 > > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java > 5195cbe > server-api/src/main/java/org/apache/atlas/services/MetadataService.java > 13d20d8 > > typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java > 6fb2087 > > typesystem/src/main/java/org/apache/atlas/typesystem/types/TypedStructHandler.java > b97669a > webapp/pom.xml de48c15 > webapp/src/main/java/org/apache/atlas/web/resources/BaseService.java > PRE-CREATION > > webapp/src/main/java/org/apache/atlas/web/resources/CatalogExceptionMapper.java > PRE-CREATION > > webapp/src/main/java/org/apache/atlas/web/resources/CatalogRuntimeExceptionMapper.java > PRE-CREATION > webapp/src/main/java/org/apache/atlas/web/resources/EntityService.java > PRE-CREATION > webapp/src/main/java/org/apache/atlas/web/resources/TaxonomyService.java > PRE-CREATION > webapp/src/main/webapp/WEB-INF/web.xml f0b606e > > Diff: https://reviews.apache.org/r/47382/diff/ > > > Testing > ------- > > Ran all existing unit tests and added new tests. > > > Thanks, > > John Speidel > >
