Unable to write parent and child documents from same map-reduce job
Using elasticsearch-hadoop, I'd like to write a map reduce job where the EsOutputFormat writes both child and parent documents. However, it appears that it can only write parent or child documents within a single job. To write child documents, one needs to set the "es.mapping.parent" property in the JobConf, which I believe then messes up writing of the parent documents, which obviously don't have a parent. Is there a way to work around this? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f1404720-8704-4a4d-88ce-654920fdf7e0%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Advice for implementing a secure graph index with ElasticSearch
Hi Mohit, Can you please ask your Lumify questions over on the Lumify google group at https://groups.google.com/d/forum/lumify? I'd rather not pollute the ElasticSearch group with unrelated messages. Thanks, Jeff On Friday, May 16, 2014 1:26:28 AM UTC-4, mohit kaushik wrote: > > Hi! jeff > You said, you are using lumify. And lumify user secure-graph which > implicitly implements cell level security that you all need. You can easily > have access controls on your user and query returns the vertices in the way > you want. I recently started working with secure-graph and want to > implement the class > "/securegraph-core/src/main/java/org/securegraph/query/GraphQuery.java" > wchich is provided in the package and as it has been much days to your post > so hope you might have figured out it. If you have please notify me. > > > Thanks > Mohit kaushik > > On Wednesday, March 5, 2014 10:40:11 PM UTC+5:30, Jeff Kunkle wrote: >> >> I've been trying to figure out how I can index a graph data structure >> using ElasticSearch and could really use some advice from someone more >> knowledgeable than me. First, let me explain the challenge. The graph model >> has individual access controls at the vertex (node), edge (relationship), >> and property level. I'd like my users to be able to search the graph for >> vertices or edges containing matching properties, with two caveats: >> >>1. They should not get vertex or edge results they don't have >>permission to see. >>2. Properties a user does not have access to see should not be >>evaluated in the query. >> >> My first thought was to index properties as either nested or child >> documents of a vertex/edge and use a custom filter to remove properties a >> user didn't have access to. The first problem I run into is when I try a >> boolean query across properties. For example, assume I want to query a >> person vertex by first name and date of birth. Since these properties are >> indexed as separate documents there is never a match. >> >> What I essentially need is the ability to query across nested or child >> documents and return the parent only when there are matches across the >> child documents. For example, assume a parent vertex with one property >> document called "full_name" set to Barak Obama and another property >> document named "political_party" set to Democrat. Is there any way for me >> to query for the parent document of these two properties by asking for one >> property with full_name="Barak Obama" and another property with >> political_party="Democrat"? >> > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3527e622-cba3-4e5e-8ed3-49df8acf53b8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Advice for implementing a secure graph index with ElasticSearch
Hi Mike, Thanks for the reply. We actually started with Titan and its a very good project, but we couldn't easily add the needed security constraints on top of it. Hence why I'm exploring this topic. It would be rather straightforward to implement the index on ElasticSearch if all the data was open to everyone. I'd be able to consolidate all of a vertex's or edge's properties in a single document. Unfortunately, that's not the case. The project I'm working on is at http://lumify.io if that's helpful in any way. Thanks Again, Jeff On Wednesday, March 5, 2014 12:41:59 PM UTC-5, Michael Sick wrote: > > https://github.com/thinkaurelius/titan/wiki > > "Titan is a distributed graph > database<http://en.wikipedia.org/wiki/Graph_database> optimized > for storing and querying > graphs<http://en.wikipedia.org/wiki/Graph_(mathematics)> represented > over a cluster of machines. The cluster can elastically scale to support a > growing dataset and user base. Titan has a pluggable storage architecture > which allows it to build on proven database technology such as Apache > Cassandra <http://cassandra.apache.org/>, Apache > HBase<http://hbase.apache.org/>, > or Oracle BerkeleyDB<http://www.oracle.com/technetwork/database/berkeleydb/>. > Furthermore, the pluggable indexing architecture supports > ElasticSearch<http://elasticsearch.com/> > and Lucene <http://lucene.apache.org/>." > > I did some basic research for ES + graph and found the Titan project > interesting. Titan separates storage from indexing and only currently > supports ES for the latter. I'm sure that you could implement a storage > engine based on ES too (which makes more sense now that ES 1.x supports > backup/restore). Didn't look into security at all but this might be a good > starting point. Hope it's helpful. --Mike > > > On Wed, Mar 5, 2014 at 12:10 PM, Jeff Kunkle > > wrote: > >> I've been trying to figure out how I can index a graph data structure >> using ElasticSearch and could really use some advice from someone more >> knowledgeable than me. First, let me explain the challenge. The graph model >> has individual access controls at the vertex (node), edge (relationship), >> and property level. I'd like my users to be able to search the graph for >> vertices or edges containing matching properties, with two caveats: >> >>1. They should not get vertex or edge results they don't have >>permission to see. >>2. Properties a user does not have access to see should not be >>evaluated in the query. >> >> My first thought was to index properties as either nested or child >> documents of a vertex/edge and use a custom filter to remove properties a >> user didn't have access to. The first problem I run into is when I try a >> boolean query across properties. For example, assume I want to query a >> person vertex by first name and date of birth. Since these properties are >> indexed as separate documents there is never a match. >> >> What I essentially need is the ability to query across nested or child >> documents and return the parent only when there are matches across the >> child documents. For example, assume a parent vertex with one property >> document called "full_name" set to Barak Obama and another property >> document named "political_party" set to Democrat. Is there any way for me >> to query for the parent document of these two properties by asking for one >> property with full_name="Barak Obama" and another property with >> political_party="Democrat"? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearc...@googlegroups.com . >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/a8beee5b-82d0-45fa-8666-31e956c03439%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/a8beee5b-82d0-45fa-8666-31e956c03439%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1e8acfc2-81db-461f-817e-de1ca0b37c3e%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Advice for implementing a secure graph index with ElasticSearch
I've been trying to figure out how I can index a graph data structure using ElasticSearch and could really use some advice from someone more knowledgeable than me. First, let me explain the challenge. The graph model has individual access controls at the vertex (node), edge (relationship), and property level. I'd like my users to be able to search the graph for vertices or edges containing matching properties, with two caveats: 1. They should not get vertex or edge results they don't have permission to see. 2. Properties a user does not have access to see should not be evaluated in the query. My first thought was to index properties as either nested or child documents of a vertex/edge and use a custom filter to remove properties a user didn't have access to. The first problem I run into is when I try a boolean query across properties. For example, assume I want to query a person vertex by first name and date of birth. Since these properties are indexed as separate documents there is never a match. What I essentially need is the ability to query across nested or child documents and return the parent only when there are matches across the child documents. For example, assume a parent vertex with one property document called "full_name" set to Barak Obama and another property document named "political_party" set to Democrat. Is there any way for me to query for the parent document of these two properties by asking for one property with full_name="Barak Obama" and another property with political_party="Democrat"? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a8beee5b-82d0-45fa-8666-31e956c03439%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
