Hi, I think this is an interesting project but is not core to "Pig" itself -- it may be more interesting / viable as a standalone project on github that uses Pig to implement graph algorithms. At this point in its development, I feel that Pig needs to concentrate on doing the things it already does, and do them better (operator efficiency, storage efficiency, better MR plan generation, etc) rather than expand to specific verticals; we should allow our users to create their own solution suites that use Pig for specific purposes. A successful example of such a standalone project is PacketPig (https://github.com/packetloop/packetpig) , a PCAP network capture analysis tool.
D On Tue, Apr 2, 2013 at 9:48 AM, burakkk <burak.isi...@gmail.com> wrote: > I know that but giraph tries to use bsp. What I'm saying is nothing shared > model except reducers. Besides I don't want to divide iteration. One phase > is still responsible for whole iteration. Every different origin vertex > will be processed in parallel. > > Thanks > Best regards... > > > On Tue, Apr 2, 2013 at 7:20 PM, Gianmarco De Francisci Morales < > g...@gdfm.me > > wrote: > > > FYI, Giraph has a Random Walk implementation. > > > > Pig does not support iteration natively, so any iterative algorithm is > not > > a very good fit for it. Just my 2c. > > > > Cheers, > > > > -- > > Gianmarco > > > > > > On Tue, Apr 2, 2013 at 10:04 AM, burakkk <burak.isi...@gmail.com> wrote: > > > > > So what do you suggest? Is it clear? > > > > > > > > > On Mon, Apr 1, 2013 at 9:35 PM, burakkk <burak.isi...@gmail.com> > wrote: > > > > > > > I'm using only WTF graph representation to fit the memory. By the > way I > > > > haven't seen any explanation from the pig 0.11 release page about WTF > > or > > > > graph models. > > > > I don't wanna use Cassovary. I believe it can be done with pig. I > > > > implement a graph representation using WTF paper to pig and then I'll > > use > > > > it to implement random walk algorithm. To do that maybe I need to > > improve > > > > some features such as joins(fuzzy join) etc or implement a new > > operator. > > > I > > > > can implement it using either existing operators or new operators. > > That's > > > > up to us and it doesn't really matter. If there is already a > > > implementation > > > > to random walker algorithm, please feel free to tell. Because I > haven't > > > > found it. > > > > Are you proposing to create an open-source implementation of those > > > > algorithms? > > > > Yes, I'm proposing to implement a random walk algorithm, new data > model > > > > which is representing graph. After that, people can use it coding the > > > pig. > > > > > > > > Do you suggest they should be Pig scripts added to the Pig project, > or > > do > > > > you want to create some new operators? > > > > Maybe, it can be UDF or new operator. > > > > > > > > I made a quick example. It may not be completely accurate, I've just > > > tried > > > > to explain it. > > > > Think about you have a graph file just like that > > > > user_id follower > > > > 1 2 > > > > 1 3 > > > > 1 10 > > > > 2 3 > > > > 3 4 > > > > 3 5 > > > > ... > > > > > > > > Vertex List is an array including sorted vertex ids > > > > node List is a matrix including vertex id and its starting position > > > > > > > > > > > > graph = load 'graph' using PigStorage() (vertex:int, follower:int) - > > > > --load the graph file > > > > vertex = COGROUP graph BY (vertex); > > > > list = FOREACH vertex GENERATE org.apache.pig.generateVertex(vertex) > as > > > > vertexList; --load the whole vertexes from HDFS into the memory > > > > list = FOREACH graph GENERATE org.apache.pig.generateNode(list) as > > > > nodeList; --load the whole vertexes from HDFS into the memory > > > > randomWalk = FOREACH vertex GENERATE > > > > flatten(org.apache.pig.RandomWalk(list, endVertex)) as score; -- > > > generate a > > > > score using the node list you can traverse the graph to the your > > > finishing > > > > position > > > > store... > > > > > > > > > > > > Thanks > > > > Best Regards... > > > > > > > > > > > > On Mon, Apr 1, 2013 at 7:20 PM, Dmitriy Ryaboy <dvrya...@gmail.com> > > > wrote: > > > > > > > >> I'm somewhat familiar with WTF code (my day job is managing the > > > analytics > > > >> infrastructure team at Twitter). WTF is implemented using Pig 0.11 > (in > > > >> fact > > > >> some of the Pig 11 features/improvements are directly due to this > > > >> project...), and mostly has to do with clever algorithms implemented > > in > > > >> Pig > > > >> (an earlier version of WTF loaded the graph into main memory on > > > large-mem > > > >> machines -- that system is open sourced, too, under > > > >> github.com/twitter/cassovary). Are you proposing to create an > > > open-source > > > >> implementation of those algorithms? Do you suggest they should be > Pig > > > >> scripts added to the Pig project, or do you want to create some new > > > >> operators? I'm not totally sure where you are going here. > > > >> > > > >> GSoC proposals for Pig are usually made by students who want to work > > on > > > >> issues labeled as GSoC candidates on the apache jira. The students > > spend > > > >> some time to understand the problem stated in the jira, familiarize > > > >> themselves with the existing codebase, and put a basic technical > > > >> implementation plan and schedule into their proposal. Since in this > > case > > > >> you are proposing something we haven't scoped or defined well for > > > >> ourselves, we need you to be very clear and specific about what you > > are > > > >> trying to do, and how you plan to go about it. I think that Graph > > > >> processing in Pig (or other Hadoop-based systems) is a really > > > interesting > > > >> topic and there is a lot of work to be done, but we really need you > to > > > be > > > >> far more detailed to be able to give you good guidance with regards > to > > > >> GSoC. > > > >> > > > >> Best, > > > >> Dmitriy > > > >> > > > >> > > > >> On Sat, Mar 30, 2013 at 10:12 AM, burakkk <burak.isi...@gmail.com> > > > wrote: > > > >> > > > >> > Sure. We can implement a graph model using "WTF: The Who to > Follow > > > >> Service > > > >> > at Twitter article we can" article.This article's said that in > this > > > way > > > >> > graph can be stored one machine's memory so that every node will > > read > > > >> from > > > >> > HDFS and cache the graph to the memory. Every node is responsible > > from > > > >> its > > > >> > bucket edge to process. I mean it can be splitted. Every node can > be > > > >> > processed its bucket using random walk algorithm for instance. > > Finally > > > >> it > > > >> > can be reduced to get to the final results. I hope it's clear :) > > > >> > > > > >> > Thanks > > > >> > Best Regards... > > > >> > > > > >> > > > > >> > On Fri, Mar 29, 2013 at 6:10 PM, Dmitriy Ryaboy < > dvrya...@gmail.com > > > > > > >> > wrote: > > > >> > > > > >> > > Hi Burakk, > > > >> > > The general idea of making graph processing easier is a good > one. > > > I'm > > > >> not > > > >> > > sure what exactly you are proposing to do, though. Could you be > > more > > > >> > > detailed about what you are thinking? > > > >> > > > > > >> > > > > > >> > > On Thu, Mar 28, 2013 at 1:28 PM, burakkk < > burak.isi...@gmail.com> > > > >> wrote: > > > >> > > > > > >> > > > Hi, > > > >> > > > I might be a little bit late. I come up with a new idea for > the > > > last > > > >> > > > minute. Currently I'm working on social graph processing. I > > think > > > we > > > >> > can > > > >> > > > implement a solution for pig. With this idea I'm thinking to > > > apply > > > >> the > > > >> > > > GSOC 2013 so that I can do some tasks about it. Is there any > > > mentor > > > >> to > > > >> > do > > > >> > > > it with me? Is there any suggestion? :) > > > >> > > > > > > >> > > > Details: > > > >> > > > Of course I can improve some join operations. I'm not sure is > > > there > > > >> any > > > >> > > > implementation about fuzzy joins for instance. These are the > > > papers > > > >> > that > > > >> > > I > > > >> > > > found > > > >> > > > > > > >> > > > Fuzzy Joins Using MapReduce > > > >> > > > http://ilpubs.stanford.edu:8090/1006/ > > > >> > > > > > > >> > > > Dimension independent similarity computation > > > >> > > > http://arxiv.org/abs/1206.2082 > > > >> > > > > > > >> > > > MapReduce is Good Enough? If All You Have is a Hammer, Throw > > Away > > > >> > > > Everything That’s Not a Nail! > > > >> > > > http://arxiv.org/pdf/1209.2191.pdf > > > >> > > > > > > >> > > > Large Graph Processing in the Cloud > > > >> > > > http://www.ntu.edu.sg/home/bshe/sigmod10_demo.pdf > > > >> > > > > > > >> > > > ..etc > > > >> > > > > > > >> > > > Thanks > > > >> > > > Best regards.. > > > >> > > > > > > >> > > > > > > >> > > > -- > > > >> > > > > > > >> > > > *BURAK ISIKLI** *| *http://burakisikli.wordpress.com* > > > >> > > > * > > > >> > > > * > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > -- > > > >> > > > > >> > *BURAK ISIKLI** *| *http://burakisikli.wordpress.com* > > > >> > * > > > >> > * > > > >> > > > > >> > > > > > > > > > > > > > > > > -- > > > > > > > > *BURAK ISIKLI** *| *http://burakisikli.wordpress.com* > > > > * > > > > * > > > > > > > > > > > > > > > > -- > > > > > > *BURAK ISIKLI** *| *http://burakisikli.wordpress.com* > > > * > > > * > > > > > > > > > -- > > *BURAK ISIKLI** *| *http://burakisikli.wordpress.com* > * > * >
