Hi Lefteris, Sorry for the late reply, In the case of spark you need to provide a configuration like configuration on the test [1], also you can add the configuration as the one in the spark configuration [2]
In the case of postgres you can take as an example the configuration of postgres platform [3] the same is true for graphchi [4]. If they are in different machines you just need to give in the path with the ip of the VM and the only thing that you need to be sure is that all the machines in the network are reachable(I mean not firewall rule that can denied the access), and if you need more help just let me know. Best regards, Bertty [1] https://github.com/apache/incubator-wayang/blob/main/wayang-benchmark/code/test/resources/rheem.properties [2] https://github.com/apache/incubator-wayang/blob/main/wayang-platforms/wayang-spark/code/main/resources/wayang-spark-defaults.properties [3] https://github.com/apache/incubator-wayang/blob/main/wayang-platforms/wayang-postgres/src/main/resources/wayang-postgres-defaults.properties [4] https://github.com/apache/incubator-wayang/blob/main/wayang-platforms/wayang-graphchi/src/main/resources/wayang-graphchi-defaults.properties On Thu, Jan 27, 2022 at 4:59 PM Lefteris Lymperopoulos < [email protected]> wrote: > Hello and thank you for your reply, > I want to run benchmarks from TPC such like this > http://tpc.org/tpcds/default5.asp which consist of simple SQL queries such > as UPDATE, DELETE and JOIN in order to measure Wayang's performance when > connected to Spark, Postgres and GraphChi. Could you please provide me with > more details about how to connect those three systems to Wayang as well as > Spark, Postgres and GraphChi plugins, especially when they run on different > VMs? Excuse me if I misunderstood something, I am a student with no prior > experience in these systems. Thank you for your time. > Best regards, > Lefteris > > On Tue, Jan 25, 2022 at 10:35 PM Bertty Contreras <[email protected]> > wrote: > > > Hi Lefteris, > > > > It is possible to run the query that you describe using Apache Wayang; > one > > example that could help you understand how to connect different platforms > > in one task is the Query3Hybrid [1]. Nevertheless, if you could explain > > more about the shape of the query will look, we can give you more hints > on > > how you could do it. > > > > I mean with a shape something like the following example: > > > > The need to run an SQL query in top Postgres, then join the result with > > data stored in HDFS or S3 and transform it to execute a page rank using > > Graphchi; after the results are processed, the information needs to be > > joined with another dataset. > > > > Let me know the shape, and we will be able provide more information. > > > > Thank you so much for your question; this will help us a lot to know what > > to write in the documentation in the example section :D > > > > Best regards, > > Bertty > > > > [1] > > > > > https://github.com/apache/incubator-wayang/blob/main/wayang-benchmark/code/main/scala/org/apache/wayang/apps/tpch/queries/Query3Hybrid.scala > > > > On Tue, Jan 25, 2022 at 4:08 PM Lefteris Lymperopoulos < > > [email protected]> wrote: > > > > > Hello Dev Team, > > > Excuse me if this mail isn't appropriate for this mail address. Since I > > > could not find any documentation for Wayang besides that in the > official > > > website I would like to ask you if you could help me with this issue. I > > > have 3 VMs that connect to each other. In the first VM I want to run > > Spark, > > > in the second I want to run Postgres and in the third GraphChi. I also > > have > > > Wayang installed in the first VM and I intend to develop my app in > Java. > > Is > > > it possible to connect Wayang to Spark, Postgres and GraphChi in order > to > > > run my queries? If yes, could you please show me how to do it? Or do > > these > > > platforms have to be in the same VM? Any help would be greatly > > appreciated. > > > Best regards, > > > Lefteris Lymperopoulos > > > > > >
