Hi Lahiru, As per the offline chat we had, please start a new mail thread on adding design details, related to approach ( writing custom source and sink connectors [3] ) you have taken on implementation of Jet execution engine support. That way community can comment on your project work.
[1] https://jet.hazelcast.org/connectors/batch-connectors/ Regards Kevin On Sun, May 19, 2019 at 1:15 PM Lahiru Jayasekera <mlpjayasek...@gmail.com> wrote: > Hi all, > This is the[1] hdfs connector for hazelcast jet. > > Here the problem is hazelcast jet is using `org.apache.hadoop.mapred` > package and gora is using `org.apache.hadoop.mapreduce` package. > > These are two APIs exposed by hadoop to create hadoop jobs and etc. > Eventhough these two are similar in functionality, both the packages are > shiped with hadoop. See this answer[2], > > So I thought of writing an InputFormat for gora with the support of older > package(`org.apache.hadoop.mapred`). > > Please correct me if I'm going in the wrong direction. Your Feedback is > appreciated. > > [1] https://docs.hazelcast.org/docs/jet/0.7/manual/#hdfs > [2] https://stackoverflow.com/a/7600339 > > Thanks and regards > Lahiru > > On Fri, Apr 26, 2019 at 6:32 PM Lahiru Jayasekera <mlpjayasek...@gmail.com > > > wrote: > > > Hi Madhawa, > > Sorry for the late reply. Sure I'll try that and let you know. > > > > Thanks > > > > On Tue, Apr 23, 2019 at 2:24 PM Madhawa Kasun Gunasekara < > > madhaw...@gmail.com> wrote: > > > >> Hi Lahiru, > >> > >> Good initiative. > >> It seems like we need to use hadoop-hdfs and hadoop common with version > >> 2.8.3. Try adding these dependencies, and exclude the old dependency > from > >> the gora, or otherwise, we can do a version upgrade in gora for Hadoop. > at > >> the moment we use Hadoop version 2.5.2 in gora but I prefer to upgrade > the > >> Hadoop versions in the gora to 2.8.3. > >> > >> Recently hazelcast developers released 3.0 version, We can try that > also. > >> > >> Thanks, > >> Madhawa > >> > >> > >> On Sat, Apr 6, 2019 at 9:53 PM Lahiru Jayasekera < > mlpjayasek...@gmail.com > >> > > >> wrote: > >> > >> > Hi all, > >> > I have come up with a poc for hazelcast jet execution engine support > for > >> > gora. > >> > > >> > > >> > https://github.com/LahiruJayasekara/gora/blob/poc-hazelcast-jet/gora-tutorial/src/main/java/org/apache/gora/tutorial/log/HazelcastJetPOC.java > >> > > >> > Here I have wrote a custom source for jet. What this does is read the > >> > AccessLog table created via LogManager example, and feed the PageView > >> > objects to jet. > >> > > >> > I have also looked into using hdfs connectors of jet. But I > encountered > >> a > >> > problem with conflicting hadoop versions. InputFormat class used in > >> gora is > >> > from pacjage 'org.apache.hadoop.mapreduce', but in jet it is ' > >> > org.apache.hadoop.mapred.InputFormat'. > >> > > >> > Anyway the proposed poc works as expected. Please add your suggestions > >> > here. > >> > > >> > Thanks and regards > >> > > >> > -- > >> > Lahiru Jayasekara > >> > Batch 15 > >> > Faculty of Information Technology > >> > University of Moratuwa > >> > 0716492170 > >> > > >> > > > > > > -- > > Lahiru Jayasekara > > Batch 15 > > Faculty of Information Technology > > University of Moratuwa > > 0716492170 > > > > > -- > Lahiru Jayasekara > Batch 15 > Faculty of Information Technology > University of Moratuwa > 0716492170 >
