If you share your current setup : HDFS vs S3, Yarn vs Mesos vs K8, Source of data etc. We would be able to effectively offer suggestions.
I will created a JIRA https://issues.apache.org/jira/browse/HUDI-3 to track the Glue guide as well. On Tue, Mar 5, 2019 at 12:11 PM Vinoth Chandar <vin...@apache.org> wrote: > IIUC Glue is a service to easily schedule spark jobs in AWS. To be clear, > its not a pre-requisite for hudi. Is this your current means of scheduling > spark jobs? > > On Tue, Mar 5, 2019 at 10:27 AM Umesh Kacha <umesh.ka...@gmail.com> wrote: > >> Hi does anybody can help with steps using Hudi in AWS ecosystem if >> possible >> using AWS Glue then it would be a really great help and start for me and I >> think for others as well looking for similar help? >> >> Thanks in advance. >> >> Regards, >> Umesh >> >> On Tue, Mar 5, 2019, 2:02 AM Vinoth Chandar <vin...@apache.org> wrote: >> >> > It did need AWS to enable a way to drop the hudi jars/support custom >> presto >> > plugins into the provisioned cluster. >> > I quickly checked the Athena docs again. Seems like this is still a >> gap. :| >> > >> > >> > >> > On Mon, Mar 4, 2019 at 12:22 PM Brandon Geise <brandonge...@gmail.com> >> > wrote: >> > >> > > Is it possible to get Athena working with Hudi or does it require AWS >> to >> > > add support directly? >> > > >> > > Thanks >> > > Brandon >> > > >> > > On 3/4/19, 1:52 PM, "Vinoth Chandar" <vin...@apache.org> wrote: >> > > >> > > +1 >> > > >> > > Hudi writer is just a spark job. We have seen folks use EMR and >> Glue >> > > like >> > > any other spark job, write data out to s3. >> > > On Presto, again I have seen it being done if you are running >> Presto >> > > on a >> > > bunch of ec2 machines. >> > > >> > > We have had discussions with AWS Athena on supporting this >> > out-of-box. >> > > But >> > > there were some blockers on their side to take it forward at that >> > time. >> > > >> > > Thanks >> > > Vinoth >> > > >> > > On Mon, Mar 4, 2019 at 4:54 AM Kabeer Ahmed <kab...@linuxmail.org >> > >> > > wrote: >> > > >> > > > Hi Umesh, >> > > > >> > > > We use it on AWS. So it definitely works on AWS (S3). I have >> read >> > > that >> > > > there is support for Presto too but we havent used Presto due to >> > > > compatibility issues between Apache Ranger and Presto. >> > > > Thanks, >> > > > Kabeer. >> > > > >> > > > On Mar 4 2019, at 10:21 am, Umesh Kacha <umesh.ka...@gmail.com> >> > > wrote: >> > > > > Hi is there out of the box support for Hudi inside AWS just >> like >> > > Presto? >> > > > > >> > > > > Thanks in advance. >> > > > > Regards, >> > > > > Umesh >> > > > > >> > > > >> > > > >> > > >> > > >> > > >> > > >> > >> >
