Re: HoodieMultiTableDeltaStreamer failing due to missing file path delimiter
Hi Pratyaksh,
Thank you very much for your help. After reading through your advice, the
workaround I came to was setting `--config-folder s3:\/\/{{
s3_config_bucket }}\/\/` (note the double forward slash at the end). This
circumvents the slash being deleted at the end of the `--config-folder`
parameter [1]. I am still unsure as to what the root cause of the problem
is, but I can move forward with my work using the work around. Thank you
again for your assistance!
Best wishes,
Philip
[1]
https://github.com/apache/hudi/blob/da65d3cae99e8fee0ede9b5ed8630a3716d284c8/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java#L345
On Fri, Oct 8, 2021 at 4:43 PM Pratyaksh Sharma
wrote:
> Hi Philip,
>
> I checked the configs that you are passing and it all looks good. Indeed
> the problem is the absence of forward slash which should not happen in
> general. Can you try printing the configs once and see if the configFile
> path is getting passed properly?
>
> Also as a workaround, you can create `s3://{{ bucket_name }}/{{
> hive_database }}_{{ hive_table }}_config.properties` as the properties file
> for table overridden properties and not mention the property
> `hoodie.deltastreamer.ingestion.{{ hive_database }}.{{ hive_table
> }}.configFile` at all in deltastreamer.properties file. You can find more
> information here -
> https://hudi.apache.org/blog/2020/08/22/ingest-multiple-tables-using-hudi.
>
> Hope that helps!
>
> On Wed, Oct 6, 2021 at 4:36 PM Philip Ittmann >
> wrote:
>
> > Good day,
> >
> > I am experiencing difficulty in getting a HoodieMultiTableDeltaStreamer
> > application to successfully run via spark-submit on an AWS EMR cluster
> with
> > the following versions:
> >
> > Hudi release: 0.7.0
> >
> > Release label:emr-6.3.0
> > Hadoop distribution:Amazon 3.2.1
> > Applications:Tez 0.9.2, Spark 3.1.1, Hive 3.1.2, Presto 0.245.1
> >
> > The error I am seeing follows below, but the gist of the problem seems to
> > be related to leaving out a forward slash between the bucket name and the
> > filename when initializing a Path object.
> >
> > After the error stack trace I include the spark-submit command as well as
> > the properties file I am using. Any help would be greatly appreciated!
> >
> > Exception in thread "main" java.lang.IllegalArgumentException:
> > java.net.URISyntaxException: Relative path in absolute URI: s3://{{
> > bucket_name }}{{ hive_database }}.{{ hive_table }}.properties
> > at org.apache.hadoop.fs.Path.initialize(Path.java:263)
> > at org.apache.hadoop.fs.Path.(Path.java:161)
> > at org.apache.hadoop.fs.Path.(Path.java:119)
> > at
> >
> >
> org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer.checkIfTableConfigFileExists(HoodieMultiTableDeltaStreamer.java:99)
> > at
> >
> >
> org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer.populateTableExecutionContextList(HoodieMultiTableDeltaStreamer.java:116)
> > at
> >
> >
> org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer.(HoodieMultiTableDeltaStreamer.java:80)
> > at
> >
> >
> org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer.main(HoodieMultiTableDeltaStreamer.java:203)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at
> >
> >
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> > at org.apache.spark.deploy.SparkSubmit.org
> > $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:959)
> > at
> > org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
> > at
> > org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
> > at
> > org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
> > at
> >
> >
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1038)
> > at
> > org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1047)
> > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> > Caused by: java.net.URISyntaxException: Relative path in absolute URI:
> > s3://{{ bucket_name }}{{ hive_database }}.{{ hive_table }}.properties
> > at java.net.URI.checkPath(URI.java:1823)
> > at java.net.URI.(URI.java:745)
> > at org.apache.hadoop.fs.Path.initialize(Path.java:260)
> > ... 18 more
> >
> > The spark-submit command I am running is:
> >
> > spark-submit \
> > --class
> > org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer \
> > --master yarn \
> > --deploy-mode client \
> > --num-executors 10 \
> > --executor-memory 3g \
> > --driver-memory 6g \
Re: Monthly or Bi-Monthly Dev meeting?
Hi Vinoth, Summertime 8 AM PST was 11 PM in China so I guess it works for some forks, but switching to wintertime it was 12 AM in China. It might be a bit late IMO. Does 3 PM UTC(7 AM PST in winter, 8 AM in summer) work? Best, Gary On Tue, Oct 5, 2021 at 9:20 PM Pratyaksh Sharma wrote: > Works for me in India :) > > On Tue, Oct 5, 2021 at 9:41 AM Vinoth Chandar wrote: > > > Looks like there is enough interest here. > > > > Moving onto timing. Does 8AM PST, on the second thursday of every > > month work for everyone? > > This is the time I find, works best for most time zones. > > > > On Thu, Sep 23, 2021 at 1:15 PM Y Ethan Guo > > wrote: > > > > > +1 on monthly community sync. > > > > > > On Thu, Sep 23, 2021 at 12:32 PM Udit Mehrotra > > wrote: > > > > > > > +1 for the monthly meeting. It would be great to start syncing up > > > > again. Thanks Vinoth for bringing it up ! > > > > > > > > On Thu, Sep 23, 2021 at 12:14 PM Sivabalan > wrote: > > > > > > > > > > +1 on monthly meet up. > > > > > > > > > > On Thu, Sep 23, 2021 at 11:01 AM vino yang > > > > wrote: > > > > > > > > > > > +1 for monthly > > > > > > > > > > > > Best, > > > > > > Vino > > > > > > > > > > > > Pratyaksh Sharma 于2021年9月23日周四 下午9:36写道： > > > > > > > > > > > > > Monthly should be good. Been a long time since we connected in > > > these > > > > > > > meetings. :) > > > > > > > > > > > > > > On Thu, Sep 23, 2021 at 7:02 PM Vinoth Chandar < > > > > > > > mail.vinoth.chan...@gmail.com> wrote: > > > > > > > > > > > > > > > 1 hour monthly is what I was proposing to be specific. > > > > > > > > > > > > > > > > On Thu, Sep 23, 2021 at 6:30 AM Gary Li > > > wrote: > > > > > > > > > > > > > > > > > +1 for monthly. > > > > > > > > > > > > > > > > > > On Thu, Sep 23, 2021 at 8:28 PM Vinoth Chandar < > > > > vin...@apache.org> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > Once upon a time, we used to have a weekly community > sync. > > > > > > Wondering > > > > > > > if > > > > > > > > > > there is interest in having a monthly or bi-monthly dev > > > > meeting? > > > > > > > > > > > > > > > > > > > > Agenda could be > > > > > > > > > > - Update/Summary of all dev work tracks > > > > > > > > > > - Show and tell, where people can present their ongoing > > work > > > > > > > > > > - Open floor discussions, bring up new issues. > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > Vinoth > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, > > > > > -Sivabalan > > > > > > > > > >
