Thanks Raymond!
yes.. we can make this a config and leave it to the user to decide if they
want to use a global table for all their hudi tables (or) keep
one error table for each hudi table..
For this effort, does it make sense to take a dependency on the
multi-writer jira HUDI-944, that liwei filed?
On Wed, Jun 10, 2020 at 7:49 PM Shiyan Xu <xu.shiyan.raym...@gmail.com>
wrote:
> Yes, Vinoth, it does go a bit too far with first class support on these
> data.
> A global error table can do the job easily. As we discussed yesterday,
> parallel local error tables with `_errors` suffix could also benefit for
> some scenarios, like different product teams manage their own tables or in
> 2B case where customers manage their own data. These would prefer good
> segregation on errors or other related data. Let me note down the points in
> RFC-20 for further discussion. Thanks for the feedback!
>
> On Wed, Jun 3, 2020 at 9:31 PM Vinoth Chandar <vin...@apache.org> wrote:
>
> > Hi Raymond,
> >
> > I am not sure generalizing this to all metadata like - errors and
> metrics -
> > would be a good idea. We can certainly implement logging errors to a
> common
> > errors hudi table, with a certain schema. But these can be just regular
> > “hudi” format tables.
> >
> > Unlike the timeline metadata, these are really external data, not related
> > to a given table’ core functioning.. we don’t necessarily want to keep
> one
> > error table per hudi table..
> >
> > Thoughts?
> >
> > On Tue, Jun 2, 2020 at 5:34 PM Shiyan Xu <xu.shiyan.raym...@gmail.com>
> > wrote:
> >
> > > I also encountered use cases where I'd like to programmatically query
> > > metadata.
> > > +1 on the idea of format(“hudi-timeline”)
> > >
> > > I also feel that the metadata can be extended further to include more
> > info
> > > like, errors, metrics/write statistics, etc. Like the newly proposed
> > error
> > > handling, we could also store all metrics or write stats there too, and
> > > relate them to the timeline actions.
> > >
> > > A potential use case could be, with all these info encapsulated within
> > > metadata, we may be able to derive some insightful results (by check
> > > against some benchmarks) and answer questions like: does table A need
> > more
> > > tuning? does table B exceed error budget?
> > >
> > > Programmatic query to these metadata can help manage many tables in
> > > diagnosis and inspection. We may need different read formats like
> > > format("hudi-errors") or format("hudi-metrics")
> > >
> > > Sorry this sidetracked from the original question..These are really
> rough
> > > high-level thoughts, and may have sign of over-engineering. Would like
> to
> > > hear some feedbacks. Thanks.
> > >
> > >
> > >
> > >
> > > On Mon, Jun 1, 2020 at 9:28 PM Satish Kotha
> <satishko...@uber.com.invalid
> > >
> > > wrote:
> > >
> > > > Got it. I'll look into implementation choices for creating a new data
> > > > source. Appreciate all the feedback.
> > > >
> > > > On Mon, Jun 1, 2020 at 7:53 PM Vinoth Chandar <vin...@apache.org>
> > wrote:
> > > >
> > > > > >Is it to separate data and metadata access?
> > > > > Correct. We already have modes for querying data using
> > format("hudi").
> > > I
> > > > > feel it will get very confusing to mix data and metadata in the
> same
> > > > > source.. for e.g a lot of options we support for data may not even
> > make
> > > > > sense for the TimelineRelation.
> > > > >
> > > > > >This class seems like a list of static methods, I'm not seeing
> where
> > > > these
> > > > > are accessed from
> > > > > That's the public API for obtaining this information for Scala/Java
> > > > Spark.
> > > > > If you have a way of calling this from python through some bridge
> > > without
> > > > > painful bridges (e.g jython), might be a tactical solution that can
> > > meet
> > > > > your needs.
> > > > >
> > > > > On Mon, Jun 1, 2020 at 5:07 PM Satish Kotha
> > > <satishko...@uber.com.invalid
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Thanks for the feedback.
> > > > > >
> > > > > > What is the advantage of doing
> > > > > > spark.read.format(“hudi-timeline”).load(basepath) as opposed to
> > doing
> > > > new
> > > > > > relation? Is it to separate data and metadata access?
> > > > > >
> > > > > > Are you looking for similar functionality as
> > HoodieDatasourceHelpers?
> > > > > > >
> > > > > > This class seems like a list of static methods, I'm not seeing
> > where
> > > > > these
> > > > > > are accessed from. But, I need a way to query metadata details
> > easily
> > > > > > in pyspark.
> > > > > >
> > > > > >
> > > > > > On Mon, Jun 1, 2020 at 8:02 AM Vinoth Chandar <vin...@apache.org
> >
> > > > wrote:
> > > > > >
> > > > > > > Also please take a look at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_HUDI-2D309&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=4xNSsHvHqd0Eym5a_ZpDVwlq_iJaZ0Rdk0u0SMLXZ0c&m=NLHsTFjPharIb29R1o1lWgYLCr1KIZZB4WGPt4IQnOE&s=fGOaSc8PxPJ8yqczQyzYtsqWMEXAbWdeKh-5xltbVG0&e=
> > > > > > > .
> > > > > > >
> > > > > > > This was an effort to make the timeline more generalized for
> > > querying
> > > > > > (for
> > > > > > > a different purpose).. but good to revisit now..
> > > > > > >
> > > > > > > On Sun, May 31, 2020 at 11:04 PM vbal...@apache.org <
> > > > > vbal...@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > I strongly recommend using a separate datasource relation
> > (option
> > > > 1)
> > > > > to
> > > > > > > > query timeline. It is elegant and fits well with spark APIs.
> > > > > > > > Thanks.Balaji.V On Saturday, May 30, 2020, 01:18:45 PM
> PDT,
> > > > Vinoth
> > > > > > > > Chandar <vin...@apache.org> wrote:
> > > > > > > >
> > > > > > > > Hi satish,
> > > > > > > >
> > > > > > > > Are you looking for similar functionality as
> > > > HoodieDatasourceHelpers?
> > > > > > > >
> > > > > > > > We have historically relied on cli to inspect the table,
> which
> > > does
> > > > > not
> > > > > > > > lend it self well to programmatic access.. overall in like
> > option
> > > > 1 -
> > > > > > > > allowing the timeline to be queryable with a standard schema
> > does
> > > > > seem
> > > > > > > way
> > > > > > > > nicer.
> > > > > > > >
> > > > > > > > I am wondering though if we should introduce a new view.
> > Instead
> > > we
> > > > > can
> > > > > > > use
> > > > > > > > a different data source name -
> > > > > > > > spark.read.format(“hudi-timeline”).load(basepath). We can
> start
> > > by
> > > > > just
> > > > > > > > allowing querying of active timeline and expand this to
> archive
> > > > > > timeline?
> > > > > > > >
> > > > > > > > What do other Think?
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, May 29, 2020 at 2:37 PM Satish Kotha
> > > > > > > <satishko...@uber.com.invalid
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello folks,
> > > > > > > > >
> > > > > > > > > We have a use case to incrementally generate data for hudi
> > > table
> > > > > (say
> > > > > > > > > 'table2') by transforming data from other hudi table(say,
> > > > table1).
> > > > > > We
> > > > > > > > want
> > > > > > > > > to atomically store commit timestamps read from table1 into
> > > > table2
> > > > > > > commit
> > > > > > > > > metadata.
> > > > > > > > >
> > > > > > > > > This is similar to how DeltaStreamer operates with kafka
> > > offsets.
> > > > > > > > However,
> > > > > > > > > DeltaStreamer is java code and can easily query kafka
> offset
> > > > > > processed
> > > > > > > by
> > > > > > > > > creating metaclient for target table. We want to use
> pyspark
> > > and
> > > > I
> > > > > > > don't
> > > > > > > > > see a good way to query commit metadata of table1 from
> > > > DataSource.
> > > > > > > > >
> > > > > > > > > I'm considering making one of the below changes to hoodie
> to
> > > make
> > > > > > this
> > > > > > > > > easier.
> > > > > > > > >
> > > > > > > > > Option1: Add new relation in hudi-spark to query commit
> > > metadata.
> > > > > > This
> > > > > > > > > relation would present a 'metadata view' to query and
> filter
> > > > > > metadata.
> > > > > > > > >
> > > > > > > > > Option2: Add other DataSource options on top of incremental
> > > > > querying
> > > > > > to
> > > > > > > > > allow fetching from source table. For example, users can
> > > specify
> > > > > > > > > 'hoodie.consume.metadata.table: table2BasePath' and issue
> > > > > > incremental
> > > > > > > > > query on table1. Then, IncrementalRelation would go read
> > table2
> > > > > > > metadata
> > > > > > > > > first to identify 'consume.start.timestamp' and start
> > > incremental
> > > > > > read
> > > > > > > on
> > > > > > > > > table1 with that timestamp.
> > > > > > > > >
> > > > > > > > > Option 2 looks simpler to implement. But, seems a bit hacky
> > > > because
> > > > > > we
> > > > > > > > are
> > > > > > > > > reading metadata from table2 when data souce is table1.
> > > > > > > > >
> > > > > > > > > Option1 is a bit more complex. But, it is cleaner and not
> > > tightly
> > > > > > > coupled
> > > > > > > > > to incremental reads. For example, use cases other than
> > > > incremental
> > > > > > > reads
> > > > > > > > > can leverage same relation to query metadata
> > > > > > > > >
> > > > > > > > > What do you guys think? Let me know if there are other
> > simpler
> > > > > > > solutions.
> > > > > > > > > Appreciate any feedback.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Satish
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
