kasakrisz commented on PR #6701:
URL: https://github.com/apache/hive/pull/6701#issuecomment-5585129193
> > @Aggarwal-Raghav , @deniskuzZ
> > My concern with how we currently determine the writer implementation is
that we select it at compile time, but it can later be overridden in
`WriterBuilder`.
> > I explored an alternative approach. Right now, the writer implementation
depends on two factors:
> >
> > * Write operation
> > * Copy-On-Write mode (IIUC, this PR calculates COW inside
`WriterBuilder` using the statement operation)
> >
> > What if we add a new enum constant inside `Context.Operation` to merge
the two?
> > ```
> > public enum Operation {UPDATE, DELETE, MERGE, IOW, OTHER, COW}
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > And move this logic
> > ```
> > boolean isCOW = IcebergTableUtil.isCopyOnWriteMode(operation,
table.properties()::getOrDefault);
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > from `WriterBuilder` to `SemanticAnalyzer`
> > ```
> > private Context.Operation getWriteOperation(String destination, Table
destinatonTable) {
> > if (destinatonTable != null &&
> > destinatonTable.getStorageHandler() != null &&
> >
destinatonTable.getStorageHandler().shouldOverwrite(destinatonTable,
ctx.getOperation())) {
> > return Context.Operation.COW;
> > }
> >
> > return deleting(destination) ? Context.Operation.DELETE :
> > updating(destination) ? Context.Operation.UPDATE :
> > merging(destination) ? Context.Operation.MERGE :
Context.Operation.OTHER;
> > }
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > We can check whether the operation is `Operation.COW` in the
`WriterBuilder` and in `HiveIcebergSerDe`.
> > WDYT?
>
> COW is not an OPERATION , it's a write mode: COW or MOR. Problem is with
how SA handles merge and update operations in Hive - rewrites them stripping
the original operation context/type.
In general, it is allowed to have more than one writer for a statement in
Hive. For example: split update, multi-insert.
1. You mentioned that these have to be in sync:
> WriterBuilder re-derives COW/MOR from table properties at runtime, and it
must land on the same answer the rewriter factory already computed at plan time:
I think the easiest way to make them sync is passing these values from the
compiler to the `WriterBuilder` and don't let the `WriterBuilder` override them.
2. Can we pass Write Mode (COW or MOR) as a second parameter to
the`WriterBuilder` instead of passing two Operations?
3. Does it matter what the write operation is in case of COW? IIUC we always
select `HiveIcebergCopyOnWriteRecordWriter` when write mode is COW.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
