[
https://issues.apache.org/jira/browse/HUDI-64?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973099#comment-16973099
]
vinoyang commented on HUDI-64:
------------------------------
[~vinoth] I have tried to find an alternative solution to handle the third
problem. It's hard to process based on Flink. There are two points:
* rdd.persist/cache: it has not been supported yet. There is a JIRA issue
FLINK-12352(another old one: FLINK-10867) describe this feature, but it has not
been finished.
* partition based on dynamic statistics (or pre-condition) has not been
supported too, this feature is also suffering from the prior issue.
In short, Flink is trying to support caching and sharing intermediate results
(they called "Interactive Programming"), but the feature has not been finished.
For now, if we want to implement this. IMO, it needs to be split into two
single jobs. The first job used to calculate the workload profiles. The second
one used to do the remaining work based on the workload profile. They may be
scheduled or coordinated by the outside. In addition, they also need to share
the workload profiles based on distributed cache or file system. No doubt, the
effect is obviously not as good as Spark.
> Estimation of compression ratio & other dynamic storage knobs based on
> historical stats
> ---------------------------------------------------------------------------------------
>
> Key: HUDI-64
> URL: https://issues.apache.org/jira/browse/HUDI-64
> Project: Apache Hudi (incubating)
> Issue Type: New Feature
> Components: Storage Management, Write Client
> Reporter: Vinoth Chandar
> Assignee: Vinoth Chandar
> Priority: Major
>
> Something core to Hudi writing is using heuristics or runtime workload
> statistics to optimize aspects of storage like file sizes, partitioning and
> so on.
> Below lists all such places.
>
> # Compression ratio for parquet
> [https://github.com/apache/incubator-hudi/blob/a4f9d7575f39bb79089714049ffea12ba5f25ec8/hudi-client/src/main/java/org/apache/hudi/config/HoodieStorageConfig.java#L46]
> . This is used by HoodieWrapperFileSystem, to estimate amount of bytes it
> has written for a given parquet file and closes the parquet file once the
> configured size has reached. DFSOutputStream level we only know bytes written
> before compression. Once enough data has been written, it should be possible
> to replace this by a simple estimate of what the avg record size would be
> (commit metadata would give you size and number of records in each file)
> # Very similar problem exists for log files
> [https://github.com/apache/incubator-hudi/blob/a4f9d7575f39bb79089714049ffea12ba5f25ec8/hudi-client/src/main/java/org/apache/hudi/config/HoodieStorageConfig.java#L52]
> We write data into logs in avro and can log updates to same record in
> parquet multiple times. We need to estimate again how large the log file(s)
> can grow to, and still we would be able to produce a parquet file of
> configured size during compaction. (hope I conveyed this clearly)
> # WorkloadProfile :
> [https://github.com/apache/incubator-hudi/blob/b19bed442d84c1cb1e48d184c9554920735bcb6c/hudi-client/src/main/java/org/apache/hudi/table/WorkloadProfile.java]
> caches the input records using Spark Caching and computes the shape of the
> workload, i.e how many records per partition, how many inserts vs updates
> etc. This is used by the Partitioner here
> [https://github.com/apache/incubator-hudi/blob/b19bed442d84c1cb1e48d184c9554920735bcb6c/hudi-client/src/main/java/org/apache/hudi/table/HoodieCopyOnWriteTable.java#L141]
> for assigning records to a file group. This is the critical one to replace
> for Flink support and probably the hardest, since we need to guess input,
> which is not always possible?
> # Within partitioner, we already derive a simple average size per record
> [https://github.com/apache/incubator-hudi/blob/b19bed442d84c1cb1e48d184c9554920735bcb6c/hudi-client/src/main/java/org/apache/hudi/table/HoodieCopyOnWriteTable.java#L756]
> from the last commit metadata alone. This can be generalized. (default :
> [https://github.com/apache/incubator-hudi/blob/b19bed442d84c1cb1e48d184c9554920735bcb6c/hudi-client/src/main/java/org/apache/hudi/config/HoodieCompactionConfig.java#L71])
>
> #
> Our goal in this Jira is to see, if could derive this information in the
> background purely using the commit metadata.. Some parts of this are
> open-ended.. Good starting point would be to see whats feasible, estimate ROI
> before aactually implementing
>
>
>
>
>
>
> Roughly along the likes of. [https://github.com/uber/hudi/issues/270]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
