[
https://issues.apache.org/jira/browse/HUDI-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu updated HUDI-1655:
-----------------------------
Description:
Add a config to allow parsing custom date format in
{{DatePartitionPathSelector}}. Currently it assumes date partition string in
the format of {{yyyy-MM-dd}}.
Also eligibleFiles.sort() throws this exception
{quote}java.lang.UnsupportedOperationException at
java.util.AbstractList.set(AbstractList.java:132) at
java.util.AbstractList$ListItr.set(AbstractList.java:426) at
java.util.List.sort(List.java:482) at
org.apache.hudi.utilities.sources.helpers.DatePartitionPathSelector.getNextFilePathsAndMaxModificationTime(DatePartitionPathSelector.java:141)
at
org.apache.hudi.utilities.sources.ParquetDFSSource.fetchNextBatch(ParquetDFSSource.java:48)
at org.apache.hudi.utilities.sources.RowSource.fetchNewData(RowSource.java:43)
at org.apache.hudi.utilities.sources.Source.fetchNext(Source.java:75) at
org.apache.hudi.utilities.deltastreamer.SourceFormatAdapter.fetchNewDataInRowFormat(SourceFormatAdapter.java:94)
at
org.apache.hudi.utilities.deltastreamer.DeltaSync.readFromSource(DeltaSync.java:338)
at
org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:255)
at
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:587)
{quote}
was:Add a config to allow parsing custom date format in
{{DatePartitionPathSelector}}. Currently it assumes date partition string in
the format of {{yyyy-MM-dd}}.
> Allow custom date format in DatePartitionPathSelector
> -----------------------------------------------------
>
> Key: HUDI-1655
> URL: https://issues.apache.org/jira/browse/HUDI-1655
> Project: Apache Hudi
> Issue Type: Improvement
> Components: DeltaStreamer
> Reporter: Raymond Xu
> Assignee: Raymond Xu
> Priority: Minor
> Labels: pull-request-available
> Fix For: 0.8.0
>
>
> Add a config to allow parsing custom date format in
> {{DatePartitionPathSelector}}. Currently it assumes date partition string in
> the format of {{yyyy-MM-dd}}.
>
> Also eligibleFiles.sort() throws this exception
> {quote}java.lang.UnsupportedOperationException at
> java.util.AbstractList.set(AbstractList.java:132) at
> java.util.AbstractList$ListItr.set(AbstractList.java:426) at
> java.util.List.sort(List.java:482) at
> org.apache.hudi.utilities.sources.helpers.DatePartitionPathSelector.getNextFilePathsAndMaxModificationTime(DatePartitionPathSelector.java:141)
> at
> org.apache.hudi.utilities.sources.ParquetDFSSource.fetchNextBatch(ParquetDFSSource.java:48)
> at
> org.apache.hudi.utilities.sources.RowSource.fetchNewData(RowSource.java:43)
> at org.apache.hudi.utilities.sources.Source.fetchNext(Source.java:75) at
> org.apache.hudi.utilities.deltastreamer.SourceFormatAdapter.fetchNewDataInRowFormat(SourceFormatAdapter.java:94)
> at
> org.apache.hudi.utilities.deltastreamer.DeltaSync.readFromSource(DeltaSync.java:338)
> at
> org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:255)
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:587)
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
