morningman commented on code in PR #44267:
URL: https://github.com/apache/doris/pull/44267#discussion_r1865305204
##########
fe/be-java-extensions/hudi-scanner/src/main/scala/org/apache/doris/hudi/BaseSplitReader.scala:
##########
@@ -497,8 +499,11 @@ abstract class BaseSplitReader(val split: HoodieSplit) {
options: Map[String, String],
hadoopConf: Configuration): PartitionedFile =>
Iterator[InternalRow] = {
partitionedFile => {
- val reader = new HoodieAvroHFileReader(
- hadoopConf, partitionedFile.filePath.toPath, new
CacheConfig(hadoopConf))
+ var hadoopStorageConfiguration = new
HadoopStorageConfiguration(hadoopConf);
+ var sotragePath = new StoragePath(partitionedFile.toPath.toUri.getPath);
Review Comment:
```suggestion
var storagePath = new
StoragePath(partitionedFile.toPath.toUri.getPath);
```
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java:
##########
@@ -142,6 +141,19 @@ public static <T> String encodeObjectToString(T t) {
@Override
protected void setScanParams(TFileRangeDesc rangeDesc, Split split) {
if (split instanceof PaimonSplit) {
+ PaimonSplit paimonSplit = (PaimonSplit) split;
+ if (rangeDesc.getFormatType() == TFileFormatType.FORMAT_JNI
+ && paimonSplit.getSplit() == null) {
+ // fall back to JNI reader
Review Comment:
```suggestion
// fall back to native reader
```
##########
be/src/vec/exec/scan/vfile_scanner.cpp:
##########
@@ -754,28 +750,10 @@ Status VFileScanner::_get_next_reader() {
// create reader for specific format
Status init_status;
- TFileFormatType::type format_type = _params->format_type;
+ TFileFormatType::type format_type = range.format_type;
// JNI reader can only push down column value range
bool push_down_predicates =
!_is_load && _params->format_type !=
TFileFormatType::FORMAT_JNI;
- if (format_type == TFileFormatType::FORMAT_JNI &&
range.__isset.table_format_params) {
Review Comment:
This need to be reserved.
Because when upgrading, the BE will be upgrade, and the old FE will send old
format request to new BE.
You can reserve this logic but mark it as deprecated in 3.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
