vinishjail97 commented on code in PR #704:
URL: https://github.com/apache/incubator-xtable/pull/704#discussion_r2074022766
##########
pom.xml:
##########
@@ -352,6 +358,16 @@
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>org.antlr</groupId>
+ <artifactId>antlr4-runtime</artifactId>
Review Comment:
Is this coming from hadoop libraries ? You can run `mvn dependency:tree
-Dincludes ` to find out which one's conflicting.
##########
xtable-service/src/main/java/org/apache/xtable/service/ConversionResource.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.xtable.service;
+
+import org.apache.xtable.service.models.ConvertTableRequest;
+import org.apache.xtable.service.models.ConvertTableResponse;
+
+import io.smallrye.common.annotation.Blocking;
+import jakarta.inject.Inject;
+import jakarta.ws.rs.Consumes;
+import jakarta.ws.rs.POST;
+import jakarta.ws.rs.Path;
+import jakarta.ws.rs.Produces;
+import jakarta.ws.rs.core.MediaType;
+
+@Path("/v1/conversion")
+@Produces(MediaType.APPLICATION_JSON)
+@Consumes(MediaType.APPLICATION_JSON)
+public class ConversionResource {
+
+ @Inject ConversionService conversionService;
+
+ @POST
+ @Path("/table")
+ @Blocking
Review Comment:
Noob question - quarkus uses an ExecutorService for executing the requests
or event loop? `@Blocking` Can you check this doesn't block the execution of
other conversion requests for different base paths?
##########
xtable-service/src/main/java/org/apache/xtable/service/models/RestTargetTable.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.xtable.service.models;
+
+import lombok.Builder;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+@Builder
+public class RestTargetTable {
Review Comment:
If we switch the protocol to gRPC this class will no longer be valid?
TargetTableDTO might be a better name?
##########
xtable-service/src/main/java/org/apache/xtable/service/ConversionService.java:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.xtable.service;
+
+import static org.apache.xtable.model.storage.TableFormat.DELTA;
+import static org.apache.xtable.model.storage.TableFormat.HUDI;
+import static org.apache.xtable.model.storage.TableFormat.ICEBERG;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+
+import org.apache.iceberg.Snapshot;
+
+import org.apache.xtable.conversion.ConversionConfig;
+import org.apache.xtable.conversion.ConversionController;
+import org.apache.xtable.conversion.ConversionSourceProvider;
+import org.apache.xtable.conversion.SourceTable;
+import org.apache.xtable.conversion.TargetTable;
+import org.apache.xtable.delta.DeltaConversionSourceProvider;
+import org.apache.xtable.hudi.HudiConversionSourceProvider;
+import org.apache.xtable.iceberg.IcebergConversionSourceProvider;
+import org.apache.xtable.service.models.ConvertTableRequest;
+import org.apache.xtable.service.models.ConvertTableResponse;
+import org.apache.xtable.service.models.RestTargetTable;
+import org.apache.xtable.service.spark.SparkHolder;
+import org.apache.xtable.service.utils.DeltaMetadataUtil;
+import org.apache.xtable.service.utils.HudiMedataUtil;
+import org.apache.xtable.service.utils.IcebergMetadataUtil;
+
+import jakarta.enterprise.context.ApplicationScoped;
+import jakarta.inject.Inject;
+
+@ApplicationScoped
+public class ConversionService {
+ private final SparkHolder sparkHolder;
+ private final ConversionControllerFactory controllerFactory;
+ private final IcebergMetadataUtil icebergUtil;
+ private final HudiMedataUtil hudiUtil;
+ private final DeltaMetadataUtil deltaUtil;
+
+ @Inject
+ public ConversionService(
+ SparkHolder sparkHolder,
+ ConversionControllerFactory controllerFactory,
+ IcebergMetadataUtil icebergUtil,
+ HudiMedataUtil hudiUtil,
+ DeltaMetadataUtil deltaUtil) {
+ this.sparkHolder = sparkHolder;
+ this.controllerFactory = controllerFactory;
+ this.icebergUtil = icebergUtil;
+ this.hudiUtil = hudiUtil;
+ this.deltaUtil = deltaUtil;
+ }
+
+ public ConvertTableResponse convertTable(ConvertTableRequest request) {
+ Configuration conf = sparkHolder.jsc().hadoopConfiguration();
+ ConversionController conversionController = controllerFactory.create(conf);
+
+ SourceTable sourceTable =
+ SourceTable.builder()
+ .name(request.getSourceTableName())
+ .basePath(request.getSourceTablePath())
+ .formatName(request.getSourceFormat())
+ .build();
+
+ List<TargetTable> targetTables = new ArrayList<>();
+ for (String targetFormat : request.getTargetFormats()) {
+ TargetTable targetTable =
+ TargetTable.builder()
+ .name(request.getSourceTableName())
+ .basePath(request.getSourceTablePath())
+ .formatName(targetFormat)
Review Comment:
These should be target format and paths?
##########
xtable-service/src/main/java/org/apache/xtable/service/utils/IcebergMetadataUtil.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.xtable.service.utils;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.SchemaParser;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableMetadata;
+import org.apache.iceberg.TableOperations;
+import org.apache.iceberg.hadoop.HadoopTables;
+
+import jakarta.enterprise.context.ApplicationScoped;
+
+@ApplicationScoped
+public class IcebergMetadataUtil {
Review Comment:
unit tests?
##########
xtable-service/src/main/java/org/apache/xtable/service/ConversionService.java:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.xtable.service;
+
+import static org.apache.xtable.model.storage.TableFormat.DELTA;
+import static org.apache.xtable.model.storage.TableFormat.HUDI;
+import static org.apache.xtable.model.storage.TableFormat.ICEBERG;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+
+import org.apache.iceberg.Snapshot;
+
+import org.apache.xtable.conversion.ConversionConfig;
+import org.apache.xtable.conversion.ConversionController;
+import org.apache.xtable.conversion.ConversionSourceProvider;
+import org.apache.xtable.conversion.SourceTable;
+import org.apache.xtable.conversion.TargetTable;
+import org.apache.xtable.delta.DeltaConversionSourceProvider;
+import org.apache.xtable.hudi.HudiConversionSourceProvider;
+import org.apache.xtable.iceberg.IcebergConversionSourceProvider;
+import org.apache.xtable.service.models.ConvertTableRequest;
+import org.apache.xtable.service.models.ConvertTableResponse;
+import org.apache.xtable.service.models.RestTargetTable;
+import org.apache.xtable.service.spark.SparkHolder;
+import org.apache.xtable.service.utils.DeltaMetadataUtil;
+import org.apache.xtable.service.utils.HudiMedataUtil;
+import org.apache.xtable.service.utils.IcebergMetadataUtil;
+
+import jakarta.enterprise.context.ApplicationScoped;
+import jakarta.inject.Inject;
+
+@ApplicationScoped
+public class ConversionService {
+ private final SparkHolder sparkHolder;
+ private final ConversionControllerFactory controllerFactory;
+ private final IcebergMetadataUtil icebergUtil;
+ private final HudiMedataUtil hudiUtil;
+ private final DeltaMetadataUtil deltaUtil;
+
+ @Inject
+ public ConversionService(
+ SparkHolder sparkHolder,
+ ConversionControllerFactory controllerFactory,
+ IcebergMetadataUtil icebergUtil,
+ HudiMedataUtil hudiUtil,
+ DeltaMetadataUtil deltaUtil) {
+ this.sparkHolder = sparkHolder;
+ this.controllerFactory = controllerFactory;
+ this.icebergUtil = icebergUtil;
+ this.hudiUtil = hudiUtil;
+ this.deltaUtil = deltaUtil;
+ }
+
+ public ConvertTableResponse convertTable(ConvertTableRequest request) {
+ Configuration conf = sparkHolder.jsc().hadoopConfiguration();
+ ConversionController conversionController = controllerFactory.create(conf);
+
+ SourceTable sourceTable =
+ SourceTable.builder()
+ .name(request.getSourceTableName())
+ .basePath(request.getSourceTablePath())
+ .formatName(request.getSourceFormat())
+ .build();
+
+ List<TargetTable> targetTables = new ArrayList<>();
+ for (String targetFormat : request.getTargetFormats()) {
+ TargetTable targetTable =
+ TargetTable.builder()
+ .name(request.getSourceTableName())
+ .basePath(request.getSourceTablePath())
+ .formatName(targetFormat)
+ .build();
+ targetTables.add(targetTable);
+ }
+ ConversionConfig conversionConfig =
+
ConversionConfig.builder().sourceTable(sourceTable).targetTables(targetTables).build();
+ ConversionSourceProvider<?> conversionSourceProvider =
+ getConversionSourceProvider(request.getSourceFormat());
+ conversionController.sync(conversionConfig, conversionSourceProvider);
+
+ List<RestTargetTable> restTargetTables = new ArrayList<>();
+ for (String targetFormat : request.getTargetFormats()) {
+ if (targetFormat.equals("ICEBERG")) {
+ Pair<String, String> responseFields =
+ icebergUtil.getIcebergSchemaAndMetadataPath(
+ request.getSourceTablePath(),
sparkHolder.jsc().hadoopConfiguration());
+ RestTargetTable icebergTable =
+ new RestTargetTable("ICEBERG", responseFields.getLeft(),
responseFields.getRight());
+ restTargetTables.add(icebergTable);
+ } else if (targetFormat.equals("HUDI")) {
+ Pair<String, String> responseFields =
+ hudiUtil.getHudiSchemaAndMetadataPath(
+ request.getSourceTablePath(),
sparkHolder.jsc().hadoopConfiguration());
+ RestTargetTable hudiTable =
+ new RestTargetTable("HUDI", responseFields.getLeft(),
responseFields.getRight());
+ restTargetTables.add(hudiTable);
+ } else if(targetFormat.equals("DELTA")){
+ Pair<String, String> responseFields =
+ deltaUtil.getDeltaSchemaAndMetadataPath(
+ request.getSourceTablePath(), sparkHolder.spark());
+ RestTargetTable deltaTable =
+ new RestTargetTable("DELTA", responseFields.getLeft(),
responseFields.getRight());
+ restTargetTables.add(deltaTable);
+ }
+ }
+ return new ConvertTableResponse(restTargetTables);
+ }
+
+ private ConversionSourceProvider<?> getConversionSourceProvider(String
sourceTableFormat) {
Review Comment:
There's a similar method in utilities, can we move this to xtable-core and
import it from there? This could just be a static method taking in
sourceTableFormat and a configuration object.
##########
xtable-service/src/main/java/org/apache/xtable/service/models/ConvertTableResponse.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.xtable.service.models;
+
+import java.util.List;
+
+import lombok.Builder;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+@Builder
+public class ConvertTableResponse {
+ @JsonProperty("conversions")
+ private List<RestTargetTable> conversions;
Review Comment:
`convertedTables` maybe a better name?
##########
xtable-service/src/main/java/org/apache/xtable/service/ConversionResource.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.xtable.service;
+
+import org.apache.xtable.service.models.ConvertTableRequest;
+import org.apache.xtable.service.models.ConvertTableResponse;
+
+import io.smallrye.common.annotation.Blocking;
+import jakarta.inject.Inject;
+import jakarta.ws.rs.Consumes;
+import jakarta.ws.rs.POST;
+import jakarta.ws.rs.Path;
+import jakarta.ws.rs.Produces;
+import jakarta.ws.rs.core.MediaType;
+
+@Path("/v1/conversion")
+@Produces(MediaType.APPLICATION_JSON)
+@Consumes(MediaType.APPLICATION_JSON)
+public class ConversionResource {
+
+ @Inject ConversionService conversionService;
+
+ @POST
+ @Path("/table")
+ @Blocking
+ public ConvertTableResponse convertTable(ConvertTableRequest req) {
Review Comment:
Rename to request as the variable? The full name variable is recommended in
java for public methods.
##########
xtable-service/src/main/java/org/apache/xtable/service/ConversionControllerFactory.java:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.xtable.service;
+
+import org.apache.hadoop.conf.Configuration;
+
+import org.apache.xtable.conversion.ConversionController;
+
+import jakarta.enterprise.context.ApplicationScoped;
+
+@ApplicationScoped
Review Comment:
java docs?
##########
xtable-service/src/main/java/org/apache/xtable/service/ConversionService.java:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.xtable.service;
+
+import static org.apache.xtable.model.storage.TableFormat.DELTA;
+import static org.apache.xtable.model.storage.TableFormat.HUDI;
+import static org.apache.xtable.model.storage.TableFormat.ICEBERG;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+
+import org.apache.iceberg.Snapshot;
+
+import org.apache.xtable.conversion.ConversionConfig;
+import org.apache.xtable.conversion.ConversionController;
+import org.apache.xtable.conversion.ConversionSourceProvider;
+import org.apache.xtable.conversion.SourceTable;
+import org.apache.xtable.conversion.TargetTable;
+import org.apache.xtable.delta.DeltaConversionSourceProvider;
+import org.apache.xtable.hudi.HudiConversionSourceProvider;
+import org.apache.xtable.iceberg.IcebergConversionSourceProvider;
+import org.apache.xtable.service.models.ConvertTableRequest;
+import org.apache.xtable.service.models.ConvertTableResponse;
+import org.apache.xtable.service.models.RestTargetTable;
+import org.apache.xtable.service.spark.SparkHolder;
+import org.apache.xtable.service.utils.DeltaMetadataUtil;
+import org.apache.xtable.service.utils.HudiMedataUtil;
+import org.apache.xtable.service.utils.IcebergMetadataUtil;
+
+import jakarta.enterprise.context.ApplicationScoped;
+import jakarta.inject.Inject;
+
+@ApplicationScoped
Review Comment:
java docs for all public classes in OSS is recommended.
##########
xtable-service/src/main/java/org/apache/xtable/service/models/ConvertTableRequest.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.xtable.service.models;
+
+import java.util.List;
+import java.util.Map;
+
+import lombok.Builder;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+@Builder
+public class ConvertTableRequest {
+ @JsonProperty("source-format")
+ private String sourceFormat;
+
+ @JsonProperty("source-table-name")
+ private String sourceTableName;
+
+ @JsonProperty("source-table-path")
+ private String sourceTablePath;
+
+ @JsonProperty("target-formats")
+ private List<String> targetFormats;
+
+ @JsonProperty("configurations")
Review Comment:
What are configurations? The term is overloaded let's use a different name?
##########
xtable-service/src/main/java/org/apache/xtable/service/ConversionService.java:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.xtable.service;
+
+import static org.apache.xtable.model.storage.TableFormat.DELTA;
+import static org.apache.xtable.model.storage.TableFormat.HUDI;
+import static org.apache.xtable.model.storage.TableFormat.ICEBERG;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+
+import org.apache.iceberg.Snapshot;
+
+import org.apache.xtable.conversion.ConversionConfig;
+import org.apache.xtable.conversion.ConversionController;
+import org.apache.xtable.conversion.ConversionSourceProvider;
+import org.apache.xtable.conversion.SourceTable;
+import org.apache.xtable.conversion.TargetTable;
+import org.apache.xtable.delta.DeltaConversionSourceProvider;
+import org.apache.xtable.hudi.HudiConversionSourceProvider;
+import org.apache.xtable.iceberg.IcebergConversionSourceProvider;
+import org.apache.xtable.service.models.ConvertTableRequest;
+import org.apache.xtable.service.models.ConvertTableResponse;
+import org.apache.xtable.service.models.RestTargetTable;
+import org.apache.xtable.service.spark.SparkHolder;
+import org.apache.xtable.service.utils.DeltaMetadataUtil;
+import org.apache.xtable.service.utils.HudiMedataUtil;
+import org.apache.xtable.service.utils.IcebergMetadataUtil;
+
+import jakarta.enterprise.context.ApplicationScoped;
+import jakarta.inject.Inject;
+
+@ApplicationScoped
+public class ConversionService {
+ private final SparkHolder sparkHolder;
+ private final ConversionControllerFactory controllerFactory;
+ private final IcebergMetadataUtil icebergUtil;
+ private final HudiMedataUtil hudiUtil;
+ private final DeltaMetadataUtil deltaUtil;
+
+ @Inject
+ public ConversionService(
+ SparkHolder sparkHolder,
+ ConversionControllerFactory controllerFactory,
+ IcebergMetadataUtil icebergUtil,
+ HudiMedataUtil hudiUtil,
+ DeltaMetadataUtil deltaUtil) {
+ this.sparkHolder = sparkHolder;
+ this.controllerFactory = controllerFactory;
+ this.icebergUtil = icebergUtil;
+ this.hudiUtil = hudiUtil;
+ this.deltaUtil = deltaUtil;
+ }
+
+ public ConvertTableResponse convertTable(ConvertTableRequest request) {
+ Configuration conf = sparkHolder.jsc().hadoopConfiguration();
+ ConversionController conversionController = controllerFactory.create(conf);
+
+ SourceTable sourceTable =
+ SourceTable.builder()
+ .name(request.getSourceTableName())
+ .basePath(request.getSourceTablePath())
+ .formatName(request.getSourceFormat())
+ .build();
+
+ List<TargetTable> targetTables = new ArrayList<>();
+ for (String targetFormat : request.getTargetFormats()) {
+ TargetTable targetTable =
+ TargetTable.builder()
+ .name(request.getSourceTableName())
+ .basePath(request.getSourceTablePath())
+ .formatName(targetFormat)
+ .build();
+ targetTables.add(targetTable);
+ }
+ ConversionConfig conversionConfig =
+
ConversionConfig.builder().sourceTable(sourceTable).targetTables(targetTables).build();
+ ConversionSourceProvider<?> conversionSourceProvider =
+ getConversionSourceProvider(request.getSourceFormat());
+ conversionController.sync(conversionConfig, conversionSourceProvider);
+
+ List<RestTargetTable> restTargetTables = new ArrayList<>();
+ for (String targetFormat : request.getTargetFormats()) {
+ if (targetFormat.equals("ICEBERG")) {
Review Comment:
Use TableFormat.ICEBERG constant available in the repo.
##########
xtable-service/src/main/java/org/apache/xtable/service/ConversionControllerFactory.java:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.xtable.service;
+
+import org.apache.hadoop.conf.Configuration;
+
+import org.apache.xtable.conversion.ConversionController;
+
+import jakarta.enterprise.context.ApplicationScoped;
+
+@ApplicationScoped
+public class ConversionControllerFactory {
Review Comment:
This class is just returning `ConversionController`? Why can't we directly
call the constructor here? A factory is useful if we have multiple
implementations of conversion controller implementing a common interface. This
will be an overkill IMO.
##########
xtable-service/src/main/java/org/apache/xtable/service/utils/HudiMedataUtil.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.xtable.service.utils;
+
+import jakarta.enterprise.context.ApplicationScoped;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+
+import java.io.IOException;
+
+@ApplicationScoped
+public class HudiMedataUtil {
Review Comment:
unit tests?
##########
xtable-service/src/test/java/org/apache/xtable/service/TestConversionResource.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.xtable.service;
+
+import static org.junit.jupiter.api.Assertions.*;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.InjectMocks;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+
+import org.apache.xtable.service.models.ConvertTableRequest;
+import org.apache.xtable.service.models.ConvertTableResponse;
+import org.apache.xtable.service.models.RestTargetTable;
+
+@ExtendWith(MockitoExtension.class)
+class TestConversionResource {
Review Comment:
Can we add an IT which writes hudi, iceberg and delta tables, bootstrap the
server and run the conversion sync? See IT tests in xtable-core.
##########
xtable-service/src/main/java/org/apache/xtable/service/ConversionService.java:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.xtable.service;
+
+import static org.apache.xtable.model.storage.TableFormat.DELTA;
+import static org.apache.xtable.model.storage.TableFormat.HUDI;
+import static org.apache.xtable.model.storage.TableFormat.ICEBERG;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+
+import org.apache.iceberg.Snapshot;
+
+import org.apache.xtable.conversion.ConversionConfig;
+import org.apache.xtable.conversion.ConversionController;
+import org.apache.xtable.conversion.ConversionSourceProvider;
+import org.apache.xtable.conversion.SourceTable;
+import org.apache.xtable.conversion.TargetTable;
+import org.apache.xtable.delta.DeltaConversionSourceProvider;
+import org.apache.xtable.hudi.HudiConversionSourceProvider;
+import org.apache.xtable.iceberg.IcebergConversionSourceProvider;
+import org.apache.xtable.service.models.ConvertTableRequest;
+import org.apache.xtable.service.models.ConvertTableResponse;
+import org.apache.xtable.service.models.RestTargetTable;
+import org.apache.xtable.service.spark.SparkHolder;
+import org.apache.xtable.service.utils.DeltaMetadataUtil;
+import org.apache.xtable.service.utils.HudiMedataUtil;
+import org.apache.xtable.service.utils.IcebergMetadataUtil;
+
+import jakarta.enterprise.context.ApplicationScoped;
+import jakarta.inject.Inject;
+
+@ApplicationScoped
+public class ConversionService {
+ private final SparkHolder sparkHolder;
+ private final ConversionControllerFactory controllerFactory;
+ private final IcebergMetadataUtil icebergUtil;
+ private final HudiMedataUtil hudiUtil;
+ private final DeltaMetadataUtil deltaUtil;
Review Comment:
Are these classes having static methods? If yes, ingesting them in the
constructor can be avoided.
##########
xtable-service/src/main/java/org/apache/xtable/service/spark/SparkHolder.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.xtable.service.spark;
+
+import java.io.InputStream;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.serializer.KryoSerializer;
+import org.apache.spark.sql.SparkSession;
+
+import io.quarkus.runtime.ShutdownEvent;
+import jakarta.enterprise.context.ApplicationScoped;
+import jakarta.enterprise.event.Observes;
+
+@ApplicationScoped
+public class SparkHolder {
+
+ private volatile SparkSession spark;
+ private volatile JavaSparkContext jsc;
+ private static final String HADOOP_DEFAULTS_PATH =
"xtable-hadoop-defaults.xml";
+
+ public SparkSession spark() {
+ if (spark == null) {
+ synchronized (this) {
+ if (spark == null) {
+ spark =
+ SparkSession.builder()
+ .config(getSparkConf())
+ .appName("xtable-conversion-service")
+ .getOrCreate();
+ jsc = JavaSparkContext.fromSparkContext(spark.sparkContext());
+ InputStream resourceStream =
+
SparkHolder.class.getClassLoader().getResourceAsStream(HADOOP_DEFAULTS_PATH);
Review Comment:
The user may be need to provide their hadoopConfiguration? So needs a
HADOOP_CONFIG_PATH similar to utilities?
##########
xtable-service/src/main/java/org/apache/xtable/service/utils/DeltaMetadataUtil.java:
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.xtable.service.utils;
+
+import jakarta.enterprise.context.ApplicationScoped;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.delta.DeltaLog;
+import org.apache.spark.sql.delta.Snapshot;
+import org.apache.spark.sql.types.StructType;
+
+@ApplicationScoped
+public class DeltaMetadataUtil {
Review Comment:
unit tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
