[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1150: [HUDI-288]: Add support for ingesting multiple kafka streams in a single DeltaStreamer deployment
vinothchandar commented on a change in pull request #1150: [HUDI-288]: Add
support for ingesting multiple kafka streams in a single DeltaStreamer
deployment
URL: https://github.com/apache/incubator-hudi/pull/1150#discussion_r379216693
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/util/DFSTablePropertiesConfiguration.java
##
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.util;
+
+import org.apache.hudi.common.model.TableConfig;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Used for parsing custom files having TableConfig objects.
+ */
+public class DFSTablePropertiesConfiguration {
Review comment:
I think your understanding is correct..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1150: [HUDI-288]: Add support for ingesting multiple kafka streams in a single DeltaStreamer deployment
vinothchandar commented on a change in pull request #1150: [HUDI-288]: Add
support for ingesting multiple kafka streams in a single DeltaStreamer
deployment
URL: https://github.com/apache/incubator-hudi/pull/1150#discussion_r379215926
##
File path:
hudi-client/src/test/java/org/apache/hudi/common/HoodieTestDataGenerator.java
##
@@ -80,16 +79,27 @@
+ "{\"name\": \"begin_lat\", \"type\": \"double\"},{\"name\":
\"begin_lon\", \"type\": \"double\"},"
+ "{\"name\": \"end_lat\", \"type\": \"double\"},{\"name\": \"end_lon\",
\"type\": \"double\"},"
+ "{\"name\":\"fare\",\"type\": \"double\"}]}";
+ public static String GROCERY_PURCHASE_SCHEMA =
"{\"type\":\"record\",\"name\":\"purchaserec\",\"fields\":["
Review comment:
That last sentence, can happen on a separate JIRA..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1150: [HUDI-288]: Add support for ingesting multiple kafka streams in a single DeltaStreamer deployment
vinothchandar commented on a change in pull request #1150: [HUDI-288]: Add
support for ingesting multiple kafka streams in a single DeltaStreamer
deployment
URL: https://github.com/apache/incubator-hudi/pull/1150#discussion_r379215764
##
File path:
hudi-client/src/test/java/org/apache/hudi/common/HoodieTestDataGenerator.java
##
@@ -80,16 +79,27 @@
+ "{\"name\": \"begin_lat\", \"type\": \"double\"},{\"name\":
\"begin_lon\", \"type\": \"double\"},"
+ "{\"name\": \"end_lat\", \"type\": \"double\"},{\"name\": \"end_lon\",
\"type\": \"double\"},"
+ "{\"name\":\"fare\",\"type\": \"double\"}]}";
+ public static String GROCERY_PURCHASE_SCHEMA =
"{\"type\":\"record\",\"name\":\"purchaserec\",\"fields\":["
Review comment:
I think what @bvaradar is trying to get at it is that the current payload is
built around the trip schema.. Overall, can we just have multiple topics with
the same trips schema? I am not sure what real benefits we get from introducing
new schema here.. if we do, I would like to spend more time structuring this
class well , since it will make this class even more unclean than what it is
today..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1150: [HUDI-288]: Add support for ingesting multiple kafka streams in a single DeltaStreamer deployment
vinothchandar commented on a change in pull request #1150: [HUDI-288]: Add
support for ingesting multiple kafka streams in a single DeltaStreamer
deployment
URL: https://github.com/apache/incubator-hudi/pull/1150#discussion_r379215764
##
File path:
hudi-client/src/test/java/org/apache/hudi/common/HoodieTestDataGenerator.java
##
@@ -80,16 +79,27 @@
+ "{\"name\": \"begin_lat\", \"type\": \"double\"},{\"name\":
\"begin_lon\", \"type\": \"double\"},"
+ "{\"name\": \"end_lat\", \"type\": \"double\"},{\"name\": \"end_lon\",
\"type\": \"double\"},"
+ "{\"name\":\"fare\",\"type\": \"double\"}]}";
+ public static String GROCERY_PURCHASE_SCHEMA =
"{\"type\":\"record\",\"name\":\"purchaserec\",\"fields\":["
Review comment:
I think what @bvaradar is trying to get at it is that the current payload is
built around the trip schema.. Overall, can we just have multiple topics with
the same trips schema? I am not sure what real benefits we get from introducing
new schema here.. if you feel strongly still, I would like to spend more time
structuring this class well , since it will make this class even more unclean
than what it is today..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1150: [HUDI-288]: Add support for ingesting multiple kafka streams in a single DeltaStreamer deployment
vinothchandar commented on a change in pull request #1150: [HUDI-288]: Add
support for ingesting multiple kafka streams in a single DeltaStreamer
deployment
URL: https://github.com/apache/incubator-hudi/pull/1150#discussion_r362275299
##
File path:
hudi-client/src/test/java/org/apache/hudi/common/HoodieTestDataGenerator.java
##
@@ -80,16 +79,27 @@
+ "{\"name\": \"begin_lat\", \"type\": \"double\"},{\"name\":
\"begin_lon\", \"type\": \"double\"},"
+ "{\"name\": \"end_lat\", \"type\": \"double\"},{\"name\": \"end_lon\",
\"type\": \"double\"},"
+ "{\"name\":\"fare\",\"type\": \"double\"}]}";
+ public static String GROCERY_PURCHASE_SCHEMA =
"{\"type\":\"record\",\"name\":\"purchaserec\",\"fields\":["
Review comment:
Could be better to re-use the same schema? if you really want two different
schema, then its time to modularize this class better?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1150: [HUDI-288]: Add support for ingesting multiple kafka streams in a single DeltaStreamer deployment
vinothchandar commented on a change in pull request #1150: [HUDI-288]: Add support for ingesting multiple kafka streams in a single DeltaStreamer deployment URL: https://github.com/apache/incubator-hudi/pull/1150#discussion_r362275428 ## File path: hudi-common/src/main/java/org/apache/hudi/common/model/TableConfig.java ## @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hudi.common.model; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; +import com.fasterxml.jackson.annotation.JsonProperty; + +import java.util.Objects; + +/* +Represents object with all the topic level overrides for multi table delta streamer execution + */ +@JsonIgnoreProperties(ignoreUnknown = true) Review comment: any reason this is in the `hudi-common` package? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
