Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
wombatu-kun commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-2028558096 > @wombatu-kun Thanks for the contribution! Are you interested in more code restructuring and core abstraction work? Feel free to reach out to me on Hudi Slack. @yihua Yes, i'm interested in code refactoring, restructuring and core abstraction work, that is why i'm here! Right now i'm not really familiar with all core abstractions and functionality of hudi in depth, but i'm trying hard to figure this out as soon as possible. So definitely I'll reach out to you on Slack in nearest future. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
yihua merged PR #10866: URL: https://github.com/apache/hudi/pull/10866 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
yihua commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-2028548116 @wombatu-kun Thanks for the contribution! Are you interested in more code restructuring and core abstraction work? Feel free to reach out to me on Hudi Slack. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
hudi-bot commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-2028045511 ## CI report: * c5f68a42d67d749e8b8218d937d808901ffefcb6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23058) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
hudi-bot commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-2028025772 ## CI report: * cc2ead88aba164d6cb0971d8a77fc668acc355b8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23005) * c5f68a42d67d749e8b8218d937d808901ffefcb6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23058) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
hudi-bot commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-2028023649 ## CI report: * cc2ead88aba164d6cb0971d8a77fc668acc355b8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23005) * c5f68a42d67d749e8b8218d937d808901ffefcb6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
wombatu-kun commented on code in PR #10866:
URL: https://github.com/apache/hudi/pull/10866#discussion_r1545284823
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/config/SourceTestConfig.java:
##
@@ -21,29 +21,36 @@
import org.apache.hudi.common.config.ConfigProperty;
+import static
org.apache.hudi.common.util.ConfigUtils.DELTA_STREAMER_CONFIG_PREFIX;
+import static org.apache.hudi.common.util.ConfigUtils.STREAMER_CONFIG_PREFIX;
+
/**
* Configurations for Test Data Sources.
*/
public class SourceTestConfig {
public static final ConfigProperty NUM_SOURCE_PARTITIONS_PROP =
ConfigProperty
- .key("hoodie.deltastreamer.source.test.num_partitions")
+ .key(STREAMER_CONFIG_PREFIX + "source.test.num_partitions")
Review Comment:
Didn't know that, thanks for comment. Found such usages in
AbstractBaseTestSource and DistributedTestDataSource, fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
yihua commented on code in PR #10866:
URL: https://github.com/apache/hudi/pull/10866#discussion_r1545129986
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/config/SourceTestConfig.java:
##
@@ -21,29 +21,36 @@
import org.apache.hudi.common.config.ConfigProperty;
+import static
org.apache.hudi.common.util.ConfigUtils.DELTA_STREAMER_CONFIG_PREFIX;
+import static org.apache.hudi.common.util.ConfigUtils.STREAMER_CONFIG_PREFIX;
+
/**
* Configurations for Test Data Sources.
*/
public class SourceTestConfig {
public static final ConfigProperty NUM_SOURCE_PARTITIONS_PROP =
ConfigProperty
- .key("hoodie.deltastreamer.source.test.num_partitions")
+ .key(STREAMER_CONFIG_PREFIX + "source.test.num_partitions")
Review Comment:
Note that `.withAlternatives` is only considered by `ConfigUtils` APIs, not
by `props.getInteger` with `.key()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
yihua commented on code in PR #10866:
URL: https://github.com/apache/hudi/pull/10866#discussion_r1545123027
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/config/SourceTestConfig.java:
##
@@ -21,29 +21,36 @@
import org.apache.hudi.common.config.ConfigProperty;
+import static
org.apache.hudi.common.util.ConfigUtils.DELTA_STREAMER_CONFIG_PREFIX;
+import static org.apache.hudi.common.util.ConfigUtils.STREAMER_CONFIG_PREFIX;
+
/**
* Configurations for Test Data Sources.
*/
public class SourceTestConfig {
public static final ConfigProperty NUM_SOURCE_PARTITIONS_PROP =
ConfigProperty
- .key("hoodie.deltastreamer.source.test.num_partitions")
+ .key(STREAMER_CONFIG_PREFIX + "source.test.num_partitions")
Review Comment:
Could you also check the usages of all the `ConfigProperty` definitions and
replace all config fetching using `.key()`, e.g.,
`props.getInteger(SourceTestConfig.MAX_UNIQUE_RECORDS_PROP.key(),
SourceTestConfig.MAX_UNIQUE_RECORDS_PROP.defaultValue());`, with `ConfigUtils`
APIs, e.g., `getStringWithAltKeys`, `getLongWithAltKeys`, etc.?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
wombatu-kun commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-2027189611 @yihua I made fixes by all your comments. Review please and merge if everything is ok. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
hudi-bot commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-2016496479 ## CI report: * cc2ead88aba164d6cb0971d8a77fc668acc355b8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23005) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
hudi-bot commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-2016480860 ## CI report: * 497a8c65ded43253b375c7504779357ef02bf560 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22966) * cc2ead88aba164d6cb0971d8a77fc668acc355b8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23005) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
hudi-bot commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-2016478967 ## CI report: * 497a8c65ded43253b375c7504779357ef02bf560 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22966) * cc2ead88aba164d6cb0971d8a77fc668acc355b8 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
hudi-bot commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-2009415118 ## CI report: * 497a8c65ded43253b375c7504779357ef02bf560 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22966) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
hudi-bot commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-2009203434 ## CI report: * 113f4e5717cdf97aed4f91cecd965c4cec02dec0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22948) * 497a8c65ded43253b375c7504779357ef02bf560 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
hudi-bot commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-2009215967 ## CI report: * 113f4e5717cdf97aed4f91cecd965c4cec02dec0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22948) * 497a8c65ded43253b375c7504779357ef02bf560 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22966) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
wombatu-kun commented on code in PR #10866:
URL: https://github.com/apache/hudi/pull/10866#discussion_r1529606423
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java:
##
@@ -2418,15 +2418,15 @@ public void testSqlSourceSource() throws Exception {
@Disabled
@Test
public void testJdbcSourceIncrementalFetchInContinuousMode() {
-try (Connection connection =
DriverManager.getConnection("jdbc:h2:mem:test_mem", "test", "jdbc")) {
+try (Connection connection =
DriverManager.getConnection("jdbc:h2:mem:test_mem", "sa", "")) {
Review Comment:
Yes, it's from another fix. It's really better to do it in separate PR
(https://github.com/apache/hudi/pull/10889). For now - reverted JDBC user name
and password changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
hudi-bot commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-2005765864 ## CI report: * 113f4e5717cdf97aed4f91cecd965c4cec02dec0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22948) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
hudi-bot commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-2005722576 ## CI report: * 538d59f11cfca29af8550391f0ce7a491279c0a9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22906) * 113f4e5717cdf97aed4f91cecd965c4cec02dec0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22948) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
hudi-bot commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-2005717768 ## CI report: * 538d59f11cfca29af8550391f0ce7a491279c0a9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22906) * 113f4e5717cdf97aed4f91cecd965c4cec02dec0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22948) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
wombatu-kun commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-2005715865 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
hudi-bot commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-2005678158 ## CI report: * 538d59f11cfca29af8550391f0ce7a491279c0a9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22906) * 113f4e5717cdf97aed4f91cecd965c4cec02dec0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22948) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
hudi-bot commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-2005673058 ## CI report: * 538d59f11cfca29af8550391f0ce7a491279c0a9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22906) * 113f4e5717cdf97aed4f91cecd965c4cec02dec0 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
wombatu-kun commented on code in PR #10866:
URL: https://github.com/apache/hudi/pull/10866#discussion_r1529616940
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamerTestBase.java:
##
@@ -266,9 +266,9 @@ public void setupTest() {
protected static void
populateInvalidTableConfigFilePathProps(TypedProperties props, String
dfsBasePath) {
props.setProperty("hoodie.datasource.write.keygenerator.class",
TestHoodieDeltaStreamer.TestGenerator.class.getName());
-
props.setProperty("hoodie.deltastreamer.keygen.timebased.output.dateformat",
"MMdd");
-props.setProperty("hoodie.deltastreamer.ingestion.tablesToBeIngested",
"uber_db.dummy_table_uber");
-
props.setProperty("hoodie.deltastreamer.ingestion.uber_db.dummy_table_uber.configFile",
dfsBasePath + "/config/invalid_uber_config.properties");
+props.setProperty("hoodie.streamer.keygen.timebased.output.dateformat",
"MMdd");
Review Comment:
Thank you, fixed it
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamerTestBase.java:
##
@@ -279,10 +279,10 @@ protected static void
populateAllCommonProps(TypedProperties props, String dfsBa
protected static void populateCommonProps(TypedProperties props, String
dfsBasePath) {
props.setProperty("hoodie.datasource.write.keygenerator.class",
TestHoodieDeltaStreamer.TestGenerator.class.getName());
-
props.setProperty("hoodie.deltastreamer.keygen.timebased.output.dateformat",
"MMdd");
-props.setProperty("hoodie.deltastreamer.ingestion.tablesToBeIngested",
"short_trip_db.dummy_table_short_trip,uber_db.dummy_table_uber");
-
props.setProperty("hoodie.deltastreamer.ingestion.uber_db.dummy_table_uber.configFile",
dfsBasePath + "/config/uber_config.properties");
-
props.setProperty("hoodie.deltastreamer.ingestion.short_trip_db.dummy_table_short_trip.configFile",
dfsBasePath + "/config/short_trip_uber_config.properties");
+props.setProperty("hoodie.streamer.keygen.timebased.output.dateformat",
"MMdd");
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
wombatu-kun commented on code in PR #10866:
URL: https://github.com/apache/hudi/pull/10866#discussion_r1529616434
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/config/SourceTestConfig.java:
##
@@ -27,22 +27,22 @@
public class SourceTestConfig {
public static final ConfigProperty NUM_SOURCE_PARTITIONS_PROP =
ConfigProperty
- .key("hoodie.deltastreamer.source.test.num_partitions")
+ .key("hoodie.streamer.source.test.num_partitions")
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
wombatu-kun commented on code in PR #10866:
URL: https://github.com/apache/hudi/pull/10866#discussion_r1529606592
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestJdbcSource.java:
##
@@ -73,12 +73,12 @@ public static void beforeAll() throws Exception {
@BeforeEach
public void setup() throws Exception {
super.setup();
-PROPS.setProperty("hoodie.deltastreamer.jdbc.url", "jdbc:h2:mem:test_mem");
-PROPS.setProperty("hoodie.deltastreamer.jdbc.driver.class",
"org.h2.Driver");
-PROPS.setProperty("hoodie.deltastreamer.jdbc.user", "test");
-PROPS.setProperty("hoodie.deltastreamer.jdbc.password", "jdbc");
-PROPS.setProperty("hoodie.deltastreamer.jdbc.table.name", "triprec");
-connection = DriverManager.getConnection("jdbc:h2:mem:test_mem", "test",
"jdbc");
+PROPS.setProperty("hoodie.streamer.jdbc.url", "jdbc:h2:mem:test_mem");
+PROPS.setProperty("hoodie.streamer.jdbc.driver.class", "org.h2.Driver");
+PROPS.setProperty("hoodie.streamer.jdbc.user", "sa");
+PROPS.setProperty("hoodie.streamer.jdbc.password", "");
+PROPS.setProperty("hoodie.streamer.jdbc.table.name", "triprec");
+connection = DriverManager.getConnection("jdbc:h2:mem:test_mem", "sa", "");
Review Comment:
reverted, will do it in separate PR.
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestJdbcSource.java:
##
@@ -438,7 +438,7 @@ public void testSourceWithStorageLevel() {
private void writeSecretToFs() throws IOException {
FileSystem fs = FileSystem.get(new Configuration());
FSDataOutputStream outputStream = fs.create(new
Path("file:///tmp/hudi/config/secret"));
-outputStream.writeBytes("jdbc");
+outputStream.writeBytes("");
Review Comment:
reverted, will do it in separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
wombatu-kun commented on code in PR #10866:
URL: https://github.com/apache/hudi/pull/10866#discussion_r1529606423
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java:
##
@@ -2418,15 +2418,15 @@ public void testSqlSourceSource() throws Exception {
@Disabled
@Test
public void testJdbcSourceIncrementalFetchInContinuousMode() {
-try (Connection connection =
DriverManager.getConnection("jdbc:h2:mem:test_mem", "test", "jdbc")) {
+try (Connection connection =
DriverManager.getConnection("jdbc:h2:mem:test_mem", "sa", "")) {
Review Comment:
Yes, it's from another fix. It's really better to do it in separate PR
later. For now - reverted JDBC user name and password changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
yihua commented on code in PR #10866:
URL: https://github.com/apache/hudi/pull/10866#discussion_r1529100966
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java:
##
@@ -2418,15 +2418,15 @@ public void testSqlSourceSource() throws Exception {
@Disabled
@Test
public void testJdbcSourceIncrementalFetchInContinuousMode() {
-try (Connection connection =
DriverManager.getConnection("jdbc:h2:mem:test_mem", "test", "jdbc")) {
+try (Connection connection =
DriverManager.getConnection("jdbc:h2:mem:test_mem", "sa", "")) {
Review Comment:
Looks like this is from another fix. Could the JDBC user and password be
put into static variables to avoid any misconfigs in tests in the future?
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestJdbcSource.java:
##
@@ -438,7 +438,7 @@ public void testSourceWithStorageLevel() {
private void writeSecretToFs() throws IOException {
FileSystem fs = FileSystem.get(new Configuration());
FSDataOutputStream outputStream = fs.create(new
Path("file:///tmp/hudi/config/secret"));
-outputStream.writeBytes("jdbc");
+outputStream.writeBytes("");
Review Comment:
Similar here for the password. And should we add the password back for
testing the secret (this refactoring of static variables and password should be
in a separate PR)?
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestJdbcSource.java:
##
@@ -73,12 +73,12 @@ public static void beforeAll() throws Exception {
@BeforeEach
public void setup() throws Exception {
super.setup();
-PROPS.setProperty("hoodie.deltastreamer.jdbc.url", "jdbc:h2:mem:test_mem");
-PROPS.setProperty("hoodie.deltastreamer.jdbc.driver.class",
"org.h2.Driver");
-PROPS.setProperty("hoodie.deltastreamer.jdbc.user", "test");
-PROPS.setProperty("hoodie.deltastreamer.jdbc.password", "jdbc");
-PROPS.setProperty("hoodie.deltastreamer.jdbc.table.name", "triprec");
-connection = DriverManager.getConnection("jdbc:h2:mem:test_mem", "test",
"jdbc");
+PROPS.setProperty("hoodie.streamer.jdbc.url", "jdbc:h2:mem:test_mem");
+PROPS.setProperty("hoodie.streamer.jdbc.driver.class", "org.h2.Driver");
+PROPS.setProperty("hoodie.streamer.jdbc.user", "sa");
+PROPS.setProperty("hoodie.streamer.jdbc.password", "");
+PROPS.setProperty("hoodie.streamer.jdbc.table.name", "triprec");
+connection = DriverManager.getConnection("jdbc:h2:mem:test_mem", "sa", "");
Review Comment:
Similar here for the JDBC user name and password.
##
hudi-utilities/src/test/resources/streamer-config/invalid_hive_sync_uber_config.properties:
##
@@ -18,6 +18,6 @@
include=base.properties
hoodie.datasource.write.recordkey.field=_row_key
hoodie.datasource.write.partitionpath.field=created_at
-hoodie.deltastreamer.source.kafka.topic=test_topic
-hoodie.deltastreamer.keygen.timebased.timestamp.type=UNIX_TIMESTAMP
-hoodie.deltastreamer.keygen.timebased.input.dateformat=-MM-dd
\ No newline at end of file
+hoodie.streamer.source.kafka.topic=test_topic
+hoodie.streamer.keygen.timebased.timestamp.type=UNIX_TIMESTAMP
Review Comment:
reminder for fixing keygen config.
##
hudi-utilities/src/test/resources/streamer-config/uber_config.properties:
##
@@ -18,10 +18,10 @@
include=base.properties
hoodie.datasource.write.recordkey.field=_row_key
hoodie.datasource.write.partitionpath.field=created_at
-hoodie.deltastreamer.source.kafka.topic=topic1
-hoodie.deltastreamer.keygen.timebased.timestamp.type=UNIX_TIMESTAMP
-hoodie.deltastreamer.keygen.timebased.input.dateformat=-MM-dd HH:mm:ss.S
+hoodie.streamer.source.kafka.topic=topic1
+hoodie.streamer.keygen.timebased.timestamp.type=UNIX_TIMESTAMP
Review Comment:
reminder for fixing keygen config.
##
hudi-utilities/src/test/resources/streamer-config/short_trip_uber_config.properties:
##
@@ -18,11 +18,11 @@
include=base.properties
hoodie.datasource.write.recordkey.field=_row_key
hoodie.datasource.write.partitionpath.field=created_at
-hoodie.deltastreamer.source.kafka.topic=topic2
-hoodie.deltastreamer.keygen.timebased.timestamp.type=UNIX_TIMESTAMP
-hoodie.deltastreamer.keygen.timebased.input.dateformat=-MM-dd HH:mm:ss.S
+hoodie.streamer.source.kafka.topic=topic2
+hoodie.streamer.keygen.timebased.timestamp.type=UNIX_TIMESTAMP
Review Comment:
reminder for fixing keygen config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
yihua commented on code in PR #10866:
URL: https://github.com/apache/hudi/pull/10866#discussion_r1529083756
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/config/SourceTestConfig.java:
##
@@ -27,22 +27,22 @@
public class SourceTestConfig {
public static final ConfigProperty NUM_SOURCE_PARTITIONS_PROP =
ConfigProperty
- .key("hoodie.deltastreamer.source.test.num_partitions")
+ .key("hoodie.streamer.source.test.num_partitions")
Review Comment:
Let's add `.withAlternatives()` to preserve the old config naming to be
backward compatible and use `STREAMER_CONFIG_PREFIX` and
`DELTA_STREAMER_CONFIG_PREFIX`. Similar for other configs in this class,
`SourceTestConfig`. See `HoodieStreamerConfig.CHECKPOINT_PROVIDER_PATH` for
reference:
```
public static final ConfigProperty CHECKPOINT_PROVIDER_PATH =
ConfigProperty
.key(STREAMER_CONFIG_PREFIX + "checkpoint.provider.path")
.noDefaultValue()
.withAlternatives(DELTA_STREAMER_CONFIG_PREFIX +
"checkpoint.provider.path")
.markAdvanced()
.withDocumentation("The path for providing the checkpoints.");
```
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamerTestBase.java:
##
@@ -266,9 +266,9 @@ public void setupTest() {
protected static void
populateInvalidTableConfigFilePathProps(TypedProperties props, String
dfsBasePath) {
props.setProperty("hoodie.datasource.write.keygenerator.class",
TestHoodieDeltaStreamer.TestGenerator.class.getName());
-
props.setProperty("hoodie.deltastreamer.keygen.timebased.output.dateformat",
"MMdd");
-props.setProperty("hoodie.deltastreamer.ingestion.tablesToBeIngested",
"uber_db.dummy_table_uber");
-
props.setProperty("hoodie.deltastreamer.ingestion.uber_db.dummy_table_uber.configFile",
dfsBasePath + "/config/invalid_uber_config.properties");
+props.setProperty("hoodie.streamer.keygen.timebased.output.dateformat",
"MMdd");
Review Comment:
For key generator-related configs, the config prefix is renamed from
`hoodie.deltastreamer.keygen.timebased.` to `hoodie.keygen.timebased.` (see
class `TimestampKeyGeneratorConfig`).
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamerTestBase.java:
##
@@ -279,10 +279,10 @@ protected static void
populateAllCommonProps(TypedProperties props, String dfsBa
protected static void populateCommonProps(TypedProperties props, String
dfsBasePath) {
props.setProperty("hoodie.datasource.write.keygenerator.class",
TestHoodieDeltaStreamer.TestGenerator.class.getName());
-
props.setProperty("hoodie.deltastreamer.keygen.timebased.output.dateformat",
"MMdd");
-props.setProperty("hoodie.deltastreamer.ingestion.tablesToBeIngested",
"short_trip_db.dummy_table_short_trip,uber_db.dummy_table_uber");
-
props.setProperty("hoodie.deltastreamer.ingestion.uber_db.dummy_table_uber.configFile",
dfsBasePath + "/config/uber_config.properties");
-
props.setProperty("hoodie.deltastreamer.ingestion.short_trip_db.dummy_table_short_trip.configFile",
dfsBasePath + "/config/short_trip_uber_config.properties");
+props.setProperty("hoodie.streamer.keygen.timebased.output.dateformat",
"MMdd");
Review Comment:
Similar here and other places for key generator configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
hudi-bot commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-1997375554 ## CI report: * 538d59f11cfca29af8550391f0ce7a491279c0a9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22906) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
hudi-bot commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-1997167795 ## CI report: * 538d59f11cfca29af8550391f0ce7a491279c0a9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22906) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7187] Fix integ test props to honor new streamer properties [hudi]
hudi-bot commented on PR #10866: URL: https://github.com/apache/hudi/pull/10866#issuecomment-1997155096 ## CI report: * 538d59f11cfca29af8550391f0ce7a491279c0a9 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
