[jira] [Commented] (HUDI-491) Problems encountered in building hudi

[Zijie Lu (Jira)]

[ 
https://issues.apache.org/jira/browse/HUDI-491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006645#comment-17006645
 ] 

Zijie Lu commented on HUDI-491:
---

[~Pratyaksh] I am on master branch. And I built with the following command: mvn 
clean package -DskipTests -DskipITs.

> Problems encountered in building hudi
> -
>
> Key: HUDI-491
> URL: https://issues.apache.org/jira/browse/HUDI-491
> Project: Apache Hudi (incubating)
>  Issue Type: Bug
>Reporter: Zijie Lu
>Priority: Major
>
> I don't why I can't build hudi and the error msg is as followed:
> incubator-hudi/hudi-cli/src/main/java/org/apache/hudi/cli/HoodieHistoryFileNameProvider.java:[30,2]
>  cannot find symbol
> [ERROR] symbol: class Order
> [ERROR] 
> /Users/code/incubator-hudi/hudi-cli/src/main/java/org/apache/hudi/cli/commands/ArchivedCommitsCommand.java:[52,2]
>  cannot find symbol
> [ERROR] symbol: class Component
> [ERROR] 
> /Users/code/incubator-hudi/hudi-cli/src/main/java/org/apache/hudi/cli/commands/CommitsCommand.java:[51,2]
>  cannot find symbol
> [ERROR] symbol: class Component
> [ERROR] 
> /Users/code/incubator-hudi/hudi-cli/src/main/java/org/apache/hudi/cli/commands/HoodieSyncCommand.java:[40,2]
>  cannot find symbol
> [ERROR] symbol: class Component
> [ERROR] 
> /Users/code/incubator-hudi/hudi-cli/src/main/java/org/apache/hudi/cli/commands/SavepointsCommand.java:[49,2]
>  cannot find symbol
> [ERROR] symbol: class Component
>  
> It seems that it is caused by missing spring context.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HUDI-484) NPE in HiveIncrementalPuller

[lamber-ken (Jira)]

[ 
https://issues.apache.org/jira/browse/HUDI-484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006643#comment-17006643
 ] 

lamber-ken commented on HUDI-484:
-

hi, [~Pratyaksh] Thanks very much for raising this issue. (y)(y)(y)

> NPE in HiveIncrementalPuller
> 
>
> Key: HUDI-484
> URL: https://issues.apache.org/jira/browse/HUDI-484
> Project: Apache Hudi (incubating)
>  Issue Type: Bug
>  Components: Incremental Pull
>Reporter: Pratyaksh Sharma
>Assignee: lamber-ken
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.1
>
> Attachments: Screenshot 2019-12-30 at 4.43.51 PM.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When we try to use HiveIncrementalPuller class to incrementally pull changes 
> from hive, it throws NPE as it is unable to find IncrementalPull.sqltemplate 
> in the bundled jar. 
> Screenshot attached which shows the exception. 
> The jar contains the template. 
> Steps to reproduce - 
>  # copy hive-jdbc-2.3.1.jar, log4j-1.2.17.jar to docker/demo/config folder
>  # run cd docker && ./setup_demo.sh
>  # cat docker/demo/data/batch_1.json | kafkacat -b kafkabroker -t stock_ticks 
> -P
>  #  {{docker exec -it adhoc-2 /bin/bash}}
>  #  {{spark-submit --class 
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer 
> $HUDI_UTILITIES_BUNDLE --storage-type COPY_ON_WRITE --source-class 
> org.apache.hudi.utilities.sources.JsonKafkaSource --source-ordering-field ts 
> --target-base-path /user/hive/warehouse/stock_ticks_cow --target-table 
> stock_ticks_cow --props /var/demo/config/kafka-source.properties 
> --schemaprovider-class 
> org.apache.hudi.utilities.schema.FilebasedSchemaProvider}}
>  #  {{/var/hoodie/ws/hudi-hive/run_sync_tool.sh --jdbc-url 
> jdbc:hive2://hiveserver:1 --user hive --pass hive --partitioned-by dt 
> --base-path /user/hive/warehouse/stock_ticks_cow --database default --table 
> stock_ticks_cow}}
>  # java -cp 
> /var/hoodie/ws/docker/demo/config/hive-jdbc-2.3.1.jar:/var/hoodie/ws/docker/demo/config/log4j-1.2.17.jar:$HUDI_UTILITIES_BUNDLE
>  org.apache.hudi.utilities.HiveIncrementalPuller --hiveUrl 
> jdbc:hive2://hiveserver:1 --hiveUser hive --hivePass hive 
> --extractSQLFile /var/hoodie/ws/docker/demo/config/incr_pull.txt --sourceDb 
> default --sourceTable stock_ticks_cow --targetDb tmp --targetTable tempTable 
> --fromCommitTime 0 --maxCommits 1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (HUDI-486) Improve documentation for using HiveIncrementalPuller

[Pratyaksh Sharma (Jira)]

[ 
https://issues.apache.org/jira/browse/HUDI-486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006641#comment-17006641
 ] 

Pratyaksh Sharma edited comment on HUDI-486 at 1/2/20 7:44 AM:
---

[~vinoth] Let me try doing that and get back to you. 


was (Author: pratyaksh):
[~vinoth] Let me try doing that and get back to you. 

> Improve documentation for using HiveIncrementalPuller
> -
>
> Key: HUDI-486
> URL: https://issues.apache.org/jira/browse/HUDI-486
> Project: Apache Hudi (incubating)
>  Issue Type: Sub-task
>  Components: Incremental Pull
>Reporter: Pratyaksh Sharma
>Assignee: Pratyaksh Sharma
>Priority: Major
>
> For using HiveIncrementalPuller, one needs to have a lot of jars in 
> classPath. These jars are not listed anywhere. As a result, one has to keep 
> on adding the jars incrementally to the classPath with every 
> NoClassDefFoundError coming up when executing. 
> We should list down the jars needed so that it becomes easy for a first-time 
> user to use the mentioned tool. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HUDI-486) Improve documentation for using HiveIncrementalPuller

[Pratyaksh Sharma (Jira)]

[ 
https://issues.apache.org/jira/browse/HUDI-486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006641#comment-17006641
 ] 

Pratyaksh Sharma commented on HUDI-486:
---

[~vinoth] Let me try doing that and get back to you. 

> Improve documentation for using HiveIncrementalPuller
> -
>
> Key: HUDI-486
> URL: https://issues.apache.org/jira/browse/HUDI-486
> Project: Apache Hudi (incubating)
>  Issue Type: Sub-task
>  Components: Incremental Pull
>Reporter: Pratyaksh Sharma
>Assignee: Pratyaksh Sharma
>Priority: Major
>
> For using HiveIncrementalPuller, one needs to have a lot of jars in 
> classPath. These jars are not listed anywhere. As a result, one has to keep 
> on adding the jars incrementally to the classPath with every 
> NoClassDefFoundError coming up when executing. 
> We should list down the jars needed so that it becomes easy for a first-time 
> user to use the mentioned tool. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HUDI-484) NPE in HiveIncrementalPuller

[Pratyaksh Sharma (Jira)]

[ 
https://issues.apache.org/jira/browse/HUDI-484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006639#comment-17006639
 ] 

Pratyaksh Sharma commented on HUDI-484:
---

[~lamber-ken] Since you already raised a PR, assigned it to you. 

> NPE in HiveIncrementalPuller
> 
>
> Key: HUDI-484
> URL: https://issues.apache.org/jira/browse/HUDI-484
> Project: Apache Hudi (incubating)
>  Issue Type: Bug
>  Components: Incremental Pull
>Reporter: Pratyaksh Sharma
>Assignee: lamber-ken
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.1
>
> Attachments: Screenshot 2019-12-30 at 4.43.51 PM.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When we try to use HiveIncrementalPuller class to incrementally pull changes 
> from hive, it throws NPE as it is unable to find IncrementalPull.sqltemplate 
> in the bundled jar. 
> Screenshot attached which shows the exception. 
> The jar contains the template. 
> Steps to reproduce - 
>  # copy hive-jdbc-2.3.1.jar, log4j-1.2.17.jar to docker/demo/config folder
>  # run cd docker && ./setup_demo.sh
>  # cat docker/demo/data/batch_1.json | kafkacat -b kafkabroker -t stock_ticks 
> -P
>  #  {{docker exec -it adhoc-2 /bin/bash}}
>  #  {{spark-submit --class 
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer 
> $HUDI_UTILITIES_BUNDLE --storage-type COPY_ON_WRITE --source-class 
> org.apache.hudi.utilities.sources.JsonKafkaSource --source-ordering-field ts 
> --target-base-path /user/hive/warehouse/stock_ticks_cow --target-table 
> stock_ticks_cow --props /var/demo/config/kafka-source.properties 
> --schemaprovider-class 
> org.apache.hudi.utilities.schema.FilebasedSchemaProvider}}
>  #  {{/var/hoodie/ws/hudi-hive/run_sync_tool.sh --jdbc-url 
> jdbc:hive2://hiveserver:1 --user hive --pass hive --partitioned-by dt 
> --base-path /user/hive/warehouse/stock_ticks_cow --database default --table 
> stock_ticks_cow}}
>  # java -cp 
> /var/hoodie/ws/docker/demo/config/hive-jdbc-2.3.1.jar:/var/hoodie/ws/docker/demo/config/log4j-1.2.17.jar:$HUDI_UTILITIES_BUNDLE
>  org.apache.hudi.utilities.HiveIncrementalPuller --hiveUrl 
> jdbc:hive2://hiveserver:1 --hiveUser hive --hivePass hive 
> --extractSQLFile /var/hoodie/ws/docker/demo/config/incr_pull.txt --sourceDb 
> default --sourceTable stock_ticks_cow --targetDb tmp --targetTable tempTable 
> --fromCommitTime 0 --maxCommits 1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HUDI-484) NPE in HiveIncrementalPuller

[Pratyaksh Sharma (Jira)]

 [ 
https://issues.apache.org/jira/browse/HUDI-484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pratyaksh Sharma reassigned HUDI-484:
-

Assignee: lamber-ken  (was: Pratyaksh Sharma)

> NPE in HiveIncrementalPuller
> 
>
> Key: HUDI-484
> URL: https://issues.apache.org/jira/browse/HUDI-484
> Project: Apache Hudi (incubating)
>  Issue Type: Bug
>  Components: Incremental Pull
>Reporter: Pratyaksh Sharma
>Assignee: lamber-ken
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.1
>
> Attachments: Screenshot 2019-12-30 at 4.43.51 PM.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When we try to use HiveIncrementalPuller class to incrementally pull changes 
> from hive, it throws NPE as it is unable to find IncrementalPull.sqltemplate 
> in the bundled jar. 
> Screenshot attached which shows the exception. 
> The jar contains the template. 
> Steps to reproduce - 
>  # copy hive-jdbc-2.3.1.jar, log4j-1.2.17.jar to docker/demo/config folder
>  # run cd docker && ./setup_demo.sh
>  # cat docker/demo/data/batch_1.json | kafkacat -b kafkabroker -t stock_ticks 
> -P
>  #  {{docker exec -it adhoc-2 /bin/bash}}
>  #  {{spark-submit --class 
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer 
> $HUDI_UTILITIES_BUNDLE --storage-type COPY_ON_WRITE --source-class 
> org.apache.hudi.utilities.sources.JsonKafkaSource --source-ordering-field ts 
> --target-base-path /user/hive/warehouse/stock_ticks_cow --target-table 
> stock_ticks_cow --props /var/demo/config/kafka-source.properties 
> --schemaprovider-class 
> org.apache.hudi.utilities.schema.FilebasedSchemaProvider}}
>  #  {{/var/hoodie/ws/hudi-hive/run_sync_tool.sh --jdbc-url 
> jdbc:hive2://hiveserver:1 --user hive --pass hive --partitioned-by dt 
> --base-path /user/hive/warehouse/stock_ticks_cow --database default --table 
> stock_ticks_cow}}
>  # java -cp 
> /var/hoodie/ws/docker/demo/config/hive-jdbc-2.3.1.jar:/var/hoodie/ws/docker/demo/config/log4j-1.2.17.jar:$HUDI_UTILITIES_BUNDLE
>  org.apache.hudi.utilities.HiveIncrementalPuller --hiveUrl 
> jdbc:hive2://hiveserver:1 --hiveUser hive --hivePass hive 
> --extractSQLFile /var/hoodie/ws/docker/demo/config/incr_pull.txt --sourceDb 
> default --sourceTable stock_ticks_cow --targetDb tmp --targetTable tempTable 
> --fromCommitTime 0 --maxCommits 1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HUDI-475) Add hudi-examples module and move example codes to it and also add some necessary codes

[Pratyaksh Sharma (Jira)]

[ 
https://issues.apache.org/jira/browse/HUDI-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006637#comment-17006637
 ] 

Pratyaksh Sharma commented on HUDI-475:
---

Big +1 for your initiative [~dengziming] . :)

> Add hudi-examples module and move example codes to it and also add some 
> necessary codes
> ---
>
> Key: HUDI-475
> URL: https://issues.apache.org/jira/browse/HUDI-475
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>Reporter: dengziming
>Priority: Major
>  Labels: examples
>
> # Hudi doesn't have an examples module and it's better to add one which will 
> be advantageous to users and developers.
>  # incubator-hudi/hudi-client/src/test/java/HoodieClientExample.java code 
> could be move to examples module
>  # incubator-hudi/hudi-spark/src/test/java/HoodieJavaApp.java and 
> HoodieJavaStreamingApp could be moved to examples module
>  # the code in [quickstart|[https://hudi.apache.org/quickstart.html]] can be 
> added to examples module
>  # other suggestions are welcomed



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HUDI-491) Problems encountered in building hudi

[Pratyaksh Sharma (Jira)]

[ 
https://issues.apache.org/jira/browse/HUDI-491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006635#comment-17006635
 ] 

Pratyaksh Sharma commented on HUDI-491:
---

[~alfredlu] how are you trying to build? Are you on master branch? Please share 
the command used to build and other environment specifics. 

> Problems encountered in building hudi
> -
>
> Key: HUDI-491
> URL: https://issues.apache.org/jira/browse/HUDI-491
> Project: Apache Hudi (incubating)
>  Issue Type: Bug
>Reporter: Zijie Lu
>Priority: Major
>
> I don't why I can't build hudi and the error msg is as followed:
> incubator-hudi/hudi-cli/src/main/java/org/apache/hudi/cli/HoodieHistoryFileNameProvider.java:[30,2]
>  cannot find symbol
> [ERROR] symbol: class Order
> [ERROR] 
> /Users/code/incubator-hudi/hudi-cli/src/main/java/org/apache/hudi/cli/commands/ArchivedCommitsCommand.java:[52,2]
>  cannot find symbol
> [ERROR] symbol: class Component
> [ERROR] 
> /Users/code/incubator-hudi/hudi-cli/src/main/java/org/apache/hudi/cli/commands/CommitsCommand.java:[51,2]
>  cannot find symbol
> [ERROR] symbol: class Component
> [ERROR] 
> /Users/code/incubator-hudi/hudi-cli/src/main/java/org/apache/hudi/cli/commands/HoodieSyncCommand.java:[40,2]
>  cannot find symbol
> [ERROR] symbol: class Component
> [ERROR] 
> /Users/code/incubator-hudi/hudi-cli/src/main/java/org/apache/hudi/cli/commands/SavepointsCommand.java:[49,2]
>  cannot find symbol
> [ERROR] symbol: class Component
>  
> It seems that it is caused by missing spring context.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-hudi] smarthi commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

smarthi commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362386992
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormat.java
 ##
 @@ -222,8 +222,8 @@ public Writer build() throws IOException, 
InterruptedException {
   // Use rollover write token as write token to create new log file 
with tokens
   logWriteToken = rolloverLogWriteToken;
 }
-LOG.info("Computed the next log version for " + logFileId + " in " + 
parentPath + " as " + logVersion
-+ " with write-token " + logWriteToken);
+LOG.info("Computed the next log version for {} in {} as {} with 
write-token {}",
+logFileId, parentPath, logVersion, logWriteToken);
 
 Review comment:
   remove extra indent ???


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] smarthi commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

smarthi commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362386748
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java
 ##
 @@ -101,8 +101,8 @@ public void trySave(int taskPartitionId) {
 fs.rename(tmpMetaPath, metaPath);
   }
 } catch (IOException ioe) {
-  LOG.warn("Error trying to save partition metadata (this is okay, as long 
as atleast 1 of these succced), "
-  + partitionPath, ioe);
+  LOG.warn("Error trying to save partition metadata (this is okay, as long 
as atleast 1 of these succced), {}",
+  partitionPath, ioe);
 
 Review comment:
   remove extra indent ??


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] pratyakshsharma commented on a change in pull request #1150: [HUDI-288]: Add support for ingesting multiple kafka streams in a single DeltaStreamer deployment

[GitBox]

pratyakshsharma commented on a change in pull request #1150: [HUDI-288]: Add 
support for ingesting multiple kafka streams in a single DeltaStreamer 
deployment
URL: https://github.com/apache/incubator-hudi/pull/1150#discussion_r362386402
 
 

 ##
 File path: 
hudi-client/src/test/java/org/apache/hudi/common/HoodieTestDataGenerator.java
 ##
 @@ -80,16 +79,27 @@
   + "{\"name\": \"begin_lat\", \"type\": \"double\"},{\"name\": 
\"begin_lon\", \"type\": \"double\"},"
   + "{\"name\": \"end_lat\", \"type\": \"double\"},{\"name\": \"end_lon\", 
\"type\": \"double\"},"
   + "{\"name\":\"fare\",\"type\": \"double\"}]}";
+  public static String GROCERY_PURCHASE_SCHEMA = 
"{\"type\":\"record\",\"name\":\"purchaserec\",\"fields\":["
 
 Review comment:
   @vinothchandar I wanted to depict a real-life scenario in test-cases. Hence 
preferred to have different schemas for every topic. By modularizing the class, 
you mean I should write a class which extends this class?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (HUDI-233) Redo log statements using SLF4J

[leesf (Jira)]

[ 
https://issues.apache.org/jira/browse/HUDI-233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006630#comment-17006630
 ] 

leesf commented on HUDI-233:


@cdmikechen, sounds reasonable

> Redo log statements using SLF4J 
> 
>
> Key: HUDI-233
> URL: https://issues.apache.org/jira/browse/HUDI-233
> Project: Apache Hudi (incubating)
>  Issue Type: Bug
>  Components: Code Cleanup, newbie
>Affects Versions: 0.5.0
>Reporter: Vinoth Chandar
>Assignee: leesf
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.2
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Currently we are not employing variable substitution aggresively in the 
> project.  ala 
> {code:java}
> LogManager.getLogger(SomeName.class.getName()).info("Message: {}, Detail: 
> {}", message, detail);
> {code}
> This can improve performance since the string concatenation is deferrable to 
> when the logging is actually in effect.  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-hudi] pratyakshsharma commented on a change in pull request #1150: [HUDI-288]: Add support for ingesting multiple kafka streams in a single DeltaStreamer deployment

[GitBox]

pratyakshsharma commented on a change in pull request #1150: [HUDI-288]: Add 
support for ingesting multiple kafka streams in a single DeltaStreamer 
deployment
URL: https://github.com/apache/incubator-hudi/pull/1150#discussion_r362385735
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/model/TableConfig.java
 ##
 @@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+import java.util.Objects;
+
+/*
+Represents object with all the topic level overrides for multi table delta 
streamer execution
+ */
+@JsonIgnoreProperties(ignoreUnknown = true)
 
 Review comment:
   Basically it is a POJO and hudi-common is having most of the POJOs. Also 
since hudi-common is added as a dependency in most of the modules, it will be 
easier this way to use this class anywhere in future if need be. :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] sev7e0 commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

sev7e0 commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362383524
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java
 ##
 @@ -101,8 +101,8 @@ public void trySave(int taskPartitionId) {
 fs.rename(tmpMetaPath, metaPath);
   }
 } catch (IOException ioe) {
-  LOG.warn("Error trying to save partition metadata (this is okay, as long 
as atleast 1 of these succced), "
-  + partitionPath, ioe);
+  LOG.warn(String.format("Error trying to save partition metadata (this is 
okay, as long as atleast 1 of these "
 
 Review comment:
   It's my error. I'll make changes right away


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] pratyakshsharma commented on a change in pull request #1154: [HUDI-406] Added default partition path in TimestampBasedKeyGenerator

[GitBox]

pratyakshsharma commented on a change in pull request #1154: [HUDI-406] Added 
default partition path in TimestampBasedKeyGenerator
URL: https://github.com/apache/incubator-hudi/pull/1154#discussion_r362382958
 
 

 ##
 File path: hudi-spark/src/main/java/org/apache/hudi/DataSourceUtils.java
 ##
 @@ -64,6 +64,14 @@ public static String 
getNullableNestedFieldValAsString(GenericRecord record, Str
 }
   }
 
+  public static Object getNullableNestedFieldVal(GenericRecord record, String 
fieldName) {
+try {
+  return getNestedFieldVal(record, fieldName);
 
 Review comment:
   Done with the changes. @bvaradar 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] wangxianghu commented on issue #1163: [HUDI-460] Redo hudi-integ-test log statements using SLF4J

[GitBox]

wangxianghu commented on issue #1163: [HUDI-460] Redo hudi-integ-test log 
statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1163#issuecomment-570130189
 
 
   @leesf @vinothchandar @yanghua  Thank you for your advice, i will verify 
these changes in my local.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] XuQianJin-Stars commented on issue #1168: [HUDI-463] Redo hudi-utilities log statements using SLF4J

[GitBox]

XuQianJin-Stars commented on issue #1168: [HUDI-463] Redo hudi-utilities log 
statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1168#issuecomment-570125816
 
 
   > Hi @XuQianJin-Stars firstly, please verify the log print is correct after 
replacing with slf4j in your local.
   
   well, ok


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on issue #1168: [HUDI-463] Redo hudi-utilities log statements using SLF4J

[GitBox]

yanghua commented on issue #1168: [HUDI-463] Redo hudi-utilities log statements 
using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1168#issuecomment-570125630
 
 
   Hi @XuQianJin-Stars firstly, please verify the log print is correct after 
replacing with slf4j in your local.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on issue #1163: [HUDI-460] Redo hudi-integ-test log statements using SLF4J

[GitBox]

yanghua commented on issue #1163: [HUDI-460] Redo hudi-integ-test log 
statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1163#issuecomment-570125454
 
 
   > @leesf @yanghua for all these diffs on slf4j, I think we should establish 
what testing is good enough, ensure some jobs are tested end-end before we 
merge? where are we on this front?
   
   Reasonable. IMO, it's hard to verify this change via end-to-end test cases 
or automatic testing. We will require the contributors of the PRs to doing the 
verification in their local. What's more, we will verify the log print again in 
our local env.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] lamber-ken closed pull request #1164: [MINOR] String format should use %n rather than \n

[GitBox]

lamber-ken closed pull request #1164: [MINOR] String format should use %n 
rather than \n
URL: https://github.com/apache/incubator-hudi/pull/1164
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] lamber-ken commented on issue #1164: [MINOR] String format should use %n rather than \n

[GitBox]

lamber-ken commented on issue #1164: [MINOR] String format should use %n rather 
than \n
URL: https://github.com/apache/incubator-hudi/pull/1164#issuecomment-570124258
 
 
   > Hi @lamber-ken thanks for your contribution. Currently, the community is 
redoing on replacing log4j with slf4j. After this work, the `String.format();` 
will be removed. Therefore, maybe we do not need this PR?
   
   Okay, I got it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (HUDI-485) Check for where clause is wrong in HiveIncrementalPuller

[Pratyaksh Sharma (Jira)]

[ 
https://issues.apache.org/jira/browse/HUDI-485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006600#comment-17006600
 ] 

Pratyaksh Sharma commented on HUDI-485:
---

Hi [~vinoth], can you please tell me more about what issues you ran into. Also 
please point me to the Jira if we have any. Happy to fix it in a better way. 

> Check for where clause is wrong in HiveIncrementalPuller
> 
>
> Key: HUDI-485
> URL: https://issues.apache.org/jira/browse/HUDI-485
> Project: Apache Hudi (incubating)
>  Issue Type: Sub-task
>  Components: Incremental Pull, newbie
>Reporter: Pratyaksh Sharma
>Assignee: Pratyaksh Sharma
>Priority: Major
>
> HiveIncrementalPuller checks the clause in incrementalSqlFile like this -> 
> if (!incrementalSQL.contains("`_hoodie_commit_time` > '%targetBasePath'"))
> { LOG.info("Incremental SQL : " + incrementalSQL + " does not contain 
> `_hoodie_commit_time` > %targetBasePath. Please add " + "this clause for 
> incremental to work properly."); throw new HoodieIncrementalPullSQLException( 
> "Incremental SQL does not have clause `_hoodie_commit_time` > 
> '%targetBasePath', which " + "means its not pulling incrementally"); }
> Basically we are trying to add a placeholder here which is later replaced 
> with config.fromCommitTime here - 
> incrementalPullSQLtemplate.add("incrementalSQL", 
> String.format(incrementalSQL, config.fromCommitTime));
> Hence, the above check needs to replaced with `_hoodie_commit_time` > %s



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-hudi] yanghua commented on issue #1164: [MINOR] String format should use %n rather than \n

[GitBox]

yanghua commented on issue #1164: [MINOR] String format should use %n rather 
than \n
URL: https://github.com/apache/incubator-hudi/pull/1164#issuecomment-570123953
 
 
   Hi @lamber-ken thanks for your contribution. Currently, the community is 
redoing on replacing log4j with slf4j. After this work, the `String.format();` 
will be removed. Therefore, maybe we do not need this PR?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (HUDI-485) Check for where clause is wrong in HiveIncrementalPuller

[Pratyaksh Sharma (Jira)]

 [ 
https://issues.apache.org/jira/browse/HUDI-485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pratyaksh Sharma updated HUDI-485:
--
Description: 
HiveIncrementalPuller checks the clause in incrementalSqlFile like this -> 

if (!incrementalSQL.contains("`_hoodie_commit_time` > '%targetBasePath'"))

{ LOG.info("Incremental SQL : " + incrementalSQL + " does not contain 
`_hoodie_commit_time` > %targetBasePath. Please add " + "this clause for 
incremental to work properly."); throw new HoodieIncrementalPullSQLException( 
"Incremental SQL does not have clause `_hoodie_commit_time` > 
'%targetBasePath', which " + "means its not pulling incrementally"); }

Basically we are trying to add a placeholder here which is later replaced with 
config.fromCommitTime here - 

incrementalPullSQLtemplate.add("incrementalSQL", String.format(incrementalSQL, 
config.fromCommitTime));

Hence, the above check needs to replaced with `_hoodie_commit_time` > %s

  was:
HiveIncrementalPuller checks the clause in incrementalSqlFile like this -> 

if (!incrementalSQL.contains("`_hoodie_commit_time` > '%targetBasePath'")) {
 LOG.info("Incremental SQL : " + incrementalSQL
 + " does not contain `_hoodie_commit_time` > %targetBasePath. Please add "
 + "this clause for incremental to work properly.");
 throw new HoodieIncrementalPullSQLException(
 "Incremental SQL does not have clause `_hoodie_commit_time` > 
'%targetBasePath', which "
 + "means its not pulling incrementally");
}

Basically we are trying to add a placeholder here which is later replaced with 
config.fromCommitTime here - 

incrementalPullSQLtemplate.add("incrementalSQL", String.format(incrementalSQL, 
config.fromCommitTime));

Hence, the above check needs to replaced with `_hoodie_commit_time` > 
%targetBasePath


> Check for where clause is wrong in HiveIncrementalPuller
> 
>
> Key: HUDI-485
> URL: https://issues.apache.org/jira/browse/HUDI-485
> Project: Apache Hudi (incubating)
>  Issue Type: Sub-task
>  Components: Incremental Pull, newbie
>Reporter: Pratyaksh Sharma
>Assignee: Pratyaksh Sharma
>Priority: Major
>
> HiveIncrementalPuller checks the clause in incrementalSqlFile like this -> 
> if (!incrementalSQL.contains("`_hoodie_commit_time` > '%targetBasePath'"))
> { LOG.info("Incremental SQL : " + incrementalSQL + " does not contain 
> `_hoodie_commit_time` > %targetBasePath. Please add " + "this clause for 
> incremental to work properly."); throw new HoodieIncrementalPullSQLException( 
> "Incremental SQL does not have clause `_hoodie_commit_time` > 
> '%targetBasePath', which " + "means its not pulling incrementally"); }
> Basically we are trying to add a placeholder here which is later replaced 
> with config.fromCommitTime here - 
> incrementalPullSQLtemplate.add("incrementalSQL", 
> String.format(incrementalSQL, config.fromCommitTime));
> Hence, the above check needs to replaced with `_hoodie_commit_time` > %s



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-hudi] lamber-ken commented on issue #1167: [HUDI-484] Fix NPE when reading IncrementalPull.sqltemplate in HiveIncrementalPuller

[GitBox]

lamber-ken commented on issue #1167: [HUDI-484] Fix NPE when reading 
IncrementalPull.sqltemplate in HiveIncrementalPuller
URL: https://github.com/apache/incubator-hudi/pull/1167#issuecomment-570117281
 
 
   > I know this probably just tests the NPE.. but little more love and add a 
actual valid working test case that does a real pull? :) , in the spirit of 
improving the tool?
   
   Reasonable, can split it to a new thread that end to end test the 
HiveIncrementalPuller? because this pr just fix the NPE.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] lamber-ken commented on a change in pull request #1167: [HUDI-484] Fix NPE when reading IncrementalPull.sqltemplate in HiveIncrementalPuller

[GitBox]

lamber-ken commented on a change in pull request #1167: [HUDI-484] Fix NPE when 
reading IncrementalPull.sqltemplate in HiveIncrementalPuller
URL: https://github.com/apache/incubator-hudi/pull/1167#discussion_r362369737
 
 

 ##
 File path: 
hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHiveIncrementalPuller.java
 ##
 @@ -0,0 +1,24 @@
+package org.apache.hudi.utilities;
+
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+public class TestHiveIncrementalPuller {
+
+  private HiveIncrementalPuller.Config config;
+
+  @Before
+  public void init() {
+config = new HiveIncrementalPuller.Config();
+  }
+
+  @Test
+  public void testInitHiveIncrementalPuller() throws Exception {
+
+HiveIncrementalPuller puller = new HiveIncrementalPuller(config);
 
 Review comment:
   Got it, thank you for your guidance.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] lamber-ken edited a comment on issue #1105: [HUDI-405] Remove HIVE_ASSUME_DATE_PARTITION_OPT_KEY config from DataSource

[GitBox]

lamber-ken edited a comment on issue #1105: [HUDI-405] Remove 
HIVE_ASSUME_DATE_PARTITION_OPT_KEY config from DataSource
URL: https://github.com/apache/incubator-hudi/pull/1105#issuecomment-570116535
 
 
   @vinothchandar Thanks, done.
   
   @bvaradar here are some useful information.
   
   **Useful detail information** 
   BUG : https://github.com/apache/incubator-hudi/issues/828
   ISSUE : https://issues.apache.org/jira/browse/HUDI-405
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] lamber-ken commented on issue #1105: [HUDI-405] Remove HIVE_ASSUME_DATE_PARTITION_OPT_KEY config from DataSource

[GitBox]

lamber-ken commented on issue #1105: [HUDI-405] Remove 
HIVE_ASSUME_DATE_PARTITION_OPT_KEY config from DataSource
URL: https://github.com/apache/incubator-hudi/pull/1105#issuecomment-570116535
 
 
   > @lamber-ken can you fix the PR title..
   > I am going to ask @bvaradar to take a quick look and confirm dropping this 
is fine at uber.. and after these two we can merge
   
   @vinothchandar Thanks, done.
   
   **Useful detail information** 
   BUG : https://github.com/apache/incubator-hudi/issues/828
   ISSUE : https://issues.apache.org/jira/browse/HUDI-405
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (HUDI-405) Remove HIVE_ASSUME_DATE_PARTITION_OPT_KEY config from DataSource

[lamber-ken (Jira)]

 [ 
https://issues.apache.org/jira/browse/HUDI-405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

lamber-ken updated HUDI-405:

Summary: Remove HIVE_ASSUME_DATE_PARTITION_OPT_KEY config from DataSource  
(was: Remove remove HIVE_ASSUME_DATE_PARTITION_OPT_KEY config from DataSource)

> Remove HIVE_ASSUME_DATE_PARTITION_OPT_KEY config from DataSource
> 
>
> Key: HUDI-405
> URL: https://issues.apache.org/jira/browse/HUDI-405
> Project: Apache Hudi (incubating)
>  Issue Type: Bug
>  Components: Hive Integration
>Reporter: lamber-ken
>Assignee: lamber-ken
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> If user custom the partition extractor, HiveSyncTool sync no partition at the 
> first commit.
> ISSUE: [https://github.com/apache/incubator-hudi/issues/828]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HUDI-405) Remove remove HIVE_ASSUME_DATE_PARTITION_OPT_KEY config from DataSource

[lamber-ken (Jira)]

 [ 
https://issues.apache.org/jira/browse/HUDI-405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

lamber-ken updated HUDI-405:

Summary: Remove remove HIVE_ASSUME_DATE_PARTITION_OPT_KEY config from 
DataSource  (was: Fix sync no hive partition at first time)

> Remove remove HIVE_ASSUME_DATE_PARTITION_OPT_KEY config from DataSource
> ---
>
> Key: HUDI-405
> URL: https://issues.apache.org/jira/browse/HUDI-405
> Project: Apache Hudi (incubating)
>  Issue Type: Bug
>  Components: Hive Integration
>Reporter: lamber-ken
>Assignee: lamber-ken
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> If user custom the partition extractor, HiveSyncTool sync no partition at the 
> first commit.
> ISSUE: [https://github.com/apache/incubator-hudi/issues/828]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1149: [WIP] [HUDI-472] Introduce configurations and new modes of sorting for bulk_insert

[GitBox]

vinothchandar commented on a change in pull request #1149: [WIP] [HUDI-472] 
Introduce configurations and new modes of sorting for bulk_insert
URL: https://github.com/apache/incubator-hudi/pull/1149#discussion_r362366288
 
 

 ##
 File path: hudi-client/src/main/java/org/apache/hudi/HoodieWriteClient.java
 ##
 @@ -367,20 +370,30 @@ public static SparkConf registerClasses(SparkConf conf) {
 }
   }
 
+  private BulkInsertMapFunction getBulkInsertMapFunction(
+  boolean isSorted, String commitTime, HoodieWriteConfig config, 
HoodieTable hoodieTable,
+  List fileIDPrefixes) {
+if (isSorted) {
+  return new BulkInsertMapFunctionForSortedRecords(
+  commitTime, config, hoodieTable, fileIDPrefixes);
+}
+return new BulkInsertMapFunctionForNonSortedRecords(
+commitTime, config, hoodieTable, fileIDPrefixes);
+  }
+
   private JavaRDD bulkInsertInternal(JavaRDD> 
dedupedRecords, String commitTime,
   HoodieTable table, Option 
bulkInsertPartitioner) {
 
 Review comment:
   This was added by @ovj and uber/marmaray project passes in an implementation 
using the RDD api 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1167: [HUDI-484] Fix NPE when reading IncrementalPull.sqltemplate in HiveIncrementalPuller

[GitBox]

vinothchandar commented on a change in pull request #1167: [HUDI-484] Fix NPE 
when reading IncrementalPull.sqltemplate in HiveIncrementalPuller
URL: https://github.com/apache/incubator-hudi/pull/1167#discussion_r362366023
 
 

 ##
 File path: 
hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHiveIncrementalPuller.java
 ##
 @@ -0,0 +1,24 @@
+package org.apache.hudi.utilities;
+
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+public class TestHiveIncrementalPuller {
+
+  private HiveIncrementalPuller.Config config;
+
+  @Before
+  public void init() {
+config = new HiveIncrementalPuller.Config();
+  }
+
+  @Test
+  public void testInitHiveIncrementalPuller() throws Exception {
+
+HiveIncrementalPuller puller = new HiveIncrementalPuller(config);
 
 Review comment:
   Not following this.. a constructor can either throw an exception or return a 
non-null object right? should we be using a try-catch and calling `fail` from 
the catch block to ensure the NPE is not thrown? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1167: [HUDI-484] Fix NPE when reading IncrementalPull.sqltemplate in HiveIncrementalPuller

[GitBox]

vinothchandar commented on a change in pull request #1167: [HUDI-484] Fix NPE 
when reading IncrementalPull.sqltemplate in HiveIncrementalPuller
URL: https://github.com/apache/incubator-hudi/pull/1167#discussion_r362365869
 
 

 ##
 File path: 
hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHiveIncrementalPuller.java
 ##
 @@ -0,0 +1,24 @@
+package org.apache.hudi.utilities;
+
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+public class TestHiveIncrementalPuller {
+
+  private HiveIncrementalPuller.Config config;
+
+  @Before
+  public void init() {
 
 Review comment:
   rename to `setup()`? most of our tests are that way I think. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] vinothchandar commented on issue #1105: [WIP] [HUDI-405] Fix sync no hive partition at first time

[GitBox]

vinothchandar commented on issue #1105: [WIP] [HUDI-405] Fix sync no hive 
partition at first time
URL: https://github.com/apache/incubator-hudi/pull/1105#issuecomment-570112392
 
 
   @lamber-ken can you fix the PR title.. 
   I am going to ask @bvaradar to take a quick look and confirm dropping this 
is fine at uber.. and after these two we can merge


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (HUDI-491) Problems encountered in building hudi

[Zijie Lu (Jira)]

Zijie Lu created HUDI-491:
-

 Summary: Problems encountered in building hudi
 Key: HUDI-491
 URL: https://issues.apache.org/jira/browse/HUDI-491
 Project: Apache Hudi (incubating)
  Issue Type: Bug
Reporter: Zijie Lu


I don't why I can't build hudi and the error msg is as followed:

incubator-hudi/hudi-cli/src/main/java/org/apache/hudi/cli/HoodieHistoryFileNameProvider.java:[30,2]
 cannot find symbol
[ERROR] symbol: class Order
[ERROR] 
/Users/code/incubator-hudi/hudi-cli/src/main/java/org/apache/hudi/cli/commands/ArchivedCommitsCommand.java:[52,2]
 cannot find symbol
[ERROR] symbol: class Component
[ERROR] 
/Users/code/incubator-hudi/hudi-cli/src/main/java/org/apache/hudi/cli/commands/CommitsCommand.java:[51,2]
 cannot find symbol
[ERROR] symbol: class Component
[ERROR] 
/Users/code/incubator-hudi/hudi-cli/src/main/java/org/apache/hudi/cli/commands/HoodieSyncCommand.java:[40,2]
 cannot find symbol
[ERROR] symbol: class Component
[ERROR] 
/Users/code/incubator-hudi/hudi-cli/src/main/java/org/apache/hudi/cli/commands/SavepointsCommand.java:[49,2]
 cannot find symbol
[ERROR] symbol: class Component

 

It seems that it is caused by missing spring context.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-hudi] vinothchandar commented on issue #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

vinothchandar commented on issue #1161: [HUDI-457]Redo hudi-common log 
statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#issuecomment-570111790
 
 
   Similar comment on how we are ensuring the log statements are indeed working 
properly inside spark and hive and presto after this change.. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Build failed in Jenkins: hudi-snapshot-deployment-0.5 #147

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 2.17 KB...]
/home/jenkins/tools/maven/apache-maven-3.5.4/bin:
m2.conf
mvn
mvn.cmd
mvnDebug
mvnDebug.cmd
mvnyjp

/home/jenkins/tools/maven/apache-maven-3.5.4/boot:
plexus-classworlds-2.5.2.jar

/home/jenkins/tools/maven/apache-maven-3.5.4/conf:
logging
settings.xml
toolchains.xml

/home/jenkins/tools/maven/apache-maven-3.5.4/conf/logging:
simplelogger.properties

/home/jenkins/tools/maven/apache-maven-3.5.4/lib:
aopalliance-1.0.jar
cdi-api-1.0.jar
cdi-api.license
commons-cli-1.4.jar
commons-cli.license
commons-io-2.5.jar
commons-io.license
commons-lang3-3.5.jar
commons-lang3.license
ext
guava-20.0.jar
guice-4.2.0-no_aop.jar
jansi-1.17.1.jar
jansi-native
javax.inject-1.jar
jcl-over-slf4j-1.7.25.jar
jcl-over-slf4j.license
jsr250-api-1.0.jar
jsr250-api.license
maven-artifact-3.5.4.jar
maven-artifact.license
maven-builder-support-3.5.4.jar
maven-builder-support.license
maven-compat-3.5.4.jar
maven-compat.license
maven-core-3.5.4.jar
maven-core.license
maven-embedder-3.5.4.jar
maven-embedder.license
maven-model-3.5.4.jar
maven-model-builder-3.5.4.jar
maven-model-builder.license
maven-model.license
maven-plugin-api-3.5.4.jar
maven-plugin-api.license
maven-repository-metadata-3.5.4.jar
maven-repository-metadata.license
maven-resolver-api-1.1.1.jar
maven-resolver-api.license
maven-resolver-connector-basic-1.1.1.jar
maven-resolver-connector-basic.license
maven-resolver-impl-1.1.1.jar
maven-resolver-impl.license
maven-resolver-provider-3.5.4.jar
maven-resolver-provider.license
maven-resolver-spi-1.1.1.jar
maven-resolver-spi.license
maven-resolver-transport-wagon-1.1.1.jar
maven-resolver-transport-wagon.license
maven-resolver-util-1.1.1.jar
maven-resolver-util.license
maven-settings-3.5.4.jar
maven-settings-builder-3.5.4.jar
maven-settings-builder.license
maven-settings.license
maven-shared-utils-3.2.1.jar
maven-shared-utils.license
maven-slf4j-provider-3.5.4.jar
maven-slf4j-provider.license
org.eclipse.sisu.inject-0.3.3.jar
org.eclipse.sisu.inject.license
org.eclipse.sisu.plexus-0.3.3.jar
org.eclipse.sisu.plexus.license
plexus-cipher-1.7.jar
plexus-cipher.license
plexus-component-annotations-1.7.1.jar
plexus-component-annotations.license
plexus-interpolation-1.24.jar
plexus-interpolation.license
plexus-sec-dispatcher-1.4.jar
plexus-sec-dispatcher.license
plexus-utils-3.1.0.jar
plexus-utils.license
slf4j-api-1.7.25.jar
slf4j-api.license
wagon-file-3.1.0.jar
wagon-file.license
wagon-http-3.1.0-shaded.jar
wagon-http.license
wagon-provider-api-3.1.0.jar
wagon-provider-api.license

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/ext:
README.txt

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native:
freebsd32
freebsd64
linux32
linux64
osx
README.txt
windows32
windows64

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/freebsd32:
libjansi.so

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/freebsd64:
libjansi.so

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/linux32:
libjansi.so

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/linux64:
libjansi.so

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/osx:
libjansi.jnilib

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/windows32:
jansi.dll

/home/jenkins/tools/maven/apache-maven-3.5.4/lib/jansi-native/windows64:
jansi.dll
Finished /home/jenkins/tools/maven/apache-maven-3.5.4 Directory Listing :
Detected current version as: 
'HUDI_home=
0.5.1-SNAPSHOT'
[INFO] Scanning for projects...
[INFO] 
[INFO] Reactor Build Order:
[INFO] 
[INFO] Hudi   [pom]
[INFO] hudi-common[jar]
[INFO] hudi-timeline-service  [jar]
[INFO] hudi-hadoop-mr [jar]
[INFO] hudi-client[jar]
[INFO] hudi-hive  [jar]
[INFO] hudi-spark [jar]
[INFO] hudi-utilities [jar]
[INFO] hudi-cli   [jar]
[INFO] hudi-hadoop-mr-bundle  [jar]
[INFO] hudi-hive-bundle   [jar]
[INFO] hudi-spark-bundle  [jar]
[INFO] hudi-presto-bundle [jar]
[INFO] hudi-utilities-bundle  [jar]
[INFO] hudi-timeline-server-bundle[j

[GitHub] [incubator-hudi] vinothchandar commented on issue #1168: [HUDI-463] Redo hudi-utilities log statements using SLF4J

[GitBox]

vinothchandar commented on issue #1168: [HUDI-463] Redo hudi-utilities log 
statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1168#issuecomment-570111643
 
 
   Same question here.. Have we verified the logs actually show up on the demo 
or some end-end test? the tests could be passing, sure.. since this is a 
logging only change.. but how are we ensuring each of these modules continues 
to log correctly? cc @leesf @yanghua  


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] vinothchandar commented on issue #1163: [HUDI-460] Redo hudi-integ-test log statements using SLF4J

[GitBox]

vinothchandar commented on issue #1163: [HUDI-460] Redo hudi-integ-test log 
statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1163#issuecomment-570111514
 
 
   @leesf @yanghua for all these diffs on slf4j, I think we should establish 
what testing is good enough, ensure some jobs are tested end-end before we 
merge? where are we on this front? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362358513
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java
 ##
 @@ -63,12 +63,12 @@
   public static final SimpleDateFormat COMMIT_FORMATTER = new 
SimpleDateFormat("MMddHHmmss");
 
   public static final Set VALID_EXTENSIONS_IN_ACTIVE_TIMELINE = new 
HashSet<>(Arrays.asList(
-  new String[]{COMMIT_EXTENSION, INFLIGHT_COMMIT_EXTENSION, 
REQUESTED_COMMIT_EXTENSION, DELTA_COMMIT_EXTENSION,
 
 Review comment:
   Can we revert this change? This PR aims to replace the log framework, right? 
Let's do one thing in a PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362357721
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java
 ##
 @@ -101,8 +101,8 @@ public void trySave(int taskPartitionId) {
 fs.rename(tmpMetaPath, metaPath);
   }
 } catch (IOException ioe) {
-  LOG.warn("Error trying to save partition metadata (this is okay, as long 
as atleast 1 of these succced), "
-  + partitionPath, ioe);
+  LOG.warn(String.format("Error trying to save partition metadata (this is 
okay, as long as atleast 1 of these "
 
 Review comment:
   Since we have used slf4j, we may not need to use `String.format`. The slf4j 
framework supports `placeholder`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362358217
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFileReader.java
 ##
 @@ -113,7 +113,7 @@ public void run() {
 try {
   close();
 } catch (Exception e) {
-  LOG.warn("unable to close input stream for log file " + logFile, e);
+  LOG.warn(String.format("unable to close input stream for log file 
%s", logFile), e);
 
 Review comment:
   ditto


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362358621
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java
 ##
 @@ -123,8 +123,8 @@ protected void refreshTimeline(HoodieTimeline 
visibleActiveTimeline) {
   }
 });
 long storePartitionsTs = timer.endTimer();
-LOG.info("addFilesToView: NumFiles=" + statuses.length + ", 
FileGroupsCreationTime=" + fgBuildTimeTakenMs
-+ ", StoreTimeTaken=" + storePartitionsTs);
+LOG.info("addFilesToView: NumFiles={}, FileGroupsCreationTime={}, 
StoreTimeTaken={}",
+statuses.length, fgBuildTimeTakenMs, storePartitionsTs);
 
 Review comment:
   ditto


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362357743
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java
 ##
 @@ -111,7 +111,7 @@ public void trySave(int taskPartitionId) {
 fs.delete(tmpMetaPath, false);
   }
 } catch (IOException ioe) {
-  LOG.warn("Error trying to clean up temporary files for " + 
partitionPath, ioe);
+  LOG.warn(String.format("Error trying to clean up temporary files for 
%s", partitionPath), ioe);
 
 Review comment:
   ditto


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362363473
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/util/collection/DiskBasedMap.java
 ##
 @@ -111,9 +111,8 @@ private void initFile(File writeOnlyFile) throws 
IOException {
   writeOnlyFile.getParentFile().mkdir();
 }
 writeOnlyFile.createNewFile();
-LOG.info("Spilling to file location " + writeOnlyFile.getAbsolutePath() + 
" in host ("
-+ InetAddress.getLocalHost().getHostAddress() + ") with hostname (" + 
InetAddress.getLocalHost().getHostName()
-+ ")");
+LOG.info("Spilling to file location {} in host ({}) with hostname ({})", 
writeOnlyFile.getAbsolutePath(),
+InetAddress.getLocalHost().getHostAddress(), 
InetAddress.getLocalHost().getHostName());
 
 Review comment:
   ditto about indent


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362358202
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
 ##
 @@ -196,34 +196,34 @@ public void scan() {
   HoodieLogBlock lastBlock = currentInstantLogBlocks.peek();
   // handle corrupt blocks separately since they may not have 
metadata
   if (lastBlock.getBlockType() == CORRUPT_BLOCK) {
-LOG.info("Rolling back the last corrupted log block read 
in " + logFile.getPath());
+LOG.info("Rolling back the last corrupted log block read 
in {}", logFile.getPath());
 currentInstantLogBlocks.pop();
 numBlocksRolledBack++;
   } else if (lastBlock.getBlockType() != CORRUPT_BLOCK
   && 
targetInstantForCommandBlock.contentEquals(lastBlock.getLogBlockHeader().get(INSTANT_TIME)))
 {
 // rollback last data block or delete block
-LOG.info("Rolling back the last log block read in " + 
logFile.getPath());
+LOG.info("Rolling back the last log block read in {}", 
logFile.getPath());
 currentInstantLogBlocks.pop();
 numBlocksRolledBack++;
   } else if (!targetInstantForCommandBlock
   
.contentEquals(currentInstantLogBlocks.peek().getLogBlockHeader().get(INSTANT_TIME)))
 {
 // invalid or extra rollback block
-LOG.warn("TargetInstantTime " + 
targetInstantForCommandBlock
-+ " invalid or extra rollback command block in " + 
logFile.getPath());
+LOG.warn("TargetInstantTime {} invalid or extra rollback 
command block in {}",
+targetInstantForCommandBlock, logFile.getPath());
 
 Review comment:
   Here, maybe there is one redundant indent?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362363286
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/util/FailSafeConsistencyGuard.java
 ##
 @@ -104,7 +104,7 @@ public void waitForFilesVisibility(String dirPath, 
List files, FileVisib
 return candidateFiles.isEmpty();
 }
   } catch (IOException ioe) {
-LOG.warn("Got IOException waiting for file event. Have tried " + 
retryNum + " time(s)", ioe);
+LOG.warn(String.format("Got IOException waiting for file event. Have 
tried %s time(s)", retryNum), ioe);
 
 Review comment:
   ditto about using placeholder


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362360577
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/table/view/FileSystemViewManager.java
 ##
 @@ -158,8 +158,8 @@ private static HoodieTableFileSystemView 
createInMemoryFileSystemView(Serializab
*/
   private static RemoteHoodieTableFileSystemView 
createRemoteFileSystemView(SerializableConfiguration conf,
   FileSystemViewStorageConfig viewConf, HoodieTableMetaClient metaClient) {
-LOG.info("Creating remote view for basePath " + metaClient.getBasePath() + 
". Server="
-+ viewConf.getRemoteViewServerHost() + ":" + 
viewConf.getRemoteViewServerPort());
+LOG.info("Creating remote view for basePath {}. Server={}:{}",
+metaClient.getBasePath(), viewConf.getRemoteViewServerHost(), 
viewConf.getRemoteViewServerPort());
 
 Review comment:
   ditto about indent


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362363335
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/util/RocksDBDAO.java
 ##
 @@ -352,8 +352,8 @@ public void delete(String columnFamilyName, String key) {
   }
 }
 
-LOG.info("Prefix Search for (query=" + prefix + ") on " + columnFamilyName 
+ ". Total Time Taken (msec)="
-+ timer.endTimer() + ". Serialization Time taken(micro)=" + 
timeTakenMicro + ", num entries=" + results.size());
+LOG.info("Prefix Search for (query={}) on {}. Total Time Taken (msec)={}. 
Serialization Time taken(micro)={}, "
++ "num entries={}", prefix, columnFamilyName, timer.endTimer(), 
timeTakenMicro, results.size());
 
 Review comment:
   ditto about indent


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362358327
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatWriter.java
 ##
 @@ -93,11 +93,11 @@
   }
   if (!isAppendSupported) {
 this.logFile = logFile.rollOver(fs, rolloverLogWriteToken);
-LOG.info("Append not supported.. Rolling over to " + logFile);
+LOG.info("Append not supported.. Rolling over to {}", logFile);
 createNewFile();
   }
 } else {
-  LOG.info(logFile + " does not exist. Create a new file");
+  LOG.info("{} does not exist. Create a new file", logFile.getPath());
 
 Review comment:
   Shall we use `logFile.getPath()` here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362363156
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/table/view/SpillableMapBasedFileSystemView.java
 ##
 @@ -66,8 +66,8 @@ public SpillableMapBasedFileSystemView(HoodieTableMetaClient 
metaClient, HoodieT
   @Override
   protected Map> createPartitionToFileGroups() {
 try {
-  LOG.info("Creating Partition To File groups map using external spillable 
Map. Max Mem=" + maxMemoryForFileGroupMap
-  + ", BaseDir=" + baseStoreDir);
+  LOG.info("Creating Partition To File groups map using external spillable 
Map.  Max Mem={}, BaseDir={}",
+  maxMemoryForPendingCompaction, baseStoreDir);
 
 Review comment:
   ditto about indent


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362358375
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatWriter.java
 ##
 @@ -180,8 +180,8 @@ private Writer rolloverIfNeeded() throws IOException, 
InterruptedException {
 if (getCurrentSize() > sizeThreshold) {
   // TODO - make an end marker which seals the old log file (no more 
appends possible to that
   // file).
-  LOG.info("CurrentSize " + getCurrentSize() + " has reached threshold " + 
sizeThreshold
-  + ". Rolling over to the next version");
+  LOG.info("CurrentSize {} has reached threshold {}. Rolling over to the 
next version",
+  getCurrentSize(), sizeThreshold);
 
 Review comment:
   ditto


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362363178
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/table/view/SpillableMapBasedFileSystemView.java
 ##
 @@ -80,8 +80,8 @@ public SpillableMapBasedFileSystemView(HoodieTableMetaClient 
metaClient, HoodieT
   protected Map> 
createFileIdToPendingCompactionMap(
   Map> 
fgIdToPendingCompaction) {
 try {
-  LOG.info("Creating Pending Compaction map using external spillable Map. 
Max Mem=" + maxMemoryForPendingCompaction
-  + ", BaseDir=" + baseStoreDir);
+  LOG.info("Creating Pending Compaction map using external spillable Map. 
Max Mem={}, BaseDir={}",
+  maxMemoryForPendingCompaction, baseStoreDir);
 
 Review comment:
   ditto about indent


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362363123
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/table/view/RocksDbBasedFileSystemView.java
 ##
 @@ -184,8 +184,8 @@ protected void storePartitionView(String partitionPath, 
List fi
 
 // record that partition is loaded.
 rocksDB.put(schemaHelper.getColFamilyForStoredPartitions(), lookupKey, 
Boolean.TRUE);
-LOG.info("Finished adding new partition (" + partitionPath + ") to ROCKSDB 
based file-system view at "
-+ config.getRocksdbBasePath() + ", Total file-groups=" + 
fileGroups.size());
+LOG.info("Finished adding new partition ({}) to ROCKSDB based file-system 
view at {}, Total file-groups={}",
+partitionPath, config.getRocksdbBasePath(), fileGroups.size());
 
 Review comment:
   ditto about indent


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362363091
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/table/view/RocksDbBasedFileSystemView.java
 ##
 @@ -157,8 +157,8 @@ protected boolean isPartitionAvailableInStore(String 
partitionPath) {
 
   @Override
   protected void storePartitionView(String partitionPath, 
List fileGroups) {
-LOG.info("Resetting and adding new partition (" + partitionPath + ") to 
ROCKSDB based file-system view at "
-+ config.getRocksdbBasePath() + ", Total file-groups=" + 
fileGroups.size());
+LOG.info("Resetting and adding new partition ({}) to ROCKSDB based 
file-system view at {}, Total file-groups={}",
+partitionPath, config.getRocksdbBasePath(), fileGroups.size());
 
 Review comment:
   ditto about indent


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common log statements using SLF4J

[GitBox]

yanghua commented on a change in pull request #1161: [HUDI-457]Redo hudi-common 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1161#discussion_r362363365
 
 

 ##
 File path: 
hudi-common/src/main/java/org/apache/hudi/common/util/TimelineDiffHelper.java
 ##
 @@ -64,8 +64,8 @@ public static TimelineDiffResult 
getNewInstantsForIncrementalSync(HoodieTimeline
   if (!lostPendingCompactions.isEmpty()) {
 // If a compaction is unscheduled, fall back to complete refresh of fs 
view since some log files could have been
 // moved. Its unsafe to incrementally sync in that case.
-LOG.warn("Some pending compactions are no longer in new timeline 
(unscheduled ?). They are :"
-+ lostPendingCompactions);
+LOG.warn("Some pending compactions are no longer in new timeline 
(unscheduled ?). They are :{}",
+lostPendingCompactions);
 
 Review comment:
   ditto about indent


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (HUDI-461) Redo hudi-spark log statements using SLF4J

[Forward Xu (Jira)]

 [ 
https://issues.apache.org/jira/browse/HUDI-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Forward Xu reassigned HUDI-461:
---

Assignee: Forward Xu

> Redo hudi-spark log statements using SLF4J
> --
>
> Key: HUDI-461
> URL: https://issues.apache.org/jira/browse/HUDI-461
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>  Components: Code Cleanup
>Reporter: leesf
>Assignee: Forward Xu
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-hudi] XuQianJin-Stars commented on issue #1168: [HUDI-463] Redo hudi-utilities log statements using SLF4J

[GitBox]

XuQianJin-Stars commented on issue #1168: [HUDI-463] Redo hudi-utilities log 
statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1168#issuecomment-570109340
 
 
   hi, @smarthi  Thank you very much, I addressed commits.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (HUDI-490) Add DeltaStream API example to hudi-examples

[dengziming (Jira)]

dengziming created HUDI-490:
---

 Summary: Add DeltaStream API example to hudi-examples
 Key: HUDI-490
 URL: https://issues.apache.org/jira/browse/HUDI-490
 Project: Apache Hudi (incubating)
  Issue Type: Sub-task
Reporter: dengziming






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HUDI-489) Add hudi DataSource API example to hudi-examples

[dengziming (Jira)]

dengziming created HUDI-489:
---

 Summary: Add hudi DataSource API example to hudi-examples
 Key: HUDI-489
 URL: https://issues.apache.org/jira/browse/HUDI-489
 Project: Apache Hudi (incubating)
  Issue Type: Sub-task
Reporter: dengziming






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HUDI-489) Add hudi DataSource API example to hudi-examples

[dengziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/HUDI-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

dengziming reassigned HUDI-489:
---

Assignee: dengziming

> Add hudi DataSource API example to hudi-examples
> 
>
> Key: HUDI-489
> URL: https://issues.apache.org/jira/browse/HUDI-489
> Project: Apache Hudi (incubating)
>  Issue Type: Sub-task
>Reporter: dengziming
>Assignee: dengziming
>Priority: Minor
>  Labels: starter
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[incubator-hudi] branch master updated: [MINOR] Fix error usage of String.format (#1169)

[leesf]

This is an automated email from the ASF dual-hosted git repository.

leesf pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git


The following commit(s) were added to refs/heads/master by this push:
 new e1e5fe3  [MINOR] Fix error usage of String.format (#1169)
e1e5fe3 is described below

commit e1e5fe33249bf511486073dd9cf48e5b7ea14816
Author: lamber-ken 
AuthorDate: Wed Jan 1 19:11:15 2020 -0600

[MINOR] Fix error usage of String.format (#1169)
---
 .../src/main/java/org/apache/hudi/utilities/HoodieSnapshotCopier.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotCopier.java
 
b/hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotCopier.java
index 06fc5de..d24319e 100644
--- 
a/hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotCopier.java
+++ 
b/hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotCopier.java
@@ -156,7 +156,7 @@ public class HoodieSnapshotCopier implements Serializable {
 // Create the _SUCCESS tag
 Path successTagPath = new Path(outputDir + "/_SUCCESS");
 if (!fs.exists(successTagPath)) {
-  LOG.info(String.format("Creating _SUCCESS under targetBasePath: $s", 
outputDir));
+  LOG.info(String.format("Creating _SUCCESS under targetBasePath: %s", 
outputDir));
   fs.createNewFile(successTagPath);
 }
   }

[GitHub] [incubator-hudi] leesf merged pull request #1169: [MINOR] Fix error usage of String.format

[GitBox]

leesf merged pull request #1169: [MINOR] Fix error usage of String.format
URL: https://github.com/apache/incubator-hudi/pull/1169
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] lamber-ken commented on issue #1105: [WIP] [HUDI-405] Fix sync no hive partition at first time

[GitBox]

lamber-ken commented on issue #1105: [WIP] [HUDI-405] Fix sync no hive 
partition at first time
URL: https://github.com/apache/incubator-hudi/pull/1105#issuecomment-570081428
 
 
   @vinothchandar Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] smarthi commented on issue #1168: [HUDI-463] Redo hudi-utilities log statements using SLF4J

[GitBox]

smarthi commented on issue #1168: [HUDI-463] Redo hudi-utilities log statements 
using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1168#issuecomment-570076813
 
 
   Please fix the incorrect imports throughtout. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] smarthi commented on a change in pull request #1168: [HUDI-463] Redo hudi-utilities log statements using SLF4J

[GitBox]

smarthi commented on a change in pull request #1168: [HUDI-463] Redo 
hudi-utilities log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1168#discussion_r362338554
 
 

 ##
 File path: 
hudi-utilities/src/main/java/org/apache/hudi/utilities/HDFSParquetImporter.java
 ##
 @@ -59,14 +57,16 @@
 import java.util.List;
 import java.util.Properties;
 
+import parquet.org.slf4j.Logger;
+import parquet.org.slf4j.LoggerFactory;
 
 Review comment:
   is this the correct import?  Are we not using the standard org.slf4j.* ? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (HUDI-459) Redo hudi-hive log statements using SLF4J

[lamber-ken (Jira)]

 [ 
https://issues.apache.org/jira/browse/HUDI-459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

lamber-ken reassigned HUDI-459:
---

Assignee: lamber-ken

> Redo hudi-hive log statements using SLF4J
> -
>
> Key: HUDI-459
> URL: https://issues.apache.org/jira/browse/HUDI-459
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>  Components: Code Cleanup
>Reporter: leesf
>Assignee: lamber-ken
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HUDI-458) Redo hudi-hadoop-mr log statements using SLF4J

[lamber-ken (Jira)]

 [ 
https://issues.apache.org/jira/browse/HUDI-458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

lamber-ken reassigned HUDI-458:
---

Assignee: lamber-ken

> Redo hudi-hadoop-mr log statements using SLF4J
> --
>
> Key: HUDI-458
> URL: https://issues.apache.org/jira/browse/HUDI-458
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>  Components: Code Cleanup
>Reporter: leesf
>Assignee: lamber-ken
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-hudi] lamber-ken commented on issue #1167: [HUDI-484] Fix NPE when reading IncrementalPull.sqltemplate in HiveIncrementalPuller

[GitBox]

lamber-ken commented on issue #1167: [HUDI-484] Fix NPE when reading 
IncrementalPull.sqltemplate in HiveIncrementalPuller
URL: https://github.com/apache/incubator-hudi/pull/1167#issuecomment-570071309
 
 
   > Could we add a unit test for this tool? its historically been not very 
popular.. But better to cover this for the future
   
   Done. 😄 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] lamber-ken opened a new pull request #1169: [MINOR] Fix error usage of String.format

[GitBox]

lamber-ken opened a new pull request #1169: [MINOR] Fix error usage of 
String.format
URL: https://github.com/apache/incubator-hudi/pull/1169
 
 
   ## What is the purpose of the pull request
   
   Fix error usage of String.format.
   
   ## Brief change log
   
 - Modify `$s` to `%s`.
   
   ## Verify this pull request
   
   This pull request is code cleanup without any test coverage.
   
   ## Committer checklist
   
- [ ] Has a corresponding JIRA in PR title & commit

- [ ] Commit message is descriptive of the change

- [ ] CI is green
   
- [ ] Necessary doc changes done or have another open PR
  
- [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] XuQianJin-Stars opened a new pull request #1168: [HUDI-463] Redo hudi-utilities log statements using SLF4J

[GitBox]

XuQianJin-Stars opened a new pull request #1168: [HUDI-463] Redo hudi-utilities 
log statements using SLF4J
URL: https://github.com/apache/incubator-hudi/pull/1168
 
 
   ## What is the purpose of the pull request
   
   Redo hudi-utilities log statements using SLF4J.
   
   ## Brief change log
   
   Modify AnnotationLocation checkstyle rule in checkstyle.xml
   
   ## Verify this pull request
   
   Use existing tests to verify the original module.
   
   ## Committer checklist
   
- [ ] Has a corresponding JIRA in PR title & commit

- [ ] Commit message is descriptive of the change

- [ ] CI is green
   
- [ ] Necessary doc changes done or have another open PR
  
- [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (HUDI-463) Redo hudi-utilities log statements using SLF4J

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HUDI-463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HUDI-463:

Labels: pull-request-available  (was: )

> Redo hudi-utilities log statements using SLF4J
> --
>
> Key: HUDI-463
> URL: https://issues.apache.org/jira/browse/HUDI-463
> Project: Apache Hudi (incubating)
>  Issue Type: Improvement
>  Components: Code Cleanup
>Reporter: leesf
>Assignee: Forward Xu
>Priority: Major
>  Labels: pull-request-available
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)