[GitHub] [hudi] codecov-commenter edited a comment on pull request #2922: [HUDI-1872] Move HoodieFlinkStreamer into hudi-utilities module

2021-05-10 Thread GitBox
codecov-commenter edited a comment on pull request #2922: URL: https://github.com/apache/hudi/pull/2922#issuecomment-834349577 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2922?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] vinothchandar commented on pull request #2740: [HUDI-1055] Remove hardcoded parquet in tests

2021-05-10 Thread GitBox
vinothchandar commented on pull request #2740: URL: https://github.com/apache/hudi/pull/2740#issuecomment-837925932 This is a very valuable cleanup! Thanks @TeRS-K ! Over to you and @yanghua to land when ready. I also renamed that one file and pushed, kicked CI again -- This is an automa

[GitHub] [hudi] vinothchandar commented on a change in pull request #2740: [HUDI-1055] Remove hardcoded parquet in tests

2021-05-10 Thread GitBox
vinothchandar commented on a change in pull request #2740: URL: https://github.com/apache/hudi/pull/2740#discussion_r629866891 ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/DataFileUtils.java ## @@ -0,0 +1,73 @@ +/* + * Licensed to the Apache Software Fou

[GitHub] [hudi] danny0405 commented on pull request #2899: [HUDI-1865] Make embedded time line service singleton

2021-05-10 Thread GitBox
danny0405 commented on pull request #2899: URL: https://github.com/apache/hudi/pull/2899#issuecomment-837889063 > each write task is sending a RPC call to itself? Could you confirm that. Yes, i can confirm this. Each task initialized a write client when it starts so a separate timeli

[GitHub] [hudi] vinothchandar commented on pull request #2899: [HUDI-1865] Make embedded time line service singleton

2021-05-10 Thread GitBox
vinothchandar commented on pull request #2899: URL: https://github.com/apache/hudi/pull/2899#issuecomment-837816065 > each task manager is, sending RPC calls to the timeline server running locally? each write task is sending a RPC call to itself? Could you confirm that. >we s

[GitHub] [hudi] xiarixiaoyao commented on pull request #2720: [HUDI-1719]hive on spark/mr,Incremental query of the mor table, the partition field is incorrect

2021-05-10 Thread GitBox
xiarixiaoyao commented on pull request #2720: URL: https://github.com/apache/hudi/pull/2720#issuecomment-837735008 @vinothchandar i will do it today, thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

[GitHub] [hudi] xiarixiaoyao commented on pull request #2722: [HUDI-1722]hive beeline/spark-sql query specified field on mor table occur NPE

2021-05-10 Thread GitBox
xiarixiaoyao commented on pull request #2722: URL: https://github.com/apache/hudi/pull/2722#issuecomment-837732845 @vinothchandar i will rebase this pr, thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[GitHub] [hudi] garyli1019 merged pull request #2930: [HUDI-1818] Validate and check required option for HoodieTable (Azure…

2021-05-10 Thread GitBox
garyli1019 merged pull request #2930: URL: https://github.com/apache/hudi/pull/2930 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[hudi] branch master updated: [HUDI-1818] Validate required fields for Flink HoodieTable (#2930)

2021-05-10 Thread garyli
This is an automated email from the ASF dual-hosted git repository. garyli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 7a5af80 [HUDI-1818] Validate required fields for

[GitHub] [hudi] pengzhiwei2018 edited a comment on issue #2935: [Support] HoodieFileIndex get a error when there is no partition path in table storage

2021-05-10 Thread GitBox
pengzhiwei2018 edited a comment on issue #2935: URL: https://github.com/apache/hudi/issues/2935#issuecomment-837684290 Hi @shenbinglife , Thanks for your feedback on this question. We have noticed this issue and after https://github.com/apache/hudi/pull/2893 merged, this will be solved.

[GitHub] [hudi] pengzhiwei2018 commented on issue #2935: [Support] HoodieFileIndex get a error when there is no partition path in table storage

2021-05-10 Thread GitBox
pengzhiwei2018 commented on issue #2935: URL: https://github.com/apache/hudi/issues/2935#issuecomment-837684290 Hi @shenbinglife , Thanks for your feedback on this question. We have noticed this issue and after https://github.com/apache/hudi/pull/2893 merge, this will be solved. -- Thi

[GitHub] [hudi] shenbinglife commented on issue #2935: HoodieFileIndex get a error when there is no partition path in table storage

2021-05-10 Thread GitBox
shenbinglife commented on issue #2935: URL: https://github.com/apache/hudi/issues/2935#issuecomment-837680457 I create a issue at https://issues.apache.org/jira/browse/HUDI-1893 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[jira] [Created] (HUDI-1893) HoodieFileIndex get a error when there is no partition path in table storage

2021-05-10 Thread shenbing (Jira)
shenbing created HUDI-1893: -- Summary: HoodieFileIndex get a error when there is no partition path in table storage Key: HUDI-1893 URL: https://issues.apache.org/jira/browse/HUDI-1893 Project: Apache Hudi

[GitHub] [hudi] shenbinglife opened a new issue #2935: HoodieFileIndex get a error when there is no partition path in table storage

2021-05-10 Thread GitBox
shenbinglife opened a new issue #2935: URL: https://github.com/apache/hudi/issues/2935 https://github.com/apache/hudi/blob/42ec7e30d74104a15ff76cb004fe7a0c2649abe6/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala#L319 May it should be : spark

[jira] [Created] (HUDI-1892) NullPointerException when using OverwriteNonDefaultsWithLatestAvroPayload at hudi 0.9.0

2021-05-10 Thread shenbing (Jira)
shenbing created HUDI-1892: -- Summary: NullPointerException when using OverwriteNonDefaultsWithLatestAvroPayload at hudi 0.9.0 Key: HUDI-1892 URL: https://issues.apache.org/jira/browse/HUDI-1892 Project: Apac

[GitHub] [hudi] danny0405 edited a comment on pull request #2899: [HUDI-1865] Make embedded time line service singleton

2021-05-10 Thread GitBox
danny0405 edited a comment on pull request #2899: URL: https://github.com/apache/hudi/pull/2899#issuecomment-837659380 > > each write task write out these records buffer (grouping by file group id) using a write client there. > > Understood. So each task manager is, sending RPC calls

[GitHub] [hudi] danny0405 commented on pull request #2899: [HUDI-1865] Make embedded time line service singleton

2021-05-10 Thread GitBox
danny0405 commented on pull request #2899: URL: https://github.com/apache/hudi/pull/2899#issuecomment-837659380 > > each write task write out these records buffer (grouping by file group id) using a write client there. > > Understood. So each task manager is, sending RPC calls to the

[GitHub] [hudi] pengzhiwei2018 edited a comment on pull request #2645: [HUDI-1659] Basic Implementation Of Spark Sql Support

2021-05-10 Thread GitBox
pengzhiwei2018 edited a comment on pull request #2645: URL: https://github.com/apache/hudi/pull/2645#issuecomment-835134388 Hi @vinothchandar @umehrot2 , The PR has updated, mainly with the follow changes: - Support atomic for CTAS - Support use timestamp type as partition field -

[GitHub] [hudi] pengzhiwei2018 edited a comment on pull request #2645: [HUDI-1659] Basic Implementation Of Spark Sql Support

2021-05-10 Thread GitBox
pengzhiwei2018 edited a comment on pull request #2645: URL: https://github.com/apache/hudi/pull/2645#issuecomment-835134388 Hi @vinothchandar @umehrot2 , The PR has updated, mainly with the follow changes: - Support atomic for CTAS - Support use timestamp type as partition field -

[jira] [Created] (HUDI-1891) Jetty Dependency conflict when upgrade to hive3.1.1 and hadoop3.0.0

2021-05-10 Thread shenbing (Jira)
shenbing created HUDI-1891: -- Summary: Jetty Dependency conflict when upgrade to hive3.1.1 and hadoop3.0.0 Key: HUDI-1891 URL: https://issues.apache.org/jira/browse/HUDI-1891 Project: Apache Hudi Is

[GitHub] [hudi] vinothchandar commented on a change in pull request #2819: [HUDI-1794] Moved static COMMIT_FORMATTER to thread local variable as SimpleDateFormat is not thread safe.

2021-05-10 Thread GitBox
vinothchandar commented on a change in pull request #2819: URL: https://github.com/apache/hudi/pull/2819#discussion_r629798021 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java ## @@ -73,6 +71,16 @@ private static final

[GitHub] [hudi] vinothchandar commented on a change in pull request #2819: [HUDI-1794] Moved static COMMIT_FORMATTER to thread local variable as SimpleDateFormat is not thread safe.

2021-05-10 Thread GitBox
vinothchandar commented on a change in pull request #2819: URL: https://github.com/apache/hudi/pull/2819#discussion_r629798021 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java ## @@ -73,6 +71,16 @@ private static final

[GitHub] [hudi] vinothchandar commented on pull request #2720: [HUDI-1719]hive on spark/mr,Incremental query of the mor table, the partition field is incorrect

2021-05-10 Thread GitBox
vinothchandar commented on pull request #2720: URL: https://github.com/apache/hudi/pull/2720#issuecomment-837638216 @xiarixiaoyao do you intend to add a UT for this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [hudi] vinothchandar commented on pull request #2879: [HUDI-1848] Adding support for HMS for running DDL queries in hive-sy…

2021-05-10 Thread GitBox
vinothchandar commented on pull request #2879: URL: https://github.com/apache/hudi/pull/2879#issuecomment-837634888 @jsbali @satishkotha do we need #2532 to land for this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

[jira] [Updated] (HUDI-1887) Make schema post processor's default as disabled

2021-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1887: - Labels: pull-request-available (was: ) > Make schema post processor's default as disabled > -

[GitHub] [hudi] vinothchandar commented on pull request #2911: [HUDI-1887] Setting default value to false for enabling schema post processor

2021-05-10 Thread GitBox
vinothchandar commented on pull request #2911: URL: https://github.com/apache/hudi/pull/2911#issuecomment-837632064 @nsivabalan are n't there any tests affected by this change? Also do we even need this post processor feature anymore? Should can the entire feature instead of just making it

[jira] [Closed] (HUDI-1890) FlinkCreateHandle and FlinkAppendHandle canWrite should always return true

2021-05-10 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-1890. -- Resolution: Fixed 42ec7e30d74104a15ff76cb004fe7a0c2649abe6 > FlinkCreateHandle and FlinkAppendHandle canWrite s

[hudi] branch master updated (aa398f7 -> 42ec7e3)

2021-05-10 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from aa398f7 [HUDI-1789] Support reading older snapshots (#2809) add 42ec7e3 [HUDI-1890] FlinkCreateHandle and Flin

[GitHub] [hudi] yanghua merged pull request #2933: [HUDI-1890] FlinkCreateHandle and FlinkAppendHandle canWrite should a…

2021-05-10 Thread GitBox
yanghua merged pull request #2933: URL: https://github.com/apache/hudi/pull/2933 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please co

[GitHub] [hudi] nsivabalan commented on a change in pull request #2927: [HUDI-1129] Adding support to ingest records with old schema after table's schema is evolved

2021-05-10 Thread GitBox
nsivabalan commented on a change in pull request #2927: URL: https://github.com/apache/hudi/pull/2927#discussion_r629763845 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java ## @@ -353,6 +360,89 @@ public static boolean isSchemaCompa

[GitHub] [hudi] vinothchandar commented on pull request #2722: [HUDI-1722]hive beeline/spark-sql query specified field on mor table occur NPE

2021-05-10 Thread GitBox
vinothchandar commented on pull request #2722: URL: https://github.com/apache/hudi/pull/2722#issuecomment-837528970 @xiarixiaoyao Could you please rebase this PR . I tried doing this myself, seems tricky -- This is an automated message from the Apache Git Service. To respond to the messa

[GitHub] [hudi] vinothchandar commented on a change in pull request #2722: [HUDI-1722]hive beeline/spark-sql query specified field on mor table occur NPE

2021-05-10 Thread GitBox
vinothchandar commented on a change in pull request #2722: URL: https://github.com/apache/hudi/pull/2722#discussion_r629744627 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java ## @@ -85,12 +85,14 @@ void addProjec

[GitHub] [hudi] vinothchandar commented on pull request #2899: [HUDI-1865] Make embedded time line service singleton

2021-05-10 Thread GitBox
vinothchandar commented on pull request #2899: URL: https://github.com/apache/hudi/pull/2899#issuecomment-837465319 >each write task write out these records buffer (grouping by file group id) using a write client there. Understood. So each task manager is, sending RPC calls to the ti

[jira] [Commented] (HUDI-1723) DFSPathSelector skips files with the same modify date when read up to source limit

2021-05-10 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342178#comment-17342178 ] Vinoth Chandar commented on HUDI-1723: -- yes. [~xushiyan] can we file an umbrella issu

[GitHub] [hudi] xushiyan commented on pull request #2845: [HUDI-1723] Fix path selector listing files with the same mod date

2021-05-10 Thread GitBox
xushiyan commented on pull request #2845: URL: https://github.com/apache/hudi/pull/2845#issuecomment-837449159 @vinothchandar sounds good. i'll add some tests in these 2 days -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

[GitHub] [hudi] vinothchandar commented on pull request #2845: [HUDI-1723] Fix path selector listing files with the same mod date

2021-05-10 Thread GitBox
vinothchandar commented on pull request #2845: URL: https://github.com/apache/hudi/pull/2845#issuecomment-837447578 @xushiyan @nsivabalan This looks like a reasonable fix, if we can add couple tests and land this. That would be great. -- This is an automated message from the Apache Git S

[GitHub] [hudi] vinothchandar commented on a change in pull request #2903: [HUDI-1850] Fixing read of a empty table but with failed write

2021-05-10 Thread GitBox
vinothchandar commented on a change in pull request #2903: URL: https://github.com/apache/hudi/pull/2903#discussion_r629724089 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/DefaultSource.scala ## @@ -105,7 +105,9 @@ class DefaultSource extends R

[hudi] branch master updated (8a48d16 -> aa398f7)

2021-05-10 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 8a48d16 [HUDI-1707] Reduces log level for too verbose messages from info to debug level. (#2714) add aa398f7 [H

[GitHub] [hudi] vinothchandar merged pull request #2809: [HUDI-1789] Support reading older snapshots

2021-05-10 Thread GitBox
vinothchandar merged pull request #2809: URL: https://github.com/apache/hudi/pull/2809 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, ple

[hudi] 01/01: Trying to reprod the schema evolution test break

2021-05-10 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a commit to branch flaky-ci in repository https://gitbox.apache.org/repos/asf/hudi.git commit 12cf04d53eb7f10be51c11888c946f5e88d11486 Author: Vinoth Chandar AuthorDate: Mon May 10 14:18:42 2021 -0700 Trying t

[hudi] branch flaky-ci created (now 12cf04d)

2021-05-10 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a change to branch flaky-ci in repository https://gitbox.apache.org/repos/asf/hudi.git. at 12cf04d Trying to reprod the schema evolution test break This branch includes the following new commits: new

[GitHub] [hudi] n3nash commented on a change in pull request #2927: [HUDI-1129] Adding support to ingest records with old schema after table's schema is evolved

2021-05-10 Thread GitBox
n3nash commented on a change in pull request #2927: URL: https://github.com/apache/hudi/pull/2927#discussion_r629687494 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java ## @@ -429,8 +432,55 @@ public static SchemaProvider createRowBasedSch

[GitHub] [hudi] n3nash commented on a change in pull request #2927: [HUDI-1129] Adding support to ingest records with old schema after table's schema is evolved

2021-05-10 Thread GitBox
n3nash commented on a change in pull request #2927: URL: https://github.com/apache/hudi/pull/2927#discussion_r629687388 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java ## @@ -24,6 +24,9 @@ import org.apache.hudi.client.common.HoodieSparkE

[GitHub] [hudi] n3nash commented on a change in pull request #2927: [HUDI-1129] Adding support to ingest records with old schema after table's schema is evolved

2021-05-10 Thread GitBox
n3nash commented on a change in pull request #2927: URL: https://github.com/apache/hudi/pull/2927#discussion_r629686478 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java ## @@ -429,8 +432,55 @@ public static SchemaProvider createRowBasedSch

[GitHub] [hudi] n3nash commented on a change in pull request #2927: [HUDI-1129] Adding support to ingest records with old schema after table's schema is evolved

2021-05-10 Thread GitBox
n3nash commented on a change in pull request #2927: URL: https://github.com/apache/hudi/pull/2927#discussion_r629685145 ## File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/HoodieSparkSqlWriterSuite.scala ## @@ -483,6 +483,17 @@ class Hoodie

[GitHub] [hudi] n3nash commented on a change in pull request #2927: [HUDI-1129] Adding support to ingest records with old schema after table's schema is evolved

2021-05-10 Thread GitBox
n3nash commented on a change in pull request #2927: URL: https://github.com/apache/hudi/pull/2927#discussion_r629685145 ## File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/HoodieSparkSqlWriterSuite.scala ## @@ -483,6 +483,17 @@ class Hoodie

[GitHub] [hudi] n3nash commented on a change in pull request #2927: [HUDI-1129] Adding support to ingest records with old schema after table's schema is evolved

2021-05-10 Thread GitBox
n3nash commented on a change in pull request #2927: URL: https://github.com/apache/hudi/pull/2927#discussion_r629684633 ## File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkUtils.scala ## @@ -103,4 +106,34 @@ class TestHoodieSparkUtils {

[GitHub] [hudi] n3nash commented on a change in pull request #2927: [HUDI-1129] Adding support to ingest records with old schema after table's schema is evolved

2021-05-10 Thread GitBox
n3nash commented on a change in pull request #2927: URL: https://github.com/apache/hudi/pull/2927#discussion_r629684346 ## File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkUtils.scala ## @@ -103,4 +106,34 @@ class TestHoodieSparkUtils {

[GitHub] [hudi] n3nash commented on a change in pull request #2927: [HUDI-1129] Adding support to ingest records with old schema after table's schema is evolved

2021-05-10 Thread GitBox
n3nash commented on a change in pull request #2927: URL: https://github.com/apache/hudi/pull/2927#discussion_r629683323 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkUtils.scala ## @@ -111,6 +112,34 @@ object HoodieSparkUtils {

[GitHub] [hudi] n3nash commented on a change in pull request #2927: [HUDI-1129] Adding support to ingest records with old schema after table's schema is evolved

2021-05-10 Thread GitBox
n3nash commented on a change in pull request #2927: URL: https://github.com/apache/hudi/pull/2927#discussion_r629682943 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java ## @@ -432,4 +522,16 @@ public static MessageType readSchemaFro

[GitHub] [hudi] n3nash commented on a change in pull request #2927: [HUDI-1129] Adding support to ingest records with old schema after table's schema is evolved

2021-05-10 Thread GitBox
n3nash commented on a change in pull request #2927: URL: https://github.com/apache/hudi/pull/2927#discussion_r629682415 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java ## @@ -353,6 +360,89 @@ public static boolean isSchemaCompatibl

[GitHub] [hudi] n3nash commented on a change in pull request #2927: [HUDI-1129] Adding support to ingest records with old schema after table's schema is evolved

2021-05-10 Thread GitBox
n3nash commented on a change in pull request #2927: URL: https://github.com/apache/hudi/pull/2927#discussion_r629680557 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java ## @@ -353,6 +360,89 @@ public static boolean isSchemaCompatibl

[GitHub] [hudi] n3nash commented on a change in pull request #2927: [HUDI-1129] Adding support to ingest records with old schema after table's schema is evolved

2021-05-10 Thread GitBox
n3nash commented on a change in pull request #2927: URL: https://github.com/apache/hudi/pull/2927#discussion_r629679886 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java ## @@ -353,6 +360,89 @@ public static boolean isSchemaCompatibl

[GitHub] [hudi] n3nash commented on a change in pull request #2927: [HUDI-1129] Adding support to ingest records with old schema after table's schema is evolved

2021-05-10 Thread GitBox
n3nash commented on a change in pull request #2927: URL: https://github.com/apache/hudi/pull/2927#discussion_r629679339 ## File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionHelper.scala ## @@ -139,6 +139,7 @@ object AvroConversionHelper {

[GitHub] [hudi] n3nash commented on a change in pull request #2927: [HUDI-1129] Adding support to ingest records with old schema after table's schema is evolved

2021-05-10 Thread GitBox
n3nash commented on a change in pull request #2927: URL: https://github.com/apache/hudi/pull/2927#discussion_r629679024 ## File path: hudi-utilities/src/test/resources/delta-streamer-config/source.avsc ## @@ -70,7 +70,7 @@ "name" : "height", "type" : { "type" :

[GitHub] [hudi] n3nash commented on a change in pull request #2927: [HUDI-1129] Adding support to ingest records with old schema after table's schema is evolved

2021-05-10 Thread GitBox
n3nash commented on a change in pull request #2927: URL: https://github.com/apache/hudi/pull/2927#discussion_r629678909 ## File path: hudi-utilities/src/test/resources/delta-streamer-config/source_evolved.avsc ## @@ -0,0 +1,136 @@ +/* + * Licensed to the Apache Software Founda

[GitHub] [hudi] satishkotha commented on pull request #2918: [HUDI-1877] Add support in clustering to not change record location

2021-05-10 Thread GitBox
satishkotha commented on pull request #2918: URL: https://github.com/apache/hudi/pull/2918#issuecomment-837283028 > @satishkotha hello , have some doubt > > 1. Just see add a test strategy . Will a formal strategy be added later? > 2. This PR is to support which Index? > 3. If e

[GitHub] [hudi] satishkotha commented on a change in pull request #2918: [HUDI-1877] Add support in clustering to not change record location

2021-05-10 Thread GitBox
satishkotha commented on a change in pull request #2918: URL: https://github.com/apache/hudi/pull/2918#discussion_r629658906 ## File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/ClusteringIdentityTestExecutionStrategy.java ## @@ -0,0 +1,102 @@ +/* + * Lice

[GitHub] [hudi] l-jhon edited a comment on issue #2123: Timestamp not parsed correctly on Athena

2021-05-10 Thread GitBox
l-jhon edited a comment on issue #2123: URL: https://github.com/apache/hudi/issues/2123#issuecomment-837270834 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries a

[GitHub] [hudi] l-jhon commented on issue #2123: Timestamp not parsed correctly on Athena

2021-05-10 Thread GitBox
l-jhon commented on issue #2123: URL: https://github.com/apache/hudi/issues/2123#issuecomment-837270834 > > For example, the output of Dataframe .show is: 2017-05-09 08:21:35, but output at Athena is +49134-01-07 05:30:00.000 > > What is the schema of the table through `DESCRIBE tabl

[GitHub] [hudi] jackye1995 commented on issue #2123: Timestamp not parsed correctly on Athena

2021-05-10 Thread GitBox
jackye1995 commented on issue #2123: URL: https://github.com/apache/hudi/issues/2123#issuecomment-837249804 > For example, the output of Dataframe .show is: 2017-05-09 08:21:35, but output at Athena is +49134-01-07 05:30:00.000 What is the schema of the table through `DESCRIBE table`

[GitHub] [hudi] l-jhon removed a comment on issue #2123: Timestamp not parsed correctly on Athena

2021-05-10 Thread GitBox
l-jhon removed a comment on issue #2123: URL: https://github.com/apache/hudi/issues/2123#issuecomment-837241384 For example, the output of Dataframe .show is: `2017-05-09 08:21:35`, but output at Athena is `+49134-01-07 05:30:00.000` -- This is an automated message from the Apache Git Se

[GitHub] [hudi] l-jhon commented on issue #2123: Timestamp not parsed correctly on Athena

2021-05-10 Thread GitBox
l-jhon commented on issue #2123: URL: https://github.com/apache/hudi/issues/2123#issuecomment-837242269 > @l-jhon could you provide more details on what your problem is? What is the table you create in Athena/Glue, and what query did you run? For example, the output o

[GitHub] [hudi] l-jhon commented on issue #2123: Timestamp not parsed correctly on Athena

2021-05-10 Thread GitBox
l-jhon commented on issue #2123: URL: https://github.com/apache/hudi/issues/2123#issuecomment-837241384 For example, the output of Dataframe .show is: `2017-05-09 08:21:35`, but output at Athena is `+49134-01-07 05:30:00.000` -- This is an automated message from the Apache Git Service. T

[GitHub] [hudi] jackye1995 edited a comment on issue #2123: Timestamp not parsed correctly on Athena

2021-05-10 Thread GitBox
jackye1995 edited a comment on issue #2123: URL: https://github.com/apache/hudi/issues/2123#issuecomment-837234410 @l-jhon could you provide more details on what your problem is? What is the table you create in Athena/Glue, and what query did you run? -- This is an automated message from

[GitHub] [hudi] jackye1995 commented on issue #2123: Timestamp not parsed correctly on Athena

2021-05-10 Thread GitBox
jackye1995 commented on issue #2123: URL: https://github.com/apache/hudi/issues/2123#issuecomment-837234410 @l-jhon could you provide more details on what your problem is? What is the table you create in Athena/Glue, and what query you ran? -- This is an automated message from the Apache

[GitHub] [hudi] l-jhon edited a comment on issue #2123: Timestamp not parsed correctly on Athena

2021-05-10 Thread GitBox
l-jhon edited a comment on issue #2123: URL: https://github.com/apache/hudi/issues/2123#issuecomment-837229546 @jackye1995 Is it necessary to change something to make it work? Because I still have a problem. My code: `spark = SparkSession \` `.builder \` `.appNam

[GitHub] [hudi] l-jhon edited a comment on issue #2123: Timestamp not parsed correctly on Athena

2021-05-10 Thread GitBox
l-jhon edited a comment on issue #2123: URL: https://github.com/apache/hudi/issues/2123#issuecomment-837229546 @jackye1995 Is it necessary to change something to make it work? Because I still have a problem. My code: `spark = SparkSession \ .builder \ .appName("d

[GitHub] [hudi] l-jhon edited a comment on issue #2123: Timestamp not parsed correctly on Athena

2021-05-10 Thread GitBox
l-jhon edited a comment on issue #2123: URL: https://github.com/apache/hudi/issues/2123#issuecomment-837229546 @jackye1995 Is it necessary to change something to make it work? Because I still have a problem. My code: `spark = SparkSession \ .builder \ .appName("d

[GitHub] [hudi] l-jhon edited a comment on issue #2123: Timestamp not parsed correctly on Athena

2021-05-10 Thread GitBox
l-jhon edited a comment on issue #2123: URL: https://github.com/apache/hudi/issues/2123#issuecomment-837229546 @jackye1995 Is it necessary to change something to make it work? Because I still have a problem. My code: `spark = SparkSession \ .builder \ .appName("d

[GitHub] [hudi] l-jhon commented on issue #2123: Timestamp not parsed correctly on Athena

2021-05-10 Thread GitBox
l-jhon commented on issue #2123: URL: https://github.com/apache/hudi/issues/2123#issuecomment-837229546 @jackye1995 Is it necessary to change something to make it work? Because I still have a problem. My code: `spark = SparkSession \ .builder \ .appName("data_ing

[jira] [Commented] (HUDI-1138) Re-implement marker files via timeline server

2021-05-10 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342017#comment-17342017 ] Vinoth Chandar commented on HUDI-1138: -- [~guoyihua] is picking this up and running wi

[jira] [Assigned] (HUDI-1138) Re-implement marker files via timeline server

2021-05-10 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar reassigned HUDI-1138: Assignee: Ethan Guo > Re-implement marker files via timeline server > -

[jira] [Updated] (HUDI-1138) Re-implement marker files via timeline server

2021-05-10 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-1138: - Status: Open (was: New) > Re-implement marker files via timeline server > ---

[GitHub] [hudi] nsivabalan commented on a change in pull request #2932: [WIP] Fixing schema evolution in sparksql_writer and deltastreamer

2021-05-10 Thread GitBox
nsivabalan commented on a change in pull request #2932: URL: https://github.com/apache/hudi/pull/2932#discussion_r629447410 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java ## @@ -374,11 +375,12 @@ public void refreshTimeline()

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2930: [HUDI-1818] Validate and check required option for HoodieTable (Azure…

2021-05-10 Thread GitBox
codecov-commenter edited a comment on pull request #2930: URL: https://github.com/apache/hudi/pull/2930#issuecomment-835670270 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [hudi] nsivabalan commented on a change in pull request #2927: [HUDI-1129][WIP] Adding support to ingest records with old schema after table's schema is evolved

2021-05-10 Thread GitBox
nsivabalan commented on a change in pull request #2927: URL: https://github.com/apache/hudi/pull/2927#discussion_r629421999 ## File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionHelper.scala ## @@ -139,6 +139,7 @@ object AvroConversionHelper

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2930: [HUDI-1818] Validate and check required option for HoodieTable (Azure…

2021-05-10 Thread GitBox
codecov-commenter edited a comment on pull request #2930: URL: https://github.com/apache/hudi/pull/2930#issuecomment-835670270 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2930?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] vinothchandar merged pull request #2714: [HUDI-1707] Reduces log level for too verbose messages from info to debug level.

2021-05-10 Thread GitBox
vinothchandar merged pull request #2714: URL: https://github.com/apache/hudi/pull/2714 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, ple

[hudi] branch master updated (511ac48 -> 8a48d16)

2021-05-10 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 511ac48 [MINOR] optimize FilePathUtils (#2931) add 8a48d16 [HUDI-1707] Reduces log level for too verbose message

[GitHub] [hudi] t0il3ts0ap opened a new issue #2934: [SUPPORT] Parquet file does not exist when trying to read hudi table incrementally

2021-05-10 Thread GitBox
t0il3ts0ap opened a new issue #2934: URL: https://github.com/apache/hudi/issues/2934 **Describe the problem you faced** My aim is to read an existing hudi table (COW) using deltastreamer, do some transformations and write it to another ( fresh ) table. I am using deltastreamer so as

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2933: [HUDI-1890] FlinkCreateHandle and FlinkAppendHandle canWrite should a…

2021-05-10 Thread GitBox
codecov-commenter edited a comment on pull request #2933: URL: https://github.com/apache/hudi/pull/2933#issuecomment-836676096 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2933?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] garyli1019 merged pull request #2931: [MINOR] optimize FilePathUtils

2021-05-10 Thread GitBox
garyli1019 merged pull request #2931: URL: https://github.com/apache/hudi/pull/2931 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[hudi] branch master updated: [MINOR] optimize FilePathUtils (#2931)

2021-05-10 Thread garyli
This is an automated email from the ASF dual-hosted git repository. garyli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 511ac48 [MINOR] optimize FilePathUtils (#2931) 51

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2933: [HUDI-1890] FlinkCreateHandle and FlinkAppendHandle canWrite should a…

2021-05-10 Thread GitBox
codecov-commenter edited a comment on pull request #2933: URL: https://github.com/apache/hudi/pull/2933#issuecomment-836676096 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [hudi] MyLanPangzi commented on a change in pull request #2930: [HUDI-1818] Validate and check required option for HoodieTable (Azure…

2021-05-10 Thread GitBox
MyLanPangzi commented on a change in pull request #2930: URL: https://github.com/apache/hudi/pull/2930#discussion_r629365106 ## File path: hudi-flink/src/test/java/org/apache/hudi/table/TestHoodieTableFactory.java ## @@ -297,4 +343,4 @@ public boolean isTemporary() { re

[GitHub] [hudi] Akshay2Agarwal commented on issue #2913: [SUPPORT] Hudi + Hive Metastore Sync

2021-05-10 Thread GitBox
Akshay2Agarwal commented on issue #2913: URL: https://github.com/apache/hudi/issues/2913#issuecomment-836704935 Closing the ticket, as I read through the code, I realized hudi is integrated with hive2 and queryable in presto as an external table and not as managed table. -- This is an a

[GitHub] [hudi] Akshay2Agarwal closed issue #2913: [SUPPORT] Hudi + Hive Metastore Sync

2021-05-10 Thread GitBox
Akshay2Agarwal closed issue #2913: URL: https://github.com/apache/hudi/issues/2913 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] codecov-commenter commented on pull request #2933: [HUDI-1890] FlinkCreateHandle and FlinkAppendHandle canWrite should a…

2021-05-10 Thread GitBox
codecov-commenter commented on pull request #2933: URL: https://github.com/apache/hudi/pull/2933#issuecomment-836676096 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2933?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache

[GitHub] [hudi] garyli1019 commented on a change in pull request #2930: [HUDI-1818] Validate and check required option for HoodieTable (Azure…

2021-05-10 Thread GitBox
garyli1019 commented on a change in pull request #2930: URL: https://github.com/apache/hudi/pull/2930#discussion_r629312247 ## File path: hudi-flink/src/test/java/org/apache/hudi/table/TestHoodieTableFactory.java ## @@ -297,4 +343,4 @@ public boolean isTemporary() { ret

[jira] [Updated] (HUDI-1890) FlinkCreateHandle and FlinkAppendHandle canWrite should always return true

2021-05-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1890: - Labels: pull-request-available (was: ) > FlinkCreateHandle and FlinkAppendHandle canWrite should

[GitHub] [hudi] danny0405 opened a new pull request #2933: [HUDI-1890] FlinkCreateHandle and FlinkAppendHandle canWrite should a…

2021-05-10 Thread GitBox
danny0405 opened a new pull request #2933: URL: https://github.com/apache/hudi/pull/2933 …lways return true The method #canWrite should always return true because they can already write based on file size, e.g. the BucketAssigner. ## *Tips* - *Thank you very much for cont

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2931: [MINOR] optimize FilePathUtils

2021-05-10 Thread GitBox
codecov-commenter edited a comment on pull request #2931: URL: https://github.com/apache/hudi/pull/2931#issuecomment-835768056 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2930: [HUDI-1818] Validate and check required option for HoodieTable (Azure…

2021-05-10 Thread GitBox
codecov-commenter edited a comment on pull request #2930: URL: https://github.com/apache/hudi/pull/2930#issuecomment-835670270 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2931: [MINOR] optimize FilePathUtils

2021-05-10 Thread GitBox
codecov-commenter edited a comment on pull request #2931: URL: https://github.com/apache/hudi/pull/2931#issuecomment-835768056 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2931?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2930: [HUDI-1818] Validate and check required option for HoodieTable (Azure…

2021-05-10 Thread GitBox
codecov-commenter edited a comment on pull request #2930: URL: https://github.com/apache/hudi/pull/2930#issuecomment-835670270 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2930?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[jira] [Created] (HUDI-1890) FlinkCreateHandle and FlinkAppendHandle canWrite should always return true

2021-05-10 Thread Danny Chen (Jira)
Danny Chen created HUDI-1890: Summary: FlinkCreateHandle and FlinkAppendHandle canWrite should always return true Key: HUDI-1890 URL: https://issues.apache.org/jira/browse/HUDI-1890 Project: Apache Hudi

[GitHub] [hudi] MyLanPangzi commented on a change in pull request #2931: [MINOR] optimize FilePathUtils

2021-05-10 Thread GitBox
MyLanPangzi commented on a change in pull request #2931: URL: https://github.com/apache/hudi/pull/2931#discussion_r629240721 ## File path: hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java ## @@ -260,12 +245,12 @@ private static void listStatusRecursivel

[GitHub] [hudi] MyLanPangzi commented on a change in pull request #2930: [HUDI-1818] Validate and check required option for HoodieTable (Azure…

2021-05-10 Thread GitBox
MyLanPangzi commented on a change in pull request #2930: URL: https://github.com/apache/hudi/pull/2930#discussion_r629237005 ## File path: hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java ## @@ -98,6 +102,33 @@ public String factoryIdentifier() { // U

[GitHub] [hudi] n3nash opened a new pull request #2932: [WIP] Fixing schema evolution in sparksql_writer and deltastreamer

2021-05-10 Thread GitBox
n3nash opened a new pull request #2932: URL: https://github.com/apache/hudi/pull/2932 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull

  1   2   >