[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[ https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149955#comment-16149955 ] Sankar Hariappan commented on HIVE-17100: - Thanks [~anishek] for the extensive review and [~thejas] for the commit! > Improve HS2 operation logs for REPL commands. > - > > Key: HIVE-17100 > URL: https://issues.apache.org/jira/browse/HIVE-17100 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch, > HIVE-17100.03.patch, HIVE-17100.04.patch, HIVE-17100.05.patch, > HIVE-17100.06.patch, HIVE-17100.07.patch, HIVE-17100.08.patch, > HIVE-17100.09.patch, HIVE-17100.10.patch > > > It is necessary to log the progress the replication tasks in a structured > manner as follows. > *+Bootstrap Dump:+* > * At the start of bootstrap dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (BOOTSTRAP) > * (Estimated) Total number of tables/views to dump > * (Estimated) Total number of functions to dump. > * Dump Start Time{color} > * After each table dump, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table dump end time > * Table dump progress. Format is Table sequence no/(Estimated) Total number > of tables and views.{color} > * After each function dump, will add a log as follows > {color:#59afe1}* Function Name > * Function dump end time > * Function dump progress. Format is Function sequence no/(Estimated) Total > number of functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > dump. > {color:#59afe1}* Database Name. > * Dump Type (BOOTSTRAP). > * Dump End Time. > * (Actual) Total number of tables/views dumped. > * (Actual) Total number of functions dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The actual and estimated number of tables/functions may not match if > any table/function is dropped when dump in progress. > *+Bootstrap Load:+* > * At the start of bootstrap load, will add one log with below details. > {color:#59afe1}* Database Name > * Dump directory > * Load Type (BOOTSTRAP) > * Total number of tables/views to load > * Total number of functions to load. > * Load Start Time{color} > * After each table load, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table load completion time > * Table load progress. Format is Table sequence no/Total number of tables and > views.{color} > * After each function load, will add a log as follows > {color:#59afe1}* Function Name > * Function load completion time > * Function load progress. Format is Function sequence no/Total number of > functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > load. > {color:#59afe1}* Database Name. > * Load Type (BOOTSTRAP). > * Load End Time. > * Total number of tables/views loaded. > * Total number of functions loaded. > * Last Repl ID of the loaded database.{color} > *+Incremental Dump:+* > * At the start of database dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (INCREMENTAL) > * (Estimated) Total number of events to dump. > * Dump Start Time{color} > * After each event dump, will add a log as follows > {color:#59afe1}* Event ID > * Event Type (CREATE_TABLE, DROP_TABLE, ALTER_TABLE, INSERT etc) > * Event dump end time > * Event dump progress. Format is Event sequence no/ (Estimated) Total number > of events.{color} > * After completion of all event dumps, will add a log as follows. > {color:#59afe1}* Database Name. > * Dump Type (INCREMENTAL). > * Dump End Time. > * (Actual) Total number of events dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The estimated number of events can be terribly inaccurate with actual > number as we don’t have the number of events upfront until we read from > metastore NotificationEvents table. > *+Incremental Load:+* > * At the start of incremental load, will add one log with below details. > {color:#59afe1}* Target Database Name > * Dump directory > * Load Type (INCREMENTAL) > * Total number of events to load > * Load Start Time{color} > * After each event load, will add a log as follows > {color:#59afe1}* Event ID > * Event Type (CREATE_TABLE, DROP_TABLE, ALTER_TABLE, INSERT etc) > * Event load end time > * Event load progress. Format is Event sequence no/ Total number of > events.{color} > * After completion of all event loads, will add a log as follows to > consolidate the load. > {col
[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[ https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149395#comment-16149395 ] ASF GitHub Bot commented on HIVE-17100: --- Github user sankarh closed the pull request at: https://github.com/apache/hive/pull/231 > Improve HS2 operation logs for REPL commands. > - > > Key: HIVE-17100 > URL: https://issues.apache.org/jira/browse/HIVE-17100 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch, > HIVE-17100.03.patch, HIVE-17100.04.patch, HIVE-17100.05.patch, > HIVE-17100.06.patch, HIVE-17100.07.patch, HIVE-17100.08.patch, > HIVE-17100.09.patch, HIVE-17100.10.patch > > > It is necessary to log the progress the replication tasks in a structured > manner as follows. > *+Bootstrap Dump:+* > * At the start of bootstrap dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (BOOTSTRAP) > * (Estimated) Total number of tables/views to dump > * (Estimated) Total number of functions to dump. > * Dump Start Time{color} > * After each table dump, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table dump end time > * Table dump progress. Format is Table sequence no/(Estimated) Total number > of tables and views.{color} > * After each function dump, will add a log as follows > {color:#59afe1}* Function Name > * Function dump end time > * Function dump progress. Format is Function sequence no/(Estimated) Total > number of functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > dump. > {color:#59afe1}* Database Name. > * Dump Type (BOOTSTRAP). > * Dump End Time. > * (Actual) Total number of tables/views dumped. > * (Actual) Total number of functions dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The actual and estimated number of tables/functions may not match if > any table/function is dropped when dump in progress. > *+Bootstrap Load:+* > * At the start of bootstrap load, will add one log with below details. > {color:#59afe1}* Database Name > * Dump directory > * Load Type (BOOTSTRAP) > * Total number of tables/views to load > * Total number of functions to load. > * Load Start Time{color} > * After each table load, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table load completion time > * Table load progress. Format is Table sequence no/Total number of tables and > views.{color} > * After each function load, will add a log as follows > {color:#59afe1}* Function Name > * Function load completion time > * Function load progress. Format is Function sequence no/Total number of > functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > load. > {color:#59afe1}* Database Name. > * Load Type (BOOTSTRAP). > * Load End Time. > * Total number of tables/views loaded. > * Total number of functions loaded. > * Last Repl ID of the loaded database.{color} > *+Incremental Dump:+* > * At the start of database dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (INCREMENTAL) > * (Estimated) Total number of events to dump. > * Dump Start Time{color} > * After each event dump, will add a log as follows > {color:#59afe1}* Event ID > * Event Type (CREATE_TABLE, DROP_TABLE, ALTER_TABLE, INSERT etc) > * Event dump end time > * Event dump progress. Format is Event sequence no/ (Estimated) Total number > of events.{color} > * After completion of all event dumps, will add a log as follows. > {color:#59afe1}* Database Name. > * Dump Type (INCREMENTAL). > * Dump End Time. > * (Actual) Total number of events dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The estimated number of events can be terribly inaccurate with actual > number as we don’t have the number of events upfront until we read from > metastore NotificationEvents table. > *+Incremental Load:+* > * At the start of incremental load, will add one log with below details. > {color:#59afe1}* Target Database Name > * Dump directory > * Load Type (INCREMENTAL) > * Total number of events to load > * Load Start Time{color} > * After each event load, will add a log as follows > {color:#59afe1}* Event ID > * Event Type (CREATE_TABLE, DROP_TABLE, ALTER_TABLE, INSERT etc) > * Event load end time > * Event load progress. Format is Event sequence no/ Total number of > events.{color} > * After completion of all event loads, will add a log as follows to > consolidate
[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[ https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148247#comment-16148247 ] Thejas M Nair commented on HIVE-17100: -- +1 > Improve HS2 operation logs for REPL commands. > - > > Key: HIVE-17100 > URL: https://issues.apache.org/jira/browse/HIVE-17100 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch, > HIVE-17100.03.patch, HIVE-17100.04.patch, HIVE-17100.05.patch, > HIVE-17100.06.patch, HIVE-17100.07.patch, HIVE-17100.08.patch, > HIVE-17100.09.patch, HIVE-17100.10.patch > > > It is necessary to log the progress the replication tasks in a structured > manner as follows. > *+Bootstrap Dump:+* > * At the start of bootstrap dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (BOOTSTRAP) > * (Estimated) Total number of tables/views to dump > * (Estimated) Total number of functions to dump. > * Dump Start Time{color} > * After each table dump, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table dump end time > * Table dump progress. Format is Table sequence no/(Estimated) Total number > of tables and views.{color} > * After each function dump, will add a log as follows > {color:#59afe1}* Function Name > * Function dump end time > * Function dump progress. Format is Function sequence no/(Estimated) Total > number of functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > dump. > {color:#59afe1}* Database Name. > * Dump Type (BOOTSTRAP). > * Dump End Time. > * (Actual) Total number of tables/views dumped. > * (Actual) Total number of functions dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The actual and estimated number of tables/functions may not match if > any table/function is dropped when dump in progress. > *+Bootstrap Load:+* > * At the start of bootstrap load, will add one log with below details. > {color:#59afe1}* Database Name > * Dump directory > * Load Type (BOOTSTRAP) > * Total number of tables/views to load > * Total number of functions to load. > * Load Start Time{color} > * After each table load, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table load completion time > * Table load progress. Format is Table sequence no/Total number of tables and > views.{color} > * After each function load, will add a log as follows > {color:#59afe1}* Function Name > * Function load completion time > * Function load progress. Format is Function sequence no/Total number of > functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > load. > {color:#59afe1}* Database Name. > * Load Type (BOOTSTRAP). > * Load End Time. > * Total number of tables/views loaded. > * Total number of functions loaded. > * Last Repl ID of the loaded database.{color} > *+Incremental Dump:+* > * At the start of database dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (INCREMENTAL) > * (Estimated) Total number of events to dump. > * Dump Start Time{color} > * After each event dump, will add a log as follows > {color:#59afe1}* Event ID > * Event Type (CREATE_TABLE, DROP_TABLE, ALTER_TABLE, INSERT etc) > * Event dump end time > * Event dump progress. Format is Event sequence no/ (Estimated) Total number > of events.{color} > * After completion of all event dumps, will add a log as follows. > {color:#59afe1}* Database Name. > * Dump Type (INCREMENTAL). > * Dump End Time. > * (Actual) Total number of events dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The estimated number of events can be terribly inaccurate with actual > number as we don’t have the number of events upfront until we read from > metastore NotificationEvents table. > *+Incremental Load:+* > * At the start of incremental load, will add one log with below details. > {color:#59afe1}* Target Database Name > * Dump directory > * Load Type (INCREMENTAL) > * Total number of events to load > * Load Start Time{color} > * After each event load, will add a log as follows > {color:#59afe1}* Event ID > * Event Type (CREATE_TABLE, DROP_TABLE, ALTER_TABLE, INSERT etc) > * Event load end time > * Event load progress. Format is Event sequence no/ Total number of > events.{color} > * After completion of all event loads, will add a log as follows to > consolidate the load. > {color:#59afe1}* Target Database Name. > * Load Type (INCREMENTAL). > * Load End
[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[ https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16147253#comment-16147253 ] Hive QA commented on HIVE-17100: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12884402/HIVE-17100.10.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 11014 tests executed *Failed tests:* {noformat} TestTxnCommandsBase - did not produce a TEST-*.xml file (likely timed out) (batchId=280) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=61) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6602/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6602/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6602/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12884402 - PreCommit-HIVE-Build > Improve HS2 operation logs for REPL commands. > - > > Key: HIVE-17100 > URL: https://issues.apache.org/jira/browse/HIVE-17100 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch, > HIVE-17100.03.patch, HIVE-17100.04.patch, HIVE-17100.05.patch, > HIVE-17100.06.patch, HIVE-17100.07.patch, HIVE-17100.08.patch, > HIVE-17100.09.patch, HIVE-17100.10.patch > > > It is necessary to log the progress the replication tasks in a structured > manner as follows. > *+Bootstrap Dump:+* > * At the start of bootstrap dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (BOOTSTRAP) > * (Estimated) Total number of tables/views to dump > * (Estimated) Total number of functions to dump. > * Dump Start Time{color} > * After each table dump, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table dump end time > * Table dump progress. Format is Table sequence no/(Estimated) Total number > of tables and views.{color} > * After each function dump, will add a log as follows > {color:#59afe1}* Function Name > * Function dump end time > * Function dump progress. Format is Function sequence no/(Estimated) Total > number of functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > dump. > {color:#59afe1}* Database Name. > * Dump Type (BOOTSTRAP). > * Dump End Time. > * (Actual) Total number of tables/views dumped. > * (Actual) Total number of functions dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The actual and estimated number of tables/functions may not match if > any table/function is dropped when dump in progress. > *+Bootstrap Load:+* > * At the start of bootstrap load, will add one log with below details. > {color:#59afe1}* Database Name > * Dump directory > * Load Type (BOOTSTRAP) > * Total number of tables/views to load > * Total number of functions to load. > * Load Start Time{color} > * After each table load, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table load completion time > * Table load progress. Format is Table sequence no/Total number of tables and > views.{color} > * After each function load, will add a log as follows > {color:#59afe1}* Function Name > * Function load completion time > * Function load progress. Format is Function sequence no/Total number of > functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > load. > {color:#59afe1}* Database Name. > * Load Type (BOOTSTRAP). > * Load End Time. > * Total number of tables/views loaded. > * Total number of functions loaded. > * Last Repl ID of the loaded database.{color} > *+Incremental Dump:+* > * At the start of database dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (INCREMENTAL) > * (Estimated) Total number of events to dump. > * Dump Start Time{color} > * After each event dump, will add a log as foll
[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[ https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16146548#comment-16146548 ] anishek commented on HIVE-17100: +1 only issue left is the dependency logger added to root, which i think you are already working on . > Improve HS2 operation logs for REPL commands. > - > > Key: HIVE-17100 > URL: https://issues.apache.org/jira/browse/HIVE-17100 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch, > HIVE-17100.03.patch, HIVE-17100.04.patch, HIVE-17100.05.patch, > HIVE-17100.06.patch, HIVE-17100.07.patch, HIVE-17100.08.patch, > HIVE-17100.09.patch > > > It is necessary to log the progress the replication tasks in a structured > manner as follows. > *+Bootstrap Dump:+* > * At the start of bootstrap dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (BOOTSTRAP) > * (Estimated) Total number of tables/views to dump > * (Estimated) Total number of functions to dump. > * Dump Start Time{color} > * After each table dump, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table dump end time > * Table dump progress. Format is Table sequence no/(Estimated) Total number > of tables and views.{color} > * After each function dump, will add a log as follows > {color:#59afe1}* Function Name > * Function dump end time > * Function dump progress. Format is Function sequence no/(Estimated) Total > number of functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > dump. > {color:#59afe1}* Database Name. > * Dump Type (BOOTSTRAP). > * Dump End Time. > * (Actual) Total number of tables/views dumped. > * (Actual) Total number of functions dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The actual and estimated number of tables/functions may not match if > any table/function is dropped when dump in progress. > *+Bootstrap Load:+* > * At the start of bootstrap load, will add one log with below details. > {color:#59afe1}* Database Name > * Dump directory > * Load Type (BOOTSTRAP) > * Total number of tables/views to load > * Total number of functions to load. > * Load Start Time{color} > * After each table load, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table load completion time > * Table load progress. Format is Table sequence no/Total number of tables and > views.{color} > * After each function load, will add a log as follows > {color:#59afe1}* Function Name > * Function load completion time > * Function load progress. Format is Function sequence no/Total number of > functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > load. > {color:#59afe1}* Database Name. > * Load Type (BOOTSTRAP). > * Load End Time. > * Total number of tables/views loaded. > * Total number of functions loaded. > * Last Repl ID of the loaded database.{color} > *+Incremental Dump:+* > * At the start of database dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (INCREMENTAL) > * (Estimated) Total number of events to dump. > * Dump Start Time{color} > * After each event dump, will add a log as follows > {color:#59afe1}* Event ID > * Event Type (CREATE_TABLE, DROP_TABLE, ALTER_TABLE, INSERT etc) > * Event dump end time > * Event dump progress. Format is Event sequence no/ (Estimated) Total number > of events.{color} > * After completion of all event dumps, will add a log as follows. > {color:#59afe1}* Database Name. > * Dump Type (INCREMENTAL). > * Dump End Time. > * (Actual) Total number of events dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The estimated number of events can be terribly inaccurate with actual > number as we don’t have the number of events upfront until we read from > metastore NotificationEvents table. > *+Incremental Load:+* > * At the start of incremental load, will add one log with below details. > {color:#59afe1}* Target Database Name > * Dump directory > * Load Type (INCREMENTAL) > * Total number of events to load > * Load Start Time{color} > * After each event load, will add a log as follows > {color:#59afe1}* Event ID > * Event Type (CREATE_TABLE, DROP_TABLE, ALTER_TABLE, INSERT etc) > * Event load end time > * Event load progress. Format is Event sequence no/ Total number of > events.{color} > * After completion of all event loads, will add a log as follows to > consolidate the load. > {color:#59afe
[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[ https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16145085#comment-16145085 ] Hive QA commented on HIVE-17100: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12884198/HIVE-17100.09.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 11014 tests executed *Failed tests:* {noformat} TestTxnCommandsBase - did not produce a TEST-*.xml file (likely timed out) (batchId=280) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=61) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=100) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234) org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testDynamicPartitionInsert (batchId=282) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6577/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6577/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6577/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12884198 - PreCommit-HIVE-Build > Improve HS2 operation logs for REPL commands. > - > > Key: HIVE-17100 > URL: https://issues.apache.org/jira/browse/HIVE-17100 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch, > HIVE-17100.03.patch, HIVE-17100.04.patch, HIVE-17100.05.patch, > HIVE-17100.06.patch, HIVE-17100.07.patch, HIVE-17100.08.patch, > HIVE-17100.09.patch > > > It is necessary to log the progress the replication tasks in a structured > manner as follows. > *+Bootstrap Dump:+* > * At the start of bootstrap dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (BOOTSTRAP) > * (Estimated) Total number of tables/views to dump > * (Estimated) Total number of functions to dump. > * Dump Start Time{color} > * After each table dump, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table dump end time > * Table dump progress. Format is Table sequence no/(Estimated) Total number > of tables and views.{color} > * After each function dump, will add a log as follows > {color:#59afe1}* Function Name > * Function dump end time > * Function dump progress. Format is Function sequence no/(Estimated) Total > number of functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > dump. > {color:#59afe1}* Database Name. > * Dump Type (BOOTSTRAP). > * Dump End Time. > * (Actual) Total number of tables/views dumped. > * (Actual) Total number of functions dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The actual and estimated number of tables/functions may not match if > any table/function is dropped when dump in progress. > *+Bootstrap Load:+* > * At the start of bootstrap load, will add one log with below details. > {color:#59afe1}* Database Name > * Dump directory > * Load Type (BOOTSTRAP) > * Total number of tables/views to load > * Total number of functions to load. > * Load Start Time{color} > * After each table load, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table load completion time > * Table load progress. Format is Table sequence no/Total number of tables and > views.{color} > * After each function load, will add a log as follows > {color:#59afe1}* Function Name > * Function load completion time > * Function load progress. Format is Function sequence no/Total number of > functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > load. > {color:#59afe1}* Database Name. > * Load Type (BOOTSTRAP). > * Load End Time. > * Total number of tables/views loaded. > * Total number of functions loaded. > * Last Repl ID of the loaded database.{color} > *+Incremental Dump:+* > * At the start of database dump, will add one log with below details. > {color:#59afe1}
[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[ https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142609#comment-16142609 ] Hive QA commented on HIVE-17100: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12883838/HIVE-17100.08.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 11005 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=61) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=235) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6552/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6552/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6552/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12883838 - PreCommit-HIVE-Build > Improve HS2 operation logs for REPL commands. > - > > Key: HIVE-17100 > URL: https://issues.apache.org/jira/browse/HIVE-17100 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch, > HIVE-17100.03.patch, HIVE-17100.04.patch, HIVE-17100.05.patch, > HIVE-17100.06.patch, HIVE-17100.07.patch, HIVE-17100.08.patch > > > It is necessary to log the progress the replication tasks in a structured > manner as follows. > *+Bootstrap Dump:+* > * At the start of bootstrap dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (BOOTSTRAP) > * (Estimated) Total number of tables/views to dump > * (Estimated) Total number of functions to dump. > * Dump Start Time{color} > * After each table dump, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table dump end time > * Table dump progress. Format is Table sequence no/(Estimated) Total number > of tables and views.{color} > * After each function dump, will add a log as follows > {color:#59afe1}* Function Name > * Function dump end time > * Function dump progress. Format is Function sequence no/(Estimated) Total > number of functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > dump. > {color:#59afe1}* Database Name. > * Dump Type (BOOTSTRAP). > * Dump End Time. > * (Actual) Total number of tables/views dumped. > * (Actual) Total number of functions dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The actual and estimated number of tables/functions may not match if > any table/function is dropped when dump in progress. > *+Bootstrap Load:+* > * At the start of bootstrap load, will add one log with below details. > {color:#59afe1}* Database Name > * Dump directory > * Load Type (BOOTSTRAP) > * Total number of tables/views to load > * Total number of functions to load. > * Load Start Time{color} > * After each table load, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table load completion time > * Table load progress. Format is Table sequence no/Total number of tables and > views.{color} > * After each function load, will add a log as follows > {color:#59afe1}* Function Name > * Function load completion time > * Function load progress. Format is Function sequence no/Total number of > functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > load. > {color:#59afe1}* Database Name. > * Load Type (BOOTSTRAP). > * Load End Time. > * Total number of tables/views loaded. > * Total number of functions loaded. > * Last Repl ID of the loaded database.{color} > *+Incremental Dump:+* > * At the start of database dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (INCREMENTAL) > * (Estimated) Total number of events to dump. > * Dump Start Time{color} > * After each event dump, will add a log as follows > {color:#59afe1}* Event ID > * Event Type (CREA
[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[ https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139959#comment-16139959 ] Hive QA commented on HIVE-17100: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12883460/HIVE-17100.07.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10993 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype] (batchId=159) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235) org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver (batchId=242) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation (batchId=180) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6516/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6516/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6516/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12883460 - PreCommit-HIVE-Build > Improve HS2 operation logs for REPL commands. > - > > Key: HIVE-17100 > URL: https://issues.apache.org/jira/browse/HIVE-17100 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch, > HIVE-17100.03.patch, HIVE-17100.04.patch, HIVE-17100.05.patch, > HIVE-17100.06.patch, HIVE-17100.07.patch > > > It is necessary to log the progress the replication tasks in a structured > manner as follows. > *+Bootstrap Dump:+* > * At the start of bootstrap dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (BOOTSTRAP) > * (Estimated) Total number of tables/views to dump > * (Estimated) Total number of functions to dump. > * Dump Start Time{color} > * After each table dump, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table dump end time > * Table dump progress. Format is Table sequence no/(Estimated) Total number > of tables and views.{color} > * After each function dump, will add a log as follows > {color:#59afe1}* Function Name > * Function dump end time > * Function dump progress. Format is Function sequence no/(Estimated) Total > number of functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > dump. > {color:#59afe1}* Database Name. > * Dump Type (BOOTSTRAP). > * Dump End Time. > * (Actual) Total number of tables/views dumped. > * (Actual) Total number of functions dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The actual and estimated number of tables/functions may not match if > any table/function is dropped when dump in progress. > *+Bootstrap Load:+* > * At the start of bootstrap load, will add one log with below details. > {color:#59afe1}* Database Name > * Dump directory > * Load Type (BOOTSTRAP) > * Total number of tables/views to load > * Total number of functions to load. > * Load Start Time{color} > * After each table load, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table load completion time > * Table load progress. Format is Table sequence no/Total number of tables and > views.{color} > * After each function load, will add a log as follows > {color:#59afe1}* Function Name > * Function load completion time > * Function load progress. Format is Function sequence no/Total number of > functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > load. > {color:#59afe1}* Database Name. > * Load Type (BOOTSTRAP). > * Load End Time. > * Total number of tables/views loaded. > * Total number of functions loaded. > * Last Repl ID of the loaded database.{color} > *+Incremental Dump:
[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[ https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139775#comment-16139775 ] Hive QA commented on HIVE-17100: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12883460/HIVE-17100.07.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10999 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=100) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=235) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation (batchId=180) org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout (batchId=228) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6511/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6511/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6511/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12883460 - PreCommit-HIVE-Build > Improve HS2 operation logs for REPL commands. > - > > Key: HIVE-17100 > URL: https://issues.apache.org/jira/browse/HIVE-17100 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch, > HIVE-17100.03.patch, HIVE-17100.04.patch, HIVE-17100.05.patch, > HIVE-17100.06.patch, HIVE-17100.07.patch > > > It is necessary to log the progress the replication tasks in a structured > manner as follows. > *+Bootstrap Dump:+* > * At the start of bootstrap dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (BOOTSTRAP) > * (Estimated) Total number of tables/views to dump > * (Estimated) Total number of functions to dump. > * Dump Start Time{color} > * After each table dump, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table dump end time > * Table dump progress. Format is Table sequence no/(Estimated) Total number > of tables and views.{color} > * After each function dump, will add a log as follows > {color:#59afe1}* Function Name > * Function dump end time > * Function dump progress. Format is Function sequence no/(Estimated) Total > number of functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > dump. > {color:#59afe1}* Database Name. > * Dump Type (BOOTSTRAP). > * Dump End Time. > * (Actual) Total number of tables/views dumped. > * (Actual) Total number of functions dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The actual and estimated number of tables/functions may not match if > any table/function is dropped when dump in progress. > *+Bootstrap Load:+* > * At the start of bootstrap load, will add one log with below details. > {color:#59afe1}* Database Name > * Dump directory > * Load Type (BOOTSTRAP) > * Total number of tables/views to load > * Total number of functions to load. > * Load Start Time{color} > * After each table load, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table load completion time > * Table load progress. Format is Table sequence no/Total number of tables and > views.{color} > * After each function load, will add a log as follows > {color:#59afe1}* Function Name > * Function load completion time > * Function load progress. Format is Function sequence no/Total number of > functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > load. > {color:#59afe1}* Database Name. > * Load Type (BOOTSTRAP). > * Load End Time. > * Total number of tables/views loaded. > * Total number of functions loaded. > * Last Repl ID of the loaded dat
[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[ https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136485#comment-16136485 ] Hive QA commented on HIVE-17100: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12882974/HIVE-17100.05.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10994 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235) org.apache.hadoop.hive.common.TestFileUtils.testCopyWithDistCpAs (batchId=250) org.apache.hadoop.hive.common.TestFileUtils.testCopyWithDistcp (batchId=250) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation (batchId=180) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6479/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6479/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6479/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12882974 - PreCommit-HIVE-Build > Improve HS2 operation logs for REPL commands. > - > > Key: HIVE-17100 > URL: https://issues.apache.org/jira/browse/HIVE-17100 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch, > HIVE-17100.03.patch, HIVE-17100.04.patch, HIVE-17100.05.patch > > > It is necessary to log the progress the replication tasks in a structured > manner as follows. > *+Bootstrap Dump:+* > * At the start of bootstrap dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (BOOTSTRAP) > * (Estimated) Total number of tables/views to dump > * (Estimated) Total number of functions to dump. > * Dump Start Time{color} > * After each table dump, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table dump end time > * Table dump progress. Format is Table sequence no/(Estimated) Total number > of tables and views.{color} > * After each function dump, will add a log as follows > {color:#59afe1}* Function Name > * Function dump end time > * Function dump progress. Format is Function sequence no/(Estimated) Total > number of functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > dump. > {color:#59afe1}* Database Name. > * Dump Type (BOOTSTRAP). > * Dump End Time. > * (Actual) Total number of tables/views dumped. > * (Actual) Total number of functions dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The actual and estimated number of tables/functions may not match if > any table/function is dropped when dump in progress. > *+Bootstrap Load:+* > * At the start of bootstrap load, will add one log with below details. > {color:#59afe1}* Database Name > * Dump directory > * Load Type (BOOTSTRAP) > * Total number of tables/views to load > * Total number of functions to load. > * Load Start Time{color} > * After each table load, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table load completion time > * Table load progress. Format is Table sequence no/Total number of tables and > views.{color} > * After each function load, will add a log as follows > {color:#59afe1}* Function Name > * Function load completion time > * Function load progress. Format is Function sequence no/Total number of > functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > load. > {color:#59afe1}* Database Name. > * Load Type (BOOTSTRAP). > * Load End Time. > * Total number of tables/views loaded. > * Total number of functions loaded. > * Last Repl ID of the loaded database.{color} > *+Incremental Dump:+* > * At the start of database dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type
[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[ https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130313#comment-16130313 ] Hive QA commented on HIVE-17100: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12882306/HIVE-17100.05.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10977 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only] (batchId=170) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=235) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation (batchId=180) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6435/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6435/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6435/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12882306 - PreCommit-HIVE-Build > Improve HS2 operation logs for REPL commands. > - > > Key: HIVE-17100 > URL: https://issues.apache.org/jira/browse/HIVE-17100 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch, > HIVE-17100.03.patch, HIVE-17100.04.patch, HIVE-17100.05.patch > > > It is necessary to log the progress the replication tasks in a structured > manner as follows. > *+Bootstrap Dump:+* > * At the start of bootstrap dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (BOOTSTRAP) > * (Estimated) Total number of tables/views to dump > * (Estimated) Total number of functions to dump. > * Dump Start Time{color} > * After each table dump, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table dump end time > * Table dump progress. Format is Table sequence no/(Estimated) Total number > of tables and views.{color} > * After each function dump, will add a log as follows > {color:#59afe1}* Function Name > * Function dump end time > * Function dump progress. Format is Function sequence no/(Estimated) Total > number of functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > dump. > {color:#59afe1}* Database Name. > * Dump Type (BOOTSTRAP). > * Dump End Time. > * (Actual) Total number of tables/views dumped. > * (Actual) Total number of functions dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The actual and estimated number of tables/functions may not match if > any table/function is dropped when dump in progress. > *+Bootstrap Load:+* > * At the start of bootstrap load, will add one log with below details. > {color:#59afe1}* Database Name > * Dump directory > * Load Type (BOOTSTRAP) > * Total number of tables/views to load > * Total number of functions to load. > * Load Start Time{color} > * After each table load, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table load completion time > * Table load progress. Format is Table sequence no/Total number of tables and > views.{color} > * After each function load, will add a log as follows > {color:#59afe1}* Function Name > * Function load completion time > * Function load progress. Format is Function sequence no/Total number of > functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > load. > {color:#59afe1}* Database Name. > * Load Type (BOOTSTRAP). > * Load End Time. > * Total number of tables/views loaded. > * Total number of functions loaded. > * Last Repl
[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[ https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130081#comment-16130081 ] Hive QA commented on HIVE-17100: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12882282/HIVE-17100.05.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6432/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6432/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6432/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Tests exited with: ExecutionException: java.util.concurrent.ExecutionException: org.apache.hive.ptest.execution.ssh.SSHExecutionException: RSyncResult [localFile=/data/hiveptest/logs/PreCommit-HIVE-Build-6432/succeeded/33-TestCliDriver-ql_rewrite_gbtoidx.q-json_serde1.q-constantPropWhen.q-and-27-more, remoteFile=/home/hiveptest/104.155.155.39-hiveptest-1/logs/, getExitCode()=255, getException()=null, getUser()=hiveptest, getHost()=104.155.155.39, getInstance()=1]: 'Warning: Permanently added '104.155.155.39' (ECDSA) to the list of known hosts. receiving incremental file list ./ TEST-33-TestCliDriver-ql_rewrite_gbtoidx.q-json_serde1.q-constantPropWhen.q-and-27-more-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml 0 0%0.00kB/s0:00:00 8,686 100%8.28MB/s0:00:00 (xfr#1, to-chk=5/7) maven-test.txt 0 0%0.00kB/s0:00:00 48,240 100%1.35MB/s0:00:00 (xfr#2, to-chk=4/7) logs/ logs/derby.log 0 0%0.00kB/s0:00:00 1,004 100% 28.84kB/s0:00:00 (xfr#3, to-chk=1/7) logs/hive.log 0 0%0.00kB/s0:00:00 48,103,424 2% 45.74MB/s0:00:34 107,479,040 6% 51.05MB/s0:00:30 165,445,632 9% 52.45MB/s0:00:28 225,116,160 13% 53.44MB/s0:00:26 284,950,528 16% 56.29MB/s0:00:24 343,638,016 20% 56.21MB/s0:00:23 402,194,432 23% 56.36MB/s0:00:22 461,766,656 27% 56.35MB/s0:00:21 521,437,184 31% 56.19MB/s0:00:20 578,584,576 34% 55.83MB/s0:00:19 637,534,208 37% 55.91MB/s0:00:18 696,745,984 41% 55.94MB/s0:00:17 755,859,456 44% 55.90MB/s0:00:16 814,743,552 48% 56.32MB/s0:00:15 873,857,024 51% 56.37MB/s0:00:13 933,265,408 55% 56.31MB/s0:00:12 993,099,776 59% 56.34MB/s0:00:11 1,052,639,232 62% 56.38MB/s0:00:10 1,112,211,456 66% 56.36MB/s0:00:09 1,172,144,128 69% 56.46MB/s0:00:08 1,232,011,264 73% 56.49MB/s0:00:07 1,291,583,488 76% 56.50MB/s0:00:06 1,351,581,696 80% 56.56MB/s0:00:05 1,411,383,296 83% 56.56MB/s0:00:04 1,470,857,216 87% 56.48MB/s0:00:03 1,530,396,672 90% 56.47MB/s0:00:02 1,590,296,576 94% 56.46MB/s0:00:01 Timeout, server 104.155.155.39 not responding. rsync: connection unexpectedly closed (1621310331 bytes received so far) [receiver] rsync error: error in rsync protocol data stream (code 12) at io.c(226) [receiver=3.1.1] rsync: connection unexpectedly closed (447 bytes received so far) [generator] rsync error: unexplained error (code 255) at io.c(226) [generator=3.1.1] ssh: connect to host 104.155.155.39 port 22: Connection timed out rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1] ssh: connect to host 104.155.155.39 port 22: Connection timed out rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1] ssh: connect to host 104.155.155.39 port 22: Connection timed out rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1] ssh: connect to host 104.155.155.39 port 22: Connection timed out rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1] ' {noformat} This message is automatically generated. ATTACHMENT ID: 12882282 - PreCommit-HIVE-Build > Improve HS2 operation logs for REPL commands. > - > > Key: HIVE-17100 > URL: https://issues.apache.org/jira/browse/HIVE-17100 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariapp
[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[ https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129712#comment-16129712 ] Hive QA commented on HIVE-17100: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12882203/HIVE-17100.04.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10977 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype] (batchId=159) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only] (batchId=170) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=100) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=235) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation (batchId=180) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6429/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6429/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6429/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12882203 - PreCommit-HIVE-Build > Improve HS2 operation logs for REPL commands. > - > > Key: HIVE-17100 > URL: https://issues.apache.org/jira/browse/HIVE-17100 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch, > HIVE-17100.03.patch, HIVE-17100.04.patch > > > It is necessary to log the progress the replication tasks in a structured > manner as follows. > *+Bootstrap Dump:+* > * At the start of bootstrap dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (BOOTSTRAP) > * (Estimated) Total number of tables/views to dump > * (Estimated) Total number of functions to dump. > * Dump Start Time{color} > * After each table dump, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table dump end time > * Table dump progress. Format is Table sequence no/(Estimated) Total number > of tables and views.{color} > * After each function dump, will add a log as follows > {color:#59afe1}* Function Name > * Function dump end time > * Function dump progress. Format is Function sequence no/(Estimated) Total > number of functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > dump. > {color:#59afe1}* Database Name. > * Dump Type (BOOTSTRAP). > * Dump End Time. > * (Actual) Total number of tables/views dumped. > * (Actual) Total number of functions dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The actual and estimated number of tables/functions may not match if > any table/function is dropped when dump in progress. > *+Bootstrap Load:+* > * At the start of bootstrap load, will add one log with below details. > {color:#59afe1}* Database Name > * Dump directory > * Load Type (BOOTSTRAP) > * Total number of tables/views to load > * Total number of functions to load. > * Load Start Time{color} > * After each table load, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table load completion time > * Table load progress. Format is Table sequence no/Total number of tables and > views.{color} > * After each function load, will add a log as follows > {color:#59afe1}* Function Name > * Function load completion time > * Function load progress. Format is Function sequence no/Total number of > functions.{color} > * After completion of all dumps, will add a log as follows to consolidate t
[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[ https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128494#comment-16128494 ] Hive QA commented on HIVE-17100: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12882080/HIVE-17100.03.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 10976 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed] (batchId=240) org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite] (batchId=240) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_varchar_simple] (batchId=73) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype] (batchId=159) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only] (batchId=170) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=235) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation (batchId=180) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6417/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6417/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6417/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 12 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12882080 - PreCommit-HIVE-Build > Improve HS2 operation logs for REPL commands. > - > > Key: HIVE-17100 > URL: https://issues.apache.org/jira/browse/HIVE-17100 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch, > HIVE-17100.03.patch > > > It is necessary to log the progress the replication tasks in a structured > manner as follows. > *+Bootstrap Dump:+* > * At the start of bootstrap dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (BOOTSTRAP) > * (Estimated) Total number of tables/views to dump > * (Estimated) Total number of functions to dump. > * Dump Start Time{color} > * After each table dump, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table dump end time > * Table dump progress. Format is Table sequence no/(Estimated) Total number > of tables and views.{color} > * After each function dump, will add a log as follows > {color:#59afe1}* Function Name > * Function dump end time > * Function dump progress. Format is Function sequence no/(Estimated) Total > number of functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > dump. > {color:#59afe1}* Database Name. > * Dump Type (BOOTSTRAP). > * Dump End Time. > * (Actual) Total number of tables/views dumped. > * (Actual) Total number of functions dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The actual and estimated number of tables/functions may not match if > any table/function is dropped when dump in progress. > *+Bootstrap Load:+* > * At the start of bootstrap load, will add one log with below details. > {color:#59afe1}* Database Name > * Dump directory > * Load Type (BOOTSTRAP) > * Total number of tables/views to load > * Total number of functions to load. > * Load Start Time{color} > * After each table load, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table load completion time > * Table load progress. Format is Table sequence no/Total number of tables and > views.{color} > * After each function load, will add a log as follows > {color:#59afe1}* Function Name > * Function load comple
[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[ https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128449#comment-16128449 ] Hive QA commented on HIVE-17100: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12882080/HIVE-17100.03.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10976 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only] (batchId=170) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=235) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation (batchId=180) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6416/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6416/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6416/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12882080 - PreCommit-HIVE-Build > Improve HS2 operation logs for REPL commands. > - > > Key: HIVE-17100 > URL: https://issues.apache.org/jira/browse/HIVE-17100 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch, > HIVE-17100.03.patch > > > It is necessary to log the progress the replication tasks in a structured > manner as follows. > *+Bootstrap Dump:+* > * At the start of bootstrap dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (BOOTSTRAP) > * (Estimated) Total number of tables/views to dump > * (Estimated) Total number of functions to dump. > * Dump Start Time{color} > * After each table dump, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table dump end time > * Table dump progress. Format is Table sequence no/(Estimated) Total number > of tables and views.{color} > * After each function dump, will add a log as follows > {color:#59afe1}* Function Name > * Function dump end time > * Function dump progress. Format is Function sequence no/(Estimated) Total > number of functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > dump. > {color:#59afe1}* Database Name. > * Dump Type (BOOTSTRAP). > * Dump End Time. > * (Actual) Total number of tables/views dumped. > * (Actual) Total number of functions dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The actual and estimated number of tables/functions may not match if > any table/function is dropped when dump in progress. > *+Bootstrap Load:+* > * At the start of bootstrap load, will add one log with below details. > {color:#59afe1}* Database Name > * Dump directory > * Load Type (BOOTSTRAP) > * Total number of tables/views to load > * Total number of functions to load. > * Load Start Time{color} > * After each table load, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table load completion time > * Table load progress. Format is Table sequence no/Total number of tables and > views.{color} > * After each function load, will add a log as follows > {color:#59afe1}* Function Name > * Function load completion time > * Function load progress. Format is Function sequence no/Total number of > functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > load. > {color:#59afe1}* Database Name. > * Load Type (BOOTSTRAP). > * Load End Time. > * Total number of tables/views loaded. > * Total number of functions loaded. > * Last Repl ID of the loaded database.{color} > *+Incr
[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[ https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16126896#comment-16126896 ] Hive QA commented on HIVE-17100: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12881874/HIVE-17100.02.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11004 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed] (batchId=240) org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite] (batchId=240) org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_merge_move] (batchId=243) org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_merge_only] (batchId=243) org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_move_only] (batchId=243) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[char_udf1] (batchId=86) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only] (batchId=170) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation (batchId=180) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6396/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6396/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6396/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 12 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12881874 - PreCommit-HIVE-Build > Improve HS2 operation logs for REPL commands. > - > > Key: HIVE-17100 > URL: https://issues.apache.org/jira/browse/HIVE-17100 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17100.01.patch, HIVE-17100.02.patch > > > It is necessary to log the progress the replication tasks in a structured > manner as follows. > *+Bootstrap Dump:+* > * At the start of bootstrap dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (BOOTSTRAP) > * (Estimated) Total number of tables/views to dump > * (Estimated) Total number of functions to dump. > * Dump Start Time{color} > * After each table dump, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table dump end time > * Table dump progress. Format is Table sequence no/(Estimated) Total number > of tables and views.{color} > * After each function dump, will add a log as follows > {color:#59afe1}* Function Name > * Function dump end time > * Function dump progress. Format is Function sequence no/(Estimated) Total > number of functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > dump. > {color:#59afe1}* Database Name. > * Dump Type (BOOTSTRAP). > * Dump End Time. > * (Actual) Total number of tables/views dumped. > * (Actual) Total number of functions dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The actual and estimated number of tables/functions may not match if > any table/function is dropped when dump in progress. > *+Bootstrap Load:+* > * At the start of bootstrap load, will add one log with below details. > {color:#59afe1}* Database Name > * Dump directory > * Load Type (BOOTSTRAP) > * Total number of tables/views to load > * Total number of functions to load. > * Load Start Time{color} > * After each table load, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table load completion time > * Table load progress. Format is Table sequence no/Total number of tables and > views.{color} > * After each function load, will add a log
[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[ https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16126643#comment-16126643 ] Hive QA commented on HIVE-17100: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12881838/HIVE-17100.01.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6391/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6391/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6391/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-08-14 23:56:04.494 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-6391/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-08-14 23:56:04.497 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 06d9a6b HIVE-16873: Remove Thread Cache From Logging (BELUGA BEHR reviewed by Aihua Xu) + git clean -f -d Removing common/src/java/org/apache/hadoop/hive/common/ndv/fm/FMSketch.java Removing metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/cache/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 06d9a6b HIVE-16873: Remove Thread Cache From Logging (BELUGA BEHR reviewed by Aihua Xu) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-08-14 23:56:11.579 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch Going to apply patch with: patch -p1 patching file metastore/src/java/org/apache/hadoop/hive/metastore/TableType.java patching file ql/if/queryplan.thrift patching file ql/src/gen/thrift/gen-cpp/queryplan_types.cpp patching file ql/src/gen/thrift/gen-cpp/queryplan_types.h patching file ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/StageType.java patching file ql/src/gen/thrift/gen-php/Types.php patching file ql/src/gen/thrift/gen-py/queryplan/ttypes.py patching file ql/src/gen/thrift/gen-rb/queryplan_types.rb patching file ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplStateLogTask.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplStateLogWork.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/ReplLoadTask.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/ReplLoadWork.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/events/filesystem/BootstrapEventsIterator.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/events/filesystem/DatabaseEventsIterator.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/LoadFunction.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java patching file ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java patching file ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSpec.java patching file ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/Utils.java patching file ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/AbstractMessageHandler.java patching file ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/CreateFunctionHandler.java patching file ql/src/java/org/apache/hadoop/hive/ql/parse/repl/log/logger/BootstrapDumpLogger.java patching file ql/src/java/org/apache/had
[jira] [Commented] (HIVE-17100) Improve HS2 operation logs for REPL commands.
[ https://issues.apache.org/jira/browse/HIVE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16126586#comment-16126586 ] ASF GitHub Bot commented on HIVE-17100: --- GitHub user sankarh opened a pull request: https://github.com/apache/hive/pull/231 HIVE-17100: Improve HS2 operation logs for REPL commands. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sankarh/hive HIVE-17100 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/231.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #231 commit 2e7de9cff5093aceab124917f23b505fb0330d7b Author: Sankar Hariappan Date: 2017-07-24T06:44:34Z HIVE-17100: Improve HS2 operation logs for REPL commands. > Improve HS2 operation logs for REPL commands. > - > > Key: HIVE-17100 > URL: https://issues.apache.org/jira/browse/HIVE-17100 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > > It is necessary to log the progress the replication tasks in a structured > manner as follows. > *+Bootstrap Dump:+* > * At the start of bootstrap dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (BOOTSTRAP) > * (Estimated) Total number of tables/views to dump > * (Estimated) Total number of functions to dump. > * Dump Start Time{color} > * After each table dump, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table dump end time > * Table dump progress. Format is Table sequence no/(Estimated) Total number > of tables and views.{color} > * After each function dump, will add a log as follows > {color:#59afe1}* Function Name > * Function dump end time > * Function dump progress. Format is Function sequence no/(Estimated) Total > number of functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > dump. > {color:#59afe1}* Database Name. > * Dump Type (BOOTSTRAP). > * Dump End Time. > * (Actual) Total number of tables/views dumped. > * (Actual) Total number of functions dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The actual and estimated number of tables/functions may not match if > any table/function is dropped when dump in progress. > *+Bootstrap Load:+* > * At the start of bootstrap load, will add one log with below details. > {color:#59afe1}* Database Name > * Dump directory > * Load Type (BOOTSTRAP) > * Total number of tables/views to load > * Total number of functions to load. > * Load Start Time{color} > * After each table load, will add a log as follows > {color:#59afe1}* Table/View Name > * Type (TABLE/VIEW/MATERIALIZED_VIEW) > * Table load completion time > * Table load progress. Format is Table sequence no/Total number of tables and > views.{color} > * After each function load, will add a log as follows > {color:#59afe1}* Function Name > * Function load completion time > * Function load progress. Format is Function sequence no/Total number of > functions.{color} > * After completion of all dumps, will add a log as follows to consolidate the > load. > {color:#59afe1}* Database Name. > * Load Type (BOOTSTRAP). > * Load End Time. > * Total number of tables/views loaded. > * Total number of functions loaded. > * Last Repl ID of the loaded database.{color} > *+Incremental Dump:+* > * At the start of database dump, will add one log with below details. > {color:#59afe1}* Database Name > * Dump Type (INCREMENTAL) > * (Estimated) Total number of events to dump. > * Dump Start Time{color} > * After each event dump, will add a log as follows > {color:#59afe1}* Event ID > * Event Type (CREATE_TABLE, DROP_TABLE, ALTER_TABLE, INSERT etc) > * Event dump end time > * Event dump progress. Format is Event sequence no/ (Estimated) Total number > of events.{color} > * After completion of all event dumps, will add a log as follows. > {color:#59afe1}* Database Name. > * Dump Type (INCREMENTAL). > * Dump End Time. > * (Actual) Total number of events dumped. > * Dump Directory. > * Last Repl ID of the dump.{color} > *Note:* The estimated number of events can be terribly inaccurate with actual > number as we don’t have the number of events upfront until we read from > metastore NotificationEvents table. > *+Incremental Load:+* > * At the start of incremental load, will add one log with below details. > {color:#59afe1}* Target Database Name > * Dump directory > * Load Type (INCREMENTAL) > * Total