This is an automated email from the ASF dual-hosted git repository. yumwang pushed a change to branch slow in repository https://gitbox.apache.org/repos/asf/spark.git
was 6650667c407 fix This change permanently discards the following revisions: discard 6650667c407 fix discard e0df57fdbbd Use ExpressionSet and add UT discard b042b755f93 [CARMEL-6796] Pull out complex aggregate expressions discard 29bd591cd2f [carmel-6785] Add configuration spark.sql.materializedView.name.prefix to support definition of available mv (#1307) discard b3cce6eb3a0 [MINOR] Bug fix for HiveUDF codegen (#1303) discard 905523e4cd2 [CARMEL-6734] Fix reorder filter condition issue (#1293) discard b3036d65479 [CARMEL-6741] Tag Queries with Join Expansion in Runtime (#1299) discard a6d9e7c163d [CARMEL-5941] Add how to request HDM data access to error message (#1301) discard 285aff3d2a4 [CARMEL-6739][SPARK-43050][SQL] Fix construct aggregate expressions by replacing grouping functions (#1300) discard d803e943edb [CARMEL-6751] More reasonable error message when heavily skewed partition (#1297) discard 2e9d18bd5dd [CARMEL-6760] Correct metadata ‘mv_updated_time’ which is used for data validation (#1298) discard 2973a317042 [CARMEL-6633] Reduce Skew Join Split Size Considering Expand Node (#1295) discard 46be35accc9 [CARMEL-6664] Fix the InterruptedException error and incorrect state for cancelled download (#1291) discard ee430edaf39 [CARMEL-6703] Take too much time to build bloom filter (#1292) discard f76533f3acc [Carmel 6640] Support create materialized view as datasource table (#1280) discard 7efe9b78cff [CARMEL-6647] Enhance DemoteBucketJoin to support Alias Aware Output Partitioning (#1290) discard e7d9e88ddcb [CARMEL-6583][SPARK-42500][SQL] ConstantPropagation support more cases (#1287) discard d8fa6e4e368 [CARMEL-6621][SPARK-42789][SQL] Rewrite multiple GetJsonObjects to a JsonTuple if their json expressions are the same (#1286) discard 5b0fffed633 [CARMEL-6705] Bug fix for query output row count (#1289) discard 96c7333cb22 [CARMEL-6615] Backport [SPARK-42052][SQL] Codegen Support for HiveSimpleUDF (#1288) discard a53b95c0c4d [CARMEL-6675] Support to enable decommission nodes when hive service discovery is disabled (#1282) discard c08771ff897 [CARMEL-6683][SPARK-31008][SQL] Support json_array_length function (#1284) discard b243ff8b59e [CARMEL-6674] Project fail to be collapsed (#1283) discard 93f89f941dc [CARMEL-6587] Support Generic Skew Join Patterns (#1281) discard 81d76f6b116 [CARMEL-6652][SPARK-40501][SQL] Add PushProjectionThroughLimit for Optimizer (#1278) discard 3e8e629a2d0 [CARMEL-6655] Do not trim whitespaces by default when downloading data as CSV file (#1279) discard b3e8faafbda [CARMEL-6651] Remove repartition if it is the child of LocalLimit (#1277) discard 2ddd418edf4 [CARMEL-6632] Fix the running time of download statement in query log (#1275) discard 9e899c166ce [CARMEL-6586] Ignore SinglePartition when determining expectedChildrenNumPartitions (#1252) discard 25a3b904b61 [CARMEL-6327] Support Broadcast Join with Stream Side Skew (#1272) discard c3f17a52289 [CARMEL-6608] Increase bucket table scan partitions (#1269) discard 434b16e0cc8 [CARMEL-6439] Define new query execution event and log column lineage asynchronously (#1268) discard e77e54fd7da [CARMEL-6609] Casts types according to bucket info support view (#1270) discard 872399e7a5e [CARMEL-6593] Upgrade parquet to 1.12.3.0.1.0 (#1271) discard 40900d24073 [CARMEL-6604] Stop posting duplicate execution event (#1267) discard e2036ae1eda [CARMEL-6591][SPARK-42597] Support unwrap date type to timestamp type (#1263) discard f7c77ff2bbd [CARMEL-6582][SPARK-42513][SQL] Push down topK through join (#1257) discard a214b2b67fd [CARMEL-6541] Support Query Level SQL Conf leveraging Hint (#1256) discard 10a27944ccf [CARMEL-6568] Analyze join operator and support data expansion check (#1265) discard 56c9bad9395 [CARMEL-6511] Disable rename temp table (#1254) discard 71489e91204 [CARMEL-6371] Partial aggregation push through left/right outer join (#1233) discard 7905f88b1f3 [CARMEL-6581] TakeOrderedAndProject should not replace project if project expression is not deterministic (#1249) discard 142b10b1580 [CARMEL-6339] Implementation of materialized view (#1244) discard 1cd2cbcaa2f [CARMEL-6556] Avoid coalesce partitions from different UNION sides (#1246) discard 413224b3270 [CARMEL-6495] Do not quit am even when all nodes are in blacklist (#1248) discard 9f306b4ddf4 [CARMEL-6439] Add configuration to enable log column lineage (#1238) discard 5f91f872777 [CARMEL-6553] Backport [SPARK-35673][SQL] Fix user-defined hint and unrecognized hint in subquery (#1236) discard d8b439948dd [CARMEL-6525][MINOR] Support tag different drivers in the queue (#1237) discard c4930985ae1 [CARMEL-6537] [Followup] Support Iceberg with maven dependency in Carmel - use correct jar (#1234) discard adbb50db673 [CARMEL-6498] handle multi-identifier in HiveAuthorizer/PrivilegesBuilder for V2 command (#1219) discard 62da19c8ae7 [CARMEL-6531] Fail to instantiate the custom v2 session catalog: DelegatingCatalog (#1221) discard 91bc3d5d8ea [CARMEL-6537] Support Iceberg with maven dependency in Carmel (#1225) discard 2101f0f784b [CARMEL-6525] Support tag different drivers in the queue (#1229) discard 6fa10a9d035 [CARMEL-6542] Change ADVISORY_PARTITION_SIZE_IN_BYTES to SKEW_JOIN_SKEWED_PARTITION_THRESHOLD in HandleOuterJoinBuildSideSkew (#1224) discard cc254f23bcf [CARMEL-6383] Allow introduce new shuffle in skew handling (#1161) discard e49b25b544b [CARMEL-6532] Fix NoSuchElementException in HandleOuterJoinBuildSideSkew (#1220) discard 7709f60dbf0 [CARMEL-6491] Collapse percentile functions if possible (#1216) discard 11334447e4e [CARMEL-6497] Left outer join and left has lots of nulls (#1202) discard deb3e8254da [CARMEL-6518] Record temp view usage in workload analysis (#1208) discard 8c7e20f9d0a [CARMEL-6513] Bug fix for reorder predicate (#1207) discard 21b3c47488b [CARMEL-6511] Add log if rename temporary table (#1215) discard 3970b4641be [SPARK-41805][SQL] Reuse expressions in WindowSpecDefinition (#1212) discard 2d2f2b094ef [CARMEL-6489][SPARK-33205][SPARK-33725][SPARK=35277][BUILD] Bump snappy-java version to 1.1.8.4 (#1198) discard 2c84685793d [CARMEL-6496] Implement bloom filter join hint #1201 discard cb107711df0 [CARMEL-6490] Lots of unfinished state sql in the query log (#1199) discard 7d73036b0e5 [CARMEL-6484] Infer IsNotNull constraints for GetJsonObject (#1196) discard 24ddd14b143 [CARMEL-6482] HiveThriftServer2Listener memory leak because of event lost (#1194) discard 5bdfd6e1f16 [CARMEL-6483] Can't zip RDDs with unequal numbers of partitions in skew join handling (#1195) discard 1329dcac486 CARMEL-6449: Enhance runtime filter (#1180) discard e03c9211bc6 CARMEL-6447: Do not push down sum if it is decimal type (#1188) discard 06e0907b6de [CARMEL-6461] Too much join expansion cause many zombie tasks (#1183) discard 33506f3581f [CARMEL-6471] Upload api sometimes throw Delegation Token error (#1191) discard 48d4f4381d5 [CARMEL-6388] Backport [SPARK-33092][SQL] Support subexpression elimination in ProjectExec (#1166) discard 5d686257756 [CARMEL-6200][FollowUp4] Improve table lineage extraction (#1181) discard 7d90243c56e HADP-47632. Update ZSTD Version to 1.4.8-1 (#1175) discard b178cb79931 [CARMEL-6426] Adjust scan parallelism dynamically improvement (#1178) discard 6c15686f9a7 [CARMEL-6400][FollowUp] Log detail sql execution metrics - improvement (#1182) discard 8b865eb753a [CARMEL-6076][FollowUp2] Disable read lock for empty paths in Apollo when table location parameter is empty (#1170) discard 405009e9688 [CARMEL-6444] Get correct query objects from AQE plan (#1176) discard ce6d038760d [CARMEL-6200][FollowUp3] Improve table lineage extraction (#1169) discard c35e50e3157 [CARMEL-6434] Expose active sql related metrics from driver (#1171) discard a99b6e326fc [CARMEL-6400] Log detail sql execution metrics (#1162) discard e3898e836a9 [CARMEL-6392] Bug fix for hive operation event (#1151) discard d6b2dacab51 [CARMEL-6396] Reduce the instances of OPTIMIZE TAG (#1165) discard 8b119607e4b [CARMEL-6427] TaskResultGetter eventloop exit unexpectedly (#1167) discard bdbb6dcf7b7 [CARMEL-6200][Follow-Up2] Improve table lineage extraction (#1164) discard 817c553f6cd [CARMEL-6394] Implement column lineage tools (#1159) discard da6af85bb05 [CARMEL-6401] Bug fix for SortMergeJoin outer join codegen (#1158) discard aae616d7ce2 [CARMEL-6398] Workaround: `UnresolvedException: Invalid call to dataType on unresolved object` (#1154) discard 33803654937 [CARMEL-6367][FOLLOW-UP] Insert bloom filter if it is skew bucket join (#1157) discard f91eb96f082 [CARMEL-6399] Download hive-1.2.1 jar files to local to improve load properties performance (#1160) discard 6b533c191eb [CARMEL-6367] Insert bloom filter if it is skew bucket join (#1152) discard 09e83b7b6a4 [CARMEL-6387] Backport [SPARK-35447][SQL] Optimize skew join before coalescing shuffle partitions (#1153) discard 4d6e245a1fa [CARMEL-6381] Remove unnecessary sql metrics for UnionExec (#1146) discard c25fdd6d460 [CARMEL-6375] Support Potential Skewed Operator Tagging (#1147) discard 3f2a8a9ae70 [CARMEL-6380] Merge task metrics in task result getter threads instead of DAGScheduler (#1145) discard e210385dd48 [CARMEL-6365] Don't store data in quota free folder when create temp table (#1148) discard 36176069f6a [CARMEL-6355][FOLLOWUP] Auto update table statistics based on write metrics (#1142) discard d1c2c3a0bf8 Revert "[CARMEL-6365] Don't store data in quota free folder when create temp … (#1126)" (#1143) discard 0bc12a616ba [CARMEL-6354] Increase parallelism for get_json_object (#1139) discard 3eddc31eaeb [CARMEL-6372] Adjust skewed join split size (#1141) discard 177c4b0b506 [CARMEL-6355] Auto update table statistics based on write metrics (#1123) discard c6c737d2f2c [CARMEL-6365] Don't store data in quota free folder when create temp … (#1126) discard 813d18323ad [CARMEL-6345] Support backup table command (#1124) discard 9a4f8c3830d [CARMEL-6174][FOLLOWUP] Change prefer shuffled hash join condition (#1099) discard 41e552438c8 [CARMEL-6324][MINOR 2] Support bucket skew detection (#1140) discard 8a942231b35 [CARMEL-6358] Change to file commit algorithm V2 in CTAS command for Hive table (#1125) discard 9c158d84810 [CARMEL-6324][MINOR] Support bucket skew detection (#1134) discard faae0248513 [CARMEL-6360] Backport date_from_unix_date, unix_date, timestamp_seconds, timestamp_millis, timestamp_micros, unix_seconds, unix_millis, unix_micros functions (#1121) discard 19e4c637e8d [CARMEL-6352] Adjust scan partition size dynamically considering potential cost (#1110) discard d0a000ee91c [CARMEL-6324] Support bucket skew detection (#1130) discard 9f3b826db3d [CARMEL-6254][FOLLOW-UP] Collapse the project after pulling out complex join condition (#1115) discard 6a3834d0552 [CARMEL-6243][FOLLOW-UP] The right side max size should larger than `spark.sql.adaptive.advisoryPartitionSizeInBytes` (#1120) discard ce67e3f8b3b [CARMEL-6361] Too many casewhen expression cause driver full gc (#1122) discard 70b48fae0a5 [CARMEL-6300][FOLLOW-UP] Casts types according to bucket info only accept decimal type (#1111) discard cbcd848144e [CARMEL-6300] Casts types according to bucket info for Equality expressions (#1107) discard 708a1d887da [CARMEL-6306] backport [SPARK-33399][SPARK-31078] Normalize output partitioning and sortorder with respect to aliases to avoid unneeded exchange/sort nodes (#1092) discard 5ba77832724 [CARMEL-6330] Support Range Partition Skew Handling (#1109) discard fe3ece3359d [HADP-46618][HADP-46404] add DelegatingCatalog to delegate to various… (#1076) discard 479e64ed9ad [CARMEL-6314] Make sure registerExecutor event must be handled in HeartBeatReceiver (#1097) discard a726a477f40 [CARMEL-6243][FOLLOW-UP] Insert partial aggregate for right side of left anti join (#1100) discard e42088d73e0 [CARMEL-6305] Support to show partitions for Delta table (#1094) discard 14b521dd8dd [CARMEL-6315] Add physicalPlanDescription column for adaptive query execution event (#1098) discard ed9aa9491bd [CARMEL-6174][FOLLOWUP] Change prefer shuffled hash join condition (#1078) discard 42640cd9902 [CARMEL-6254] Pull out complex join condition (#1091) discard af0d2d09875 [CARMEL-6231] Cleanup shuffle merge data after query finished (#1075) discard b78322f5329 [CARMEL-6299] Expose stage/task retry count to Carmel Overview (#1090) discard 1be4a01343f [CARMEL-6200][Follow-Up2] Fix task not serializable exception for table lineage (#1086) discard 9147a5cbd0a [CARMEL-6292] Fix spilled query result leak (#1089) discard ee779238d9e [CARMEL-6185] Expose row count for RepeatableIterator (#1074) discard e0ff5303e89 [CARMEL-6280] Data download query stuck in job commit phase (#1084) discard aa62055bef3 [CARMEL-6243] Handle outer join build side skew (#1088) discard ea35516ae1c [CARMEL-6209] Make defaultJoin in BroadcastNestedLoopJoinExec running in parallel (#1080) discard 5ee3529e6ea [CARMEL-6152] [Followup3] Handle FileNotFoundException when listing event log files (#1083) discard 5eafda27cd4 [CARMEL-6278] Avoid generating constraints if isnull and isnotnull refer to multiple expressions (#1082) discard 280b61976db [CARMEL-6283] DAGScheduler may stuck when there are too many result task and result is spilled (#1085) discard d8092193bb4 [CARMEL-6265] Only push down low cost expression (#1081) discard c67115ac4b5 [CARMEL-6222]Improve congestion in shuffle service (#1068) discard 6b6221e43f2 [CARMEL-6218] Throw exceptions to ensure the completeness of event log files (#1073) discard 86b6e50a65f [CARMEL-6208] Extract important information from physicalPlanDescription in EventLog (#1069) discard fd3df9d4d0f [CARMEL-6200][Follow-Up] Improve table lineage extraction (#1070) discard 950b4174ca8 [CARMEL-6148] Adjust Partition Size considering ExpandExec project size in AQE (#1072) discard eb2fd9a70cb [CARMEL-6117] Demote Bucket Join if Shuffle Cost Acceptable (#1066) discard 8cf4ec759c6 [CARMEL-6200] Add unified command to extract table lineage information for Hadoop platform (#1067) discard 5e6eb523ad3 [CARMEL-6174][FOLLOWUP] Change prefer shuffled hash join condition (#1064) discard ac84911529c [CARMEL-6190] Upgrade Scala to 2.12.16 (#1063) discard c95d3418b52 [CARMEL-6178] Too many duplicate values of the join key (#1062) discard 9387627ee51 [CARMEL-6184] Make outputtofile meaningful (#1057) discard f02d2ee27b3 [CARMEL-6166] Push local limit to both sides if join condition is empty discard 62e82b577f5 [CARMEL-6186] Skip constraint validation when CTAS for Hive tables (#1058) discard 063400d900c [CARMEL-6152] [Followup2] Calculate more accurate information in the view- fix StackOverflowError (#1061) discard e9e5f39780a [CARMEL-4174] Backport Hash join PR / Merge pull request #1054 from carmel/hash_join discard 97b0ece899a [CARMEL-6152] [Follow-up] Calculate more accurate information in the view (#1055) discard 156d8d94515 [CARMEL-6151] Exchange Push Down through Aggregate (#1053) discard 590dc6bace4 [CARMEL-4174] Backport [CARMEL-5757] Avoid transform SMJ to BHJ in AQE if rows > 50000000 detected discard fc6a448e57b [SPARK-37753][FOLLOWUP][SQL] Add comments to unit test discard f34f4eb38d1 [SPARK-37753][FOLLOWUP][SQL] Fix unit tests sometimes failing discard 2b43ba9bfd0 [SPARK-37753][SQL] Fine tune logic to demote Broadcast hash join in DynamicJoinSelection discard c67900a379e [SPARK-36898][SQL] Make the shuffle hash join factor configurable discard bc1dd6d5b6a [SPARK-35984][SQL][TEST] Config to force applying shuffled hash join discard 961174e4bdf [SPARK-35282][SQL] Support AQE side shuffled hash join formula using rule discard 53f3368d6bf [CARMEL-6159] Generate download records in carmel_ql_object_v (#1052) discard f5a43a2bb68 [CARMEL-6156] Fix PredicateReorder issue (#1047) discard 4cab0abb286 [CARMEL-6121][Followup] Skip subsequent checks after the first authorization is passed (#1051) discard f1e565aba9e [CARMEL-6144] Add external shuffle service to blacklist if connection creation fails (#1039) discard 5067bae9b68 [CARMEL-6152] Support analyze view command to detect view information (#1045) discard 1a7fb37896a [CARMEL-6155] Fix limit only query contain aggregate functions (#1044) discard 6dd2e0b1955 [CARMEL-5851] Push partial aggregate through join (#1043) discard e12e8d3d533 [CARMEL-6113] Support BHJ fallback to SMJ in AQE (#1042) discard 20fb4393a0a [Carmel-6124] backport [SPARK-37618][CORE] Remove shuffle blocks using the shuffle service for released executors (#1041) discard 58c55ead3a3 [CARMEL-6143] Refactor bucket information in workload analysis (#1040) discard 9f6405ea976 [CARMEL-6133][FOLLOWUP] Fix scala.MatchError (#1038) discard 21caf477856 [CARMEL-6134] Backport [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number (#1033) discard 734e1fc64ff [CARMEL-6133] Improve adaptively adjust spark.sql.adaptive.advisoryPartitionSizeInBytes (#1032) discard d933af5ec44 [CARMEL-6139] Task thread stuck when coalese and multi-thread scan is enable (#1037) discard a52df6a12ac [CARMEL-6136] Use bucket scan size limit instead of normal limit for disabled bucketing (#1036) discard 8cd02349b6d [CARMEL-6135] Report Selected Buckets Number and Tasks Count when Bucket Scan Disabled by Planner (#1034) discard 038befb5d6e [CARMEL-6120] [Follow Up] Support track the optimized plan using tag (#1035) discard f5378c8f647 [CARMEL-6129] Backport [SPARK-39989][SQL] Support estimate column statistics if it is foldable expression (#1031) discard bcdbf824c89 [CARMEL-5936] Backport [SPARK-30602]SPIP: Support push-based shuffle to improve shuffle efficiency discard f50697160c4 [CARMEL-6121] Check subquery privilege objects in UPDATE and DELETE command (#1025) discard 24da01a40d7 [CARMEL-6100] Reduce Hadoop RPC call when compacting table (#1029) discard 4a56f858459 [CARMEL-6122] Add queue and application info into workload analysis data (#1027) discard 04b98c499d3 [CARMEL-6120] Support track the optimized plan using tag (#1026) discard 336cd980e71 [CARMEL-6106] Improve join stats estimation if one side can keep uniqueness (#1024) discard 0040dda69ce [CARMEL-6024]Bug fix for fetching large shuffle block discard be522499491 [CARMEL-5936][BACKPORT][CARMEL-3545] Shuffle memory usage improvement discard dbe3670b5b2 [CARMEL-5936] Limit merge writer tasks number discard df9c75bebfb [CARMEL-5936] Bug fix for bind exception from netty server discard a1cda81bb3c [CARMEL-5936] Optimize merge mapStatuses and mergeStatuses code discard 663a32fdbf9 [CARMEL-5936] Limit the number of merge locations by spark.shuffle.push.maxMergerLocations discard 8d87c4199cd [CARMEL-5936] Optimize RemoteBlockPushResolver discard bcd665b94ac [SPARK-39072][SHUFFLE]Fast fail the remaining push blocks if shuffle stage finalized discard c7c5febe5f6 [BACKPORT][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized discard 169bdfa21bc [BACKPORT][SPARK-35548][CORE][SHUFFLE] Handling new attempt has started error message in BlockPushErrorHandler in client discard e52f762cc70 [SPARK-39080][SHUFFLE]Optimize shuffle error handler discard 63b327ea2c5 [CARMEL-5936] Optimize RemoteBlockPushResolver discard 13ee589ae88 [CARMEL-5936] Change external shuffle service port to 7338 discard 8a988351085 [SPARK-36772] FinalizeShuffleMerge fails with an exception due to attempt id not matching discard ede862c7298 [SPARK-33206][CORE] Fix shuffle index cache weight calculation for small index files discard ee0aa819a4e [SPARK-36378][SHUFFLE] Switch to using RPCResponse to communicate common block push failures to the client discard aa7cb12b29f [SPARK-33037][SHUFFLE] Remove knownManagers to support user's custom shuffle manager plugin discard 46fcdae487f [SPARK-32077][CORE] Support host-local shuffle data reading when external shuffle service is disabled discard 820b44411c2 [SPARK-37481][CORE][WEBUI] Fix disappearance of skipped stages after they retry discard ad360b72d95 [MINOR]Bug fix for testCases discard 91462cf14d6 [SPARK-34826][SHUFFLE] Adaptively fetch shuffle mergers for push based shuffle discard de3e938f071 [SPARK-33701][SHUFFLE] Adaptive shuffle merge finalization for push-based shuffle discard a6a831f4a6e [SPARK-37023][CORE] Avoid fetching merge status when shuffleMergeEnabled is false for a shuffleDependency during retry discard 14df3d5297b [SPARK-36892][CORE] Disable batch fetch for a shuffle when push based shuffle is enabled discard ea53c4fe17a [SPARK-36705][FOLLOW-UP] Support the case when user's classes need to register for Kryo serialization discard d539145fc16 [SPARK-36705][FOLLOW-UP] Fix unnecessary logWarning when PUSH_BASED_SHUFFLE_ENABLED is set to false discard 8bc42a8d961 [SPARK-36705][SHUFFLE] Disable push based shuffle when IO encryption is enabled or serializer is not relocatable discard 8ab806a20c5 [SPARK-32923][FOLLOW-UP] Clean up older shuffleMergeId shuffle files when finalize request for higher shuffleMergeId is received discard 8ba68f24fac [SPARK-36389][CORE][SHUFFLE] Revert the change that accepts negative mapId in ShuffleBlockId discard 728d09a933d [SPARK-32919][FOLLOW-UP] Filter out driver in the merger locations and fix the return type of RemoveShufflePushMergerLocations discard d0164c9d4a6 [SPARK-32923][CORE][SHUFFLE] Handle indeterminate stage retries for push-based shuffle discard 1f3188d4476 [SPARK-36362][CORE][SQL][FOLLOWUP] Fix java linter errors discard 00e0bc78511 [SPARK-36362][CORE][SQL][TESTS] Omnibus Java code static analyzer warning fixes discard 2cefa8e890e [SPARK-36374][FOLLOW-UP] Change config key spark.shuffle.server.mergedShuffleFileManagerImpl to spark.shuffle.push.server.mergedShuffleFileManagerImpl discard f9b2e51451b [SPARK-36374][SHUFFLE][DOC] Push-based shuffle high level user documentation discard dacdc0ef9df [SPARK-32920][CORE][FOLLOW-UP] Fix string interpolator in the log discard 228aef5ba73 [SPARK-36266][SHUFFLE] Rename classes in shuffle RPC used for block push operations discard d79213d57b5 [SPARK-32920][FOLLOW-UP] Fix shuffleMergeFinalized directly calling rdd.getNumPartitions as RDD is not serialized to executor discard a2e4e844eca [SPARK-32920][FOLLOW-UP][CORE] Shutdown shuffleMergeFinalizeScheduler when DAGScheduler stop discard 70f9b4a0689 [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions discard 09db3a0eb1e [SPARK-36273][SHUFFLE] Fix identical values comparison discard 0427756cc43 [SPARK-35546][SHUFFLE] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better way discard 0940c41ea2a [SPARK-32922][SHUFFLE][CORE][FOLLOWUP] Fixes few issues when the executor tries to fetch push-merged blocks discard 52ba7f22c0b [SPARK-32920][CORE][SHUFFLE][FOLLOW-UP] Fix to run push-based shuffle tests in DAGSchedulerSuite in ad-hoc manner discard 1de132197e9 [SPARK-32922][SHUFFLE][CORE] Adds support for executors to fetch local and remote merged shuffle data discard 562fb49eb56 [SPARK-35836][SHUFFLE][CORE] Removed the reference to spark.shuffle.push.based.enabled in ShuffleBlockPusherSuite discard 0cd82f5082f [SPARK-35671][SHUFFLE][CORE] Add support in the ESS to serve merged shuffle block meta and data to executors discard 49db4ca817c [SPARK-33350][SHUFFLE] Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data discard 6fb49cac6db [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage discard 0fbcb39439b [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost discard cb902fef436 [SPARK-32210][CORE] Fix NegativeArraySizeException in MapOutputTracker with large spark.default.parallelism discard 4593e4ae265 [SPARK-32921][SHUFFLE] MapOutputTracker extensions to support push-based shuffle discard 573345a2f57 [SPARK-32055][CORE][SQL] Unify getReader and getReaderForRange in ShuffleManager discard b6cace76d37 [SPARK-34840][SHUFFLE] Fixes cases of corruption in merged shuffle … discard c0e93b34152 [SPARK-32917][SHUFFLE][CORE] Adds support for executors to push shuffle blocks after successful map task completion discard 8ea13672912 [SPARK-32916][SHUFFLE][TEST-MAVEN][TEST-HADOOP2.7] Ensure the number of chunks in meta file and index file are equal discard 3fef6a6d3d5 [SPARK-32918][SHUFFLE] RPC implementation to support control plane coordination for push-based shuffle discard 4b4e1b30159 [SPARK-32919][SHUFFLE][TEST-MAVEN][TEST-HADOOP2.7] Driver side changes for coordinating push based shuffle by selecting external shuffle services for merging partitions discard 427bf6817cc [SPARK-32103][CORE] Support IPv6 host/port in core module discard ef3c3436b9e [SPARK-32916][SHUFFLE][TEST-MAVEN][TEST-HADOOP2.7] Remove the newly added YarnShuffleServiceSuite.java discard b9cb3c86669 [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode discard fd7f265360b [SPARK-32915][CORE] Network-layer and shuffle RPC layer changes to support push shuffle blocks discard b11685d693d [CARMEL-6055] Backport code-gen code for SortMergeJoin (#987) discard a1ec6fb223f [CARMEL-6057] Fix limit push down through window (#1016) discard dbb8d88fa9f [CARMEL-6103] Log final spark plan and query id in Event Analysis (#1023) discard 8d8ab5780bd [CARMEL-6050] Support Remove Redundant Partial Aggregate (#1022) discard 06c431b382a [CARMEL-6109] Improve adaptive bloom filter join makes it only works after both side shuffle (#1021) discard a16abafc3c2 [CARMEL-6061] Skip the storage detection for access-denied paths (#1013) discard 70beb72d98c [CARMEL-6076][Followup] Fix the incorrect qualified table path to determine the read lock (#1020) discard 34780adeb1c [CARMEL-6077] Use `TakeOrderedAndProject` if maxRows below the topKSortMaxRowsThreshold (#1015) discard de7f557b696 [CARMEL-6087] User is continually blocked by circuit breaker (#1009) discard dabbefe1579 [CARMEL-6086] DAGScheduler exit because of jobId not exist (#1008) discard 396aa517b7e [CARMEL-6085] Show metrics after running analyze event command (#1012) discard 1e147656e91 [CARMEL-6055] Backport [SPARK-29375][SPARK-28940][SPARK-32041][SQL] Whole plan exchange and subquery reuse (#993) discard 25d0233595e [CARMEL-6055][SPARK-35874][SQL] AQE Shuffle should wait for its subqueries to finish before materializing (#998) discard c31f7276aa9 [CARMEL-5845] Add a logical plan visitor to propagate the distinct attributes (#884) discard 8adae4c167a [CARMEL-6083] Support batch account for non-columnar format data read (#1007) discard e5a79edcaa9 [CARMEL-6076][Followup] Fix UT failure caused by compacting empty directory (#1010) discard 01fb33611df [CARMEL-6076] Only obtain read lock in Hermes cluster when file listing for empty directory (#1005) discard c03dd28ac2d [CARMEL-6074] Should not insert adaptive bloom filter join if it is partial aggregate #1001 discard c3ccc2d8c21 [CARMEL-6072] Improve error message for SchemaColumnConvertNotSupportedException (#1004) discard 2f8a1e6a294 [SPARK-34807][SQL] Transpose Window nodes with Project between them (#995) discard 48ecfcee899 [CARMEL-6068][SPARK-36137][SQL] HiveShim should fallback to getAllPartitionsOf even if directSQL is enabled in remote HMS discard ceb72aae9ac [CARMEL-6073] Make PartitionRecombinationExec effective when bucketing disabled by planner (#1000) discard 3e89b9963d0 [CARMEL-5818] Support analyze event log command (#994) discard c1cd2be8afd [CARMEL-6069] Improve error message when cannot broadcast the table with more than 50000000 rows discard 2b932c08756 [CARMEL-6057] Limit push down through window (#990) discard 27841f2be96 [CARMEL-6044] Push down limit 1 for right side of left semi/anti join if join condition is empty (#984) discard d9f56b90e83 [CARMEL-6030][FollowUp] Fix OutOfMemoryError-original fix may cause task thread stuck (#986) discard b06e3a1150b [CARMEL-6043] Avoid file number exceeds spark.sql.dynamic.partition.maxCreatedFiles (#983) discard b0cf5e225f4 [CARMEL-5997][FOLLOW UP] Support more sql patterns for deciding bucketed scan dynamically (#980) discard 20b9c9bc9ea [CARMEL-6030] Fix OutOfMemoryError: unable to create new native thread (#979) discard 70a1bfe3896 [CARMEL-6033] NullPointException in uploading when NOT NULL constraint enforced (#976) discard 2a16fb31aec [CARMEL-6012] Enhance rewrite table location to absolute location based on database location (#970) discard 8b1c5a39be7 [CARMEL-6010] Distinguish duplicate execution event log for command query (#963) discard 5550fdb67ee [CARMEL-5997] Support more sql patterns for deciding bucketed scan dynamically (#975) discard 0fe346f24dc [CARMEL-5970] Avoid introducing lots of partitions when bucketing disabled by SQL Planner (#973) discard 5819623bb51 [CARMEL-5986] Make Coalesce/Rebucketing effective when bucketing disabled by planner (#961) discard d6fd502829b [CARMEL-6009] Add realUser to caller context to support NN throttling (#968) discard a1b61bfa364 [CARMEL-6006] Log stack trace to track the source of query cancelation (#962) discard 6fc4feabca4 [CARMEL-6021] Improve error message for user specified column list (#971) discard b836220cfac [CARMEL-5987][SPARK-39217] Makes DPP support the pruning side has Union (#967) discard 347f4bfc963 [CARMEL-5999] Fix incorrect sql details in event log for jobs that detect small files (#956) discard 2a2758989f1 [CARMEL-5979] Add view usage information in event log for workload analysis (#945) discard 1e1391c32c6 [CARMEL-5995] Backport [SPARK-39216][SQL] Do not collapse projects in CombineUnions if it hasCorrelatedSubquery (#966) discard f1b868e3422 [CARMEL-6008]Bug fix for extracting db name from relation multipartIdentifier (#959) discard 0d2a6e762e7 [CARMEL-5962][backport][SPARK-33480][SQL] Support char/varchar type (#936) discard 72445922d94 [CARMEL-5994] Lock conflict between drop table and metadata scan (#954) discard 9c19847ebc4 [CARMEL-5963] CartesianProduct may introduce many concurrent connections to JDBC Relation (#949) discard 720ad125978 [CARMEL-5983] Backport [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function (#941) discard a4b54a83a71 [CARMEL-5869] Delete kafka.xml from source code(#885) discard f101f70b421 [CARMEL-5984] Backport [SPARK-35264][SQL] Support AQE side broadcastJoin threshold (#942) discard fa58ac39d7f [CARMEL-5981][SPARK-34037][SQL] Remove unnecessary upcasting for Avg & Sum which handle by themself internally (#940) discard 1522b2ee22c [CARMEL-4914][FOLLOWUP] Fix Number of partitions (0) must be positive discard 8379a2a3a7b [CARMEL-5918]backport upstream performace changes (#913) discard dcf55a1fcfc [CARMEL-5948] Backport SPARK-32859 and related follow-up: Introduce p… (#937) discard 9804e2141a3 [CARMEL-5914] Adaptive Bloom filter Join (#905) discard 0d7b7c54224 [CARMEL-5978]Read table with a lot of empty patitions might fail discard 828c80ebda1 [CARMEL-5915][FollowUp] Filter view when listing tables to avoid AnalysisException discard b57689a9496 [CARMEL-5955] Drop NOT NULL constraint when alter column type (#930) discard 72c944a9ccf [CARMEL-5873] Upgrade Parquet to 1.12.2 (#901) discard 5035f66e5d0 [CARMEL-5912] Support PARALLEL Hint against Delta Table (#935) discard 0dd2021f507 [CARMEL-5939]Delete the temp local block file after reading it (#920) discard 45da4c1b72a [CARMEL-5779] Compact Auto Scheduling (#922) discard b1508a177e0 [CARMEL-5908] Support Table PARALLEL Hint in Number (#918) discard 2d3f2737a8f [CARMEL-5941] Add request permission through https://bdp.vip.ebay.com/aap/request-entry to error message discard 43597eb391d [CARMEL-5933] Runtime Filter supports pruning side has window (#909) discard 8c2a7dde3d7 [CARMEL-5915] Add command to detect HDM storage leak (#904) discard f27f1d56081 [CARMEL-5854] Data Read DML concurrent conflict (#906) discard 669368b1717 [CARMEL-5931] Avoid file number exceeds spark.sql.dynamic.partition.maxCreatedFiles(#908) discard d647e582032 [CARMEL-5878] Backport upstream changes #888 from hongyzhang/CARMEL-5878 discard 0ef658501a0 [CARMEL-5906] Expose constraint information in DESC command for Delta table (#902) discard ab97079c958 [CARMEL-5892] Support DPP if there is no selective predicate on the filtering side (#903) discard 0825e220177 [CARMEL-5902] Take dynamicPruningMaxInsetNum of rows to avoid Driver OOM (#898) discard 7221f3fd19c [SPARK-36874][SQL] DeduplicateRelations should copy dataset_id tag to avoid ambiguous self join discard a3427999f76 [CARMEL-5878]backp upstream test cases discard d4efe4dbb51 [CARMEL-5855]Compact Transparency - Other DDL concurrent conflict (#878) discard d6ae21e4799 [SPARK-37524][SQL] We should drop all tables after testing dynamic partition pruning discard f566f43ceff [SPARK-37196][SQL] HiveDecimal enforcePrecisionScale failed return null discard ec878aa9723 [SPARK-37076][SQL] Implement StructType.toString explicitly for Scala 2.13 discard 84d9ade193a [SPARK-37046][SQL] Alter view does not preserve column case discard 2b8ca4e7850 [SPARK-36798][CORE] Wait for listeners to finish before flushing metrics discard 80c025038ba [SPARK-36717][CORE] Incorrect order of variable initialization may lead incorrect behavior discard 544f12e8c08 [SPARK-36919][SQL] Make BadRecordException fields transient discard ccc92aa51e7 [SPARK-37556][SQL] Deser void class fail with Java serialization discard 0d2eee5a285 [CARMEL-5852] CHECK constraint release (#882) discard 1578ee28720 [CARMEL-5867] ShowCreateTableCommand and DescribeTableCommand miss nullable information (#880) discard 4c840e36ee5 [CARMEL-5868] Backport SPARK-37199: Add deterministic field to QueryPlan (#879) discard c9524eece44 [Minor]Bug fix for driver env config (#881) discard e443776d9e7 [CARMEL-5861]Reduce references to LiveExecutionData in HiveThriftServer2Listener (#876) discard f907c133d79 [CARMEL-5804] Enable NOT NULL definition for normal parquet table - Fix (#875) discard 7a0764736ee [CARMEL-5861] Limit the max size of statement and executePlan (#874) discard 5e7ce1be507 Log error message instead of stopping spark context (#872) discard 651ea9972b7 [MINOR] Add close useless sessions through Hermes console to error message (#869) discard 24a9955d937 [CARMEL-5846] Disable Bloom filter pruning if filtering side has scalar subquery (#868) discard c83a4619679 [CARMEL-5467][FOLLOWUP]Support contains join (#867) discard c0065f56b0c [CARMEL-5804] Enable NOT NULL definition for normal parquet table (#857) discard b069b9df893 [CARMEL-5841] Limit the row reference count in Range Index (#865) discard a86d25efb1e [CARMEL-4737][follow up] Do not add lock in inmemory file index, since we may ready a empty table that we can not write discard 21fb8f5c268 [CARMEL-4737] add lock for compact transparency (#849) discard a2099bba252 [CARMEL-5467]Support contains join (#858) discard 8694675e695 [FOLLOWUP][CARMEL-5736]Add SQL command to register HDM table readiness (#859) discard bca89e22305 [SPARK-36183][SQL][FOLLOWUP] Fix push down limit 1 through Aggregate (#861) discard cdf792c9a33 [CARMEL-5736]Add SQL command to register HDM table readiness (#839) discard f7c546e805d [CARMEL-5782][SPARK-37915][SQL] Combine unions if there is a project between them (#855) discard 2b9f41e428c [CARMEL-5783] Backport [SPARK-37922][SQL] Combine to one cast if we can safely up-cast two casts discard 49585ca882a [CARMEL-5750] Refactor basic table compaction functions for Delta reuse (#841) discard 5c6ea274543 [SPARK-35185][SQL] Improve Distinct statistics estimation (#846) discard 31efe91c2ed [CARMEL-5625] Support Table PARALLEL Hint in Size (#819) discard 4f09165a492 [CARMEL-5757] Avoid transform SMJ to BHJ in AQE if rows > 50000000 detected (#844) discard 0363b2c013a [CARMEL-5733] repartition number can be very small for small table with huge number of small files (#842) discard fffcdce6e13 [CARMEL-5758] Download support keep origin data type (#843) discard d30861970d4 [CARMEL-5708] Runtime filter result may not be correct because of weakReference Broadcast (#835) discard e0ebba62e27 [CARMEL-5672] Use hive-service-rpc dep instead of inline the generated code (#828) discard 40a1c9627d9 [CARMEL-5668] Only show optimized logical plan with stats, physical plan and rules time (#833) discard 0f5c22224e5 [CARMEL-5677][FollowUp]Set the class log level on demand (#834) discard 0d793a3f0bc [CARMEL-5493]Quit spark thrift server when the SparkContext is stopped (#821) discard 9afe4b96bc2 [CARMEL-5673] Track the Memory Usage of Large Objects in Driver (#829) discard e40187eba6f [CARMEL-5677]Dynamic set class log level (#830) discard 31d4a0ddc76 [CARMEL-5669] Use weakReference to store broadcast relation in driver (#827) discard 5fd4104304c [CARMEL-5661] Upgrade mysql-connector-java to 8.0.27 (#831) discard 20deccde05f [CARMEL-5643]Add rpc consumed records metrcis (#815) discard b4f7b7ad0e1 [CARMEL-5556] Invalid HDFS ACL set for empty table created by CTAS (#813) discard 82b7c866459 [CARMEL-5626] Use File Commit V2 for CTAS (#812) discard 26ae84e55da [CARMEL-5658] NullPointerException in BlockManagerMasterEndpoint.removeBlockManager (#826) discard 0e04c1e3c71 [SPARK-20977][CORE] Use a non-final field for the state of CollectionAccumulator (#824) discard f785b02f417 Revert "Revert "[CARMEL-5007] Make BlockManagerMasterEndpoint to run in multiple threads (#567)" (#822)" (#825) discard 436560b09c2 Revert "[CARMEL-5007] Make BlockManagerMasterEndpoint to run in multiple threads (#567)" (#822) discard 16a077c126b [CARMEL-5619] Port [CARMEL-3436] Mechanism to protect Carmel away from uncertified UDFs (#810) discard a0ae2cb3662 [SPARK-34622][SQL] Push down limit through Project with Join (#818) discard bc870099a4b [CARMEL-5007] Make BlockManagerMasterEndpoint to run in multiple threads (#567) discard a480eaa9516 [CARMEL-5529] ignore thread interrupted signal and do not remove the … (#795) discard 9c0d8feafa7 [CARMEL-5606] Use File Commit V2 instead of V1 for temp table creation (#808) discard d9cb9542656 [CARMEL-5566] Large result getter thread slow (#807) discard 473a503525c [CARMEL-5564] Backport [SPARK-37270][SQL] Fix push foldable into CaseWhen branches if elseValue is empty (#806) discard cb48c83cdbb [CARMEL-5554] Failed to rename FileStatus when inserting dynamic partition by quota-free committing with V1 algorithm (#797) discard 51e485a7881 [CARMEL-5527] NPE throws in HiveThriftServer2Listener (#802) discard 009e2fdc08e [MINOT] Fix String format exception (#801) discard c575b509bd0 [CARMEL-5426] Upgrade Delta version to 0.9.0.carmel0.1-SNAPSHOHT (#803) discard 8026567ce63 [CARMEL-5541] Clean blockmgr temp files when no reference (#796) discard d157adcd2b6 [CARMEL-5515] Filter push down through window if partitionSpec isEmpty (#798) discard f190f66785a [CARMEL-5526] Skip cancelled stages when calculate active tasks for queries(#792) discard a291c18b0b1 [CARMEL-5426] Upgrade Delta version to 0.8.0.carmel0.4-SNAPSHOH (#794) discard d7126e86949 [CARMEL-5504] Improve error message when creating table in non exists database discard cecda22144d [CARMEL-5514] Push down limit through JDBC relation (#788) discard 9ee791e8609 [CARMEL-5506] False positive probability must be within range (0.0, 1.0) (#787) discard 2d44e1d3b31 [CARMEL-4798] switch to admin when insert into whitelist (#781) discard 9a4ac945e53 SPARK-31809: Infer IsNotNull for join condition (#782) discard 58c8d8e6ed6 [CARMEL-5426] Upgrade Delta version in Carmel (#784) discard 9fa4561ecd3 [CARMEL-5063][FollowUp] Fix dynamic allocation issue and make stats type configurable (#783) discard 9d8e60d827f [CARMEL-5501] Update Update/DELETE is only supported with v2 tables messages (#780) discard db42727596e [CARMEL-5494] Clean physical plan information before stage execution (#779) discard 5ab2d826351 [KUDU-88] Implement SupportsReportStatistics in V1ScanWrapper for reading Kudu tables (#775) discard 81ec8a48ee3 [CARMEL-5063][FollowUp] Fix task number not accurate issue when doing dynamic allocation (#777) discard 08c28c2aa33 [CARMEL-5442] Fall back to true if InSet size exceed DYNAMIC_PRUNING_MAX_INSET_NUM (#762) discard 460dd4d1fbf [KUDU-90] DDL for carmel table should be case insensitive (#766) discard 412134fce01 [CARMEL-5426] Upgrade Delta version in Carmel discard bac5e112bd3 [CARMEL-4798][FOLLOWUP] Improve errormessage (#771) discard 8ab7c99dc36 [CARMEL-5458][FOLLOWUP]Add maxAccurateSkewedBlockNumber if shuffle blocks size skewed (#773) discard 90d8a2d05e3 [CARMEL-5400][FollowUp] Add more log for shuffle fetch request congestion control (#774) discard c7326684744 [CARMEL-5063][FollowUp] Fix dynamic allocation mechanism issue (#765) discard 589127a9c7f [CARMEL-5431] Limit the cached FileStatus size in driver memory (#767) discard d56ca0e08f5 [CARMEL-5458]Add spark.shuffle.accurateBlockSkewedFactor parameter to determine whether to report a shuffle block size(#770) discard cc1fa67c4c0 [CARMEL-5454] Tasks from some users cannot be scheduled in pymt queue (#768) discard 45b04560839 [CARMEL-5440] Correct running tasks by considering the status update delay (#758) discard 2ebf9c3ba1d [CARMEL-4798] block small file table insert and unblock using compact command (#522) discard 6617923dd04 [CARMEL-5400] Too many chunkFetchRequest make shuffle service very slow (#764) discard 5dbff328a4c [CARMEL-5437] Log sessionId in hadoop rpc caller context (#756) discard 8a8643f9b17 [CARMEL-5307][FOLLOWUP] Skip authorization check for ExplainOptimizeCommand (#760) discard 5cf793ee2ca [CARMEL-5448] Fix scala.MatchError in ExplainOptimizeCommand (#763) discard 7c2a20bb551 Revert "[CARMEL-5426] Upgrade Delta version in Carmel (#753)" (#761) discard 632ae75904c [CARMEL-5426] Upgrade Delta version in Carmel (#753) discard 93a030a3cad [CARMEL-5435] Implement bloom filter join to improve join performance (#743) discard 420362a29ad [CARMEL-4636] Add partition predicate if scan partitioned tables (#752) discard f581aa565b3 [CARMEL-5415] Use CPU time instead of system time to calculate analysis time (#754) discard 8823e715753 [CARMEL-5394] Query result not sorted as expected when spill happens (#749) discard a9acf92a76b [CARMEL-5396] Enable Range Join with Interval Overlap Pattern (#747) discard cc8f402e07d [CARMEL-5185][FollowUp] Fix disk leak issue when spill large broadcast result (#748) discard 08779be0149 [CARMEL-5333]fix NegativeArraySizeException for rownumber optimization (#744) discard 38eecd5f383 [CARMEL-5393] Don't store broadcast relation in driver's memory (#746) discard cff7f62414d [CARMEL-5347] Fix NPE issue for user fair scheduling (#745) discard 4e74261ac60 [KUDU-86] Support create a Kudu table with clause "USING carmel" (#740) discard 5b78893f94e [KUDU-37][FOLLOW-UP] Fix authorization mistake (#741) discard f0ceb923b98 [KUDU-83] Log task wait/block time (#742) discard cbba4b80bf9 [CARMEL-5185] Spill result to disk for large broadcast (#649) discard 6751b0b774b [CARMEL-5243] Support Range Join by Identifying Range Pattern from Co… (#739) discard e3446de365c [CARMEL-5370] Local HDFS cache may not be applied after upgrade to Sp… (#735) discard 6527961c290 [CARMEL-5063][FollowUp] Improve spark dynamic allocation mechanism (#736) discard 82c51caeff9 [CARMEL-5308][FollowUp] Fix minus stats issue for running sql (#737) discard db8977f6fbf [CARMEL-5356][FollowUp] Make even one partition data don't stored the whole deserialized result rows in memory (#738) discard 83c1a7c7d48 [CARMEL-5347] Fair task scheduler consider user resource usage (#734) discard 91d39457f43 [CARMEL-5063] Improve spark dynamic allocation mechanism (#699) discard 76f3f75a1ef [CARMEL-5356] Avoid too many deserialized result rows stored in driver memory (#733) discard baf20002292 [CARMEL-5319] Make BroadcastNestloopJoin interruptible (#732) discard c7e488e47ce [CARMEL-5337] Suppress logs related to session event logger (#730) discard a6b7da31ecb [CARMEL-5335] Fix shuffle service slow issue (#731) discard feec4c95714 [CARMEL-5308][FollowUp] Fix stage stats endtime and other issue (#728) discard 91a03f06cae [CARMEL-5327] Use IP as thrift server address instead of hostname (#729) discard 3c3eac47328 [KUDU-18] Refine KUDU table DDL (#726) discard 1a58c662695 [CARMEL-5308][FollowUp] Expose running sqls and stats to hermes console for monitoring (#727) discard 3a64526731d [CARMEL-4546][FOLLOWUP] Avoid unnecessary shuffle when enable bucketing (#698) discard 1609c5e9085 [followup][CARMEL-5264&CARMEL-4546]enable window function optimize & bucket compact by default (#705) discard c3e8727301c [CARMEL-5308] Expose running sqls and stats to hermes console for monitoring (#723) discard b891aa32348 [SPARK-34812][SQL] RowNumberLike and RankLike should not be nullable (#725) discard 50cc69ab99d [SPARK-36444][SQL] Remove OptimizeSubqueries from batch of PartitionPruning (#724) discard 625769f30bc [CARMEL-5307] Skip authorization check for ExplainOptimizeCommand (#710) discard 92f45b602c4 [CARMEL-5272] Improve coalesce shuffle partitions (#688) discard e8b17190e00 [CARMEL-5119] Don't block chunkfetchrequest in the chunkrequesthandler thread (#692) discard d984d93088a [CARMEL-3532] Input metrics miss for file scan (#701) discard 7dab2085624 [CARMEL-5264] improve window function (#695) discard eb39aa0ac89 [KUDU-65] Failed to push down project to a DSv2 relation(KuduRelation) in a subquery (#648) discard 037ebd18df7 [CARMEL-5268] Authorization fail throw improper error message (#690) discard 3c6ca685604 [CARMEL-5202] Backport SPARK-36086, SPARK-33225 and SPARK-36093 to fix NoSuchElementException when creating DELTA table (#664) discard 565c670babf [CARMEL-4193][FOLLOWUP]skip privilege check if authorization is disabled (#689) discard 3388aaf818c [CARMEL-5255] Update-Join and Delete-Join command parse failed (#685) discard 085e5aed99a [CARMEL-4546]support compact command for bucket table (#686) discard f9e8217470a [CARMEL-5234][FOLLOW-UP] Support Reuse BroadcastRangeExchangeExec (#684) discard 33e2c8dfe69 [CARMEL-5229]Resolve table schema with upload csv header in not case insens… (#679) discard 719d7c0f759 [CARMEL-5234] Split BroadcastRangeExchangeExec from BroadcastRangeJoinExec (#681) discard 9eb6597635d [CARMEL-5203][Follow-Up] A lot of tables missing metadata when running DESC EXTENDED (#678) discard fdca37616ed [CARMEL-5223] Backport [SPARK-36183][SQL] Push down limit 1 through Aggregate if it is group only (#680) discard afd3fd06bf0 [CARMEL-5218] Fully Support Partial Range Join (#676) discard f393e8f2d43 [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks (#665) discard d2223e7ceba [HOT-FIX] Disable Parquet not in filter push down if its values exceeds pushDownInFilterThreshold (#675) discard 9ebc4750b6e [CARMEL-5210][SPARK-28863][SQL] Make sure optimized plan will not be re-analyzed (#673) discard 6ee9c7bc809 [CARMEL-5205] Follow-up of [CARMEL-5124] Fix missing shuffle in a com… (#671) discard f81d589e8b4 [CARMEL-5171]Add scan size limitation for bucketed RDD (#642) discard 7df88cca918 [CARMEL-5173]Show correct error message if table not exists in CTAS (#666) discard 3eb33849b70 [MINOT] Fix flaky test in BucketedReadSuite (#672) discard 7576d44efb5 [CARMEL-5200]ODBC Call for Decimal Data Precision returned as zero (#667) discard e79968d48cd [CARMEL-5199][FOLLOWUP] Broadcast outer join stream side support AQE (#668) discard 206a4610125 [CARMEL-5208] Support Partial RangeJoin (#670) discard d2e5bd4b6d2 [CARMEL-5207] Carmel UT failed with newly built Delta jar (#669) discard 12eec2e7dfd [CARMEL-2514] keep the output partition name align with partition schema (#96) discard f694ae2b342 [CARMEL-5150][Follow-Up] Failed to rename FileStatus (#658) discard 1a6c71cd551 [CARMEL-5199] Broadcast the stream side of outer join (#660) discard 8c1e1a8f9ca [CARMEL-5201] Support Cross Join with Range Pattern (#661) discard f3d01b549bc [CARMEL-5198] Backport [SPARK-32792][SQL] Improve Parquet In filter pushdown (#663) discard 1eb6bbd48b0 Fix bug of single stage skewed join (#656) discard 072ffee04d9 [CARMEL-5194] Support disabling timeline in Spark UI to fix slowness of page loading (#654) discard fa7689de47a [CARMEL-5190] Enhance makeOffer lock and dump when wait too long (#653) discard 1f558536a44 [followup][CARMEL-5114]fix serialization failure for ExecutionEnd event (#645) discard 56cf4c41b62 [CARMEL-5143] Add configuration to enable final check (#646) discard de8e80ee394 [CARMEL-5155] Expose executors number to track cluster resource usage (#638) discard 2b3e5c9f5c3 [CARMEL-5150] Failed to rename FileStatus (#639) discard f3864318b4d [CARMEL-5151][SPARK-35886][SQL][3.0] PromotePrecision should not overwrite genCode discard 25fe015fea1 [Followup][CARMEL-5114]fix kryo dependency conflict (#640) discard 70d07eb3813 [CARMEL-5139]Resolve aggregate column reference in window expression (#618) discard 3cc457374e1 [CARMEL-5154] Correct the data size of range index (#631) discard d10bfa10bb7 [CARMEL-5143] Add a final check to ensure no Shuffle/Sort is missing before execution (#622) discard ca961150306 [CARMEL-5157] KafkaBatchEventProducer fail to stop (#636) discard 2ef95c8b799 [CARMEL-5114] support kryo serializer in kafka event queue (#626) discard 8fbc892b6ab [CARMEL-5153] Add more UT for Range Join and Disable Range Overlap (#630) discard b8ffd429f77 [CARMEL-5142]Bug fix for ArrayIndexOutOfBoundsException during messag… (#620) discard c9fb44ed620 [CARMEL-3990][FOLLOW-UP] Add query submitted time (#619) discard 2fb61344994 [CARMEL-5083] Support Range Join in Spark 3.0 (#611) discard 0f0edb48082 [SPARK-32038][SQL] Regression in handling NaN values in COUNT(DISTINCT) (#616) discard 822eec3b6d3 [CARMEL-5094]Backport [SPARK-33741][CORE] Add min threshold time spec… (#586) discard ed91d5076e5 [CARMEL-5138] Fix the makeBatchOffers ArrayIndexOutOfBoundsException during decomm (#617) discard 56e9db51b9d [CARMEL-1077][FOLLOW-UP] Improve bucket table scan performance, if the matched bucket number is only a little, treat the bucket table as normal table (#604) discard 5c4dcd66e9c [CARMEL-4232] Support list all tables AND their size in a specific HDM (#553) discard 10b2945ee50 [SPARK-33726][SQL] Fix for Duplicate field names during Aggregation (#607) discard 665886db27b [SPARK-32559][SQL][3.0] Fix the trim logic in UTF8String.toInt/toLong did't handle non-ASCII characters correctly (#613) discard 036d078d7b5 [SPARK-32753][SQL][3.0] Only copy tags to node with no tags (#612) discard b9173c3bbcd [CARMEL-5126] Backport [SPARK-32908][SQL] Fix target error calculation in percentile_approx() (#605) discard c3b8761c188 [SPARK-33260][SQL] Fix incorrect results from SortExec when sortOrder is Stream (#609) discard de32f22f598 [SPARK-33404][SQL][3.0] Fix incorrect results in `date_trunc` expression (#608) discard b0d1c6f3cfb [SPARK-32136][SQL] NormalizeFloatingNumbers should work on null struct (#615) discard d68bd8a9ab4 [SPARK-32764][SQL] -0.0 should be equal to 0.0 (#610) discard d6007d865ab update codestyle (#606) discard 734e7fda666 Revert "[CARMEL-5061] Backport [SPARK-35568][SQL] Add the BroadcastExchange after re-optimizing the physical plan to fix the UnsupportedOperationException when enabling both AQE and DPP" discard b25f4211c18 [CARMEL-5129]Bug fix for thread leak in JobCancellationSuite (#603) discard 62885330ac6 [CARMEL-5124] Fix missing shuffle in a complex query (#602) discard 8aa4b9a78da [CARMEL-5124] add more debug log to EnsureRequirements (#601) discard 5a8c4b3a0c2 [CARMEL-5110] Clean up quota free directory when session closed (#598) discard a42f5636f08 [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window (#597) discard 3f2eca01b99 [CARMEL-5106] Fix ReusedSubqueryExec cannot be cast to LogicalPlan (#593) discard d84d589be0b [CARMEL-5102] Support Sync Decomm Nodes to Yarn (#590) discard 8c1c43fbff3 [CARMEL-5095]Add limitation of user sql size (#587) discard de2e66f75d6 [CARMEL-5101]Do not cleanup ResultShareGroup in SparkExecuteStatementOperation (#588) discard 54c3acb2b1b [CARMEL-5061] Backport [SPARK-35568][SQL] Add the BroadcastExchange after re-optimizing the physical plan to fix the UnsupportedOperationException when enabling both AQE and DPP discard 7bc7132a92a [CARMEL-5088] "java.util.NoSuchElementException: key not found" in DAGScheduler (#589) discard 2f338fc836b [CARMEL-1077] Improve bucket table scan performance, if the matched bucket number is only a little, treat the bucket table as normal table (#591) discard 2b7e9768323 [CARMEL-5080] Optimize Kafka event serializer dispatcher (#585) discard 71d45b50f17 [KUDU-37] Support authorization of DSv2 update/delete operation & Support drop Kudu table (#584) discard 6cfe0fd36b4 [CARMEL-5078] AssertionError throws for some sql (#583) discard 68a72c508cb [CARMEL-5070] Support more accurate CPU usage calculation (#580) discard be3930160b4 [CARMEL-4037] Unnecessary sort is introduced when skew handling is applied on join with only one query stage (#581) discard 761580971de [CARMEL-5087][SPARK-33979][SQL] Order filters based on selectivity and cost (#356) discard 51fcf14b0c1 [CARMEL-5084] Backport [SPARK-34808][SQL] Removes outer join if it only has DISTINCT on streamed side (#579) discard c7afce9f429 [CARMEL-5061][FOLLOWUP] Move DynamicPartitionPruningSuiteAEOn to ExtendedSQLTest (#582) discard 596554ffe86 [CARMEL-4974][FOLLOWUP] Improve lock cententions of the TaskResult handling (#572) discard 8bd8e4a58ae [CARMEL-4959][follow up]kafka event queue improve and add config for capacity (#569) discard b2eff041e99 [CARMEL-5048] Support Decommission Nodes (#577) discard 840196dcae3 [KUDU-48] add missing configuration to publish jar (#578) discard ea2029b6ddc [CARMEL-5061] Avoid UnsupportedOperationException when enabling both AQE and DPP discard e30fa0e2554 [CARMEL-5062] Fix the AM un-expected restart due to RM failover (#575) discard ff9fc5fd810 [CARMEL-5054] Backport [SPARK-20628][CORE][K8S] Start to improve Spark decommissioning & preemption support (#574) discard e85150fb22e [CARMEL-5019] fix case mismatch for hive table (#563) discard 1bc0c8c1918 [KUDU-48] Publish Carmel-Spark to eBay repo for Kudu reference (#568) discard d27a74a95ee [CARMEL-5030] Query failed with exception 'zip RDDs with unequal numbers of partitions' (#566) discard 257fd794bcd [CARMEL-5029] Fix query plan time in ql_metrics table incorrect (#564) discard 72fe85d7b4e [CARMEL-5021] Backport [SPARK-33822] Use the CastSupport.cast method in HashJoin (#562) discard facc6c8be56 [CARMEL-4488] Misleading DSQuotaExceededException Error (#535) discard f6e1e59cf2b [CARMEL-5024] Support task set summary for topN resource usage (#560) discard d06cb31600c [CARMEL-4959] optimize produer to handle skew when serialize (#536) discard e512c9e1703 [CARMEL-5005][FOLLOW-UP] Enable executor related events when UI disabled (#559) discard 4cbf52f650e [CARMEL-5016] Fix the display problem of access non-parquet table to make jdbc checking in one execution (#558) discard cc41abbbbd3 [CARMEL-4974] Optimize task result handling logic, move the TaskSetManager status update logic to TaskResultGetter (#521) discard 3f86b58fa44 [CARMEL-5000][FOLLOWUP] Restore origin AQE config after ExplainOptimizeCommand discard ed21b2566b3 [CARMEL-3166] Fully support DPP on AE discard 3738d489276 [CARMEL-5020] Backport [SPARK-35286][SQL] Replace SessionState.start with SessionState.setCurrentSessionState (#539) discard 4a947763816 [CARMEL-5015][FOLLOW-UP]Fix shuffle metrics compute logic (#555) discard fc735def9ae [CARMEL-5015][FOLLOW-UP] Fix NPE and Concurrent issue for shuffle metrics compute (#554) discard 8f99e149d01 [CARMEL-5015][FOLLOW-UP] Fix NPE (#552) discard ea3a8a36af5 [CARMEL-5014] Fix the memory leak of InMemoryStore(kvstore) (#550) discard e29565b8843 [CARMEL-5005] Support disable Spark UI (#548) discard 884d6bb0c30 [CARMEL-5015] Move query metrics sink and shuffle size check out of S… (#551) discard 93ad9894e43 [CARMEL-4929][FOLLOWUP] Support spark driver runs as Unmanaged AM (#545) discard a5f0d5f1fea [CARMEL-5009] Backport [SPARK-33687][SQL] Support analyze all tables in a specific database (#264) discard 125891d3028 [CARMEL-4964] Make SparkSession purely serializable (#515) discard a143acdd03d [CAREML-4766][FollowUp] Close all shuffle connections in seperate threads (#544) discard 6cb318d8023 [CARMEL-4978]Retain 7 days logs on server to avoid too much disk usage (#530) discard 1f6c7295f5a [CARMEL-4987] Remove useless logical relation in InsertIntoDataSource (#541) discard 00031209c72 [CARMEL-5000] Workaround to make explain works with AQE (#543) discard c2041f8680d [CARMEL-4998] Improve Rewrite NOT IN to NOT EXISTS (#540) discard 2fc761241b4 [CARMEL-4996] Fix ExplainOptimizeCommand privilege issue (#538) discard 9d08b2e38c5 [CARMEL-4980] Cache the RejectedExecutionException if the executor pool is shutdown (#528) discard 4cf49255c98 [CARMEL-4898][follow up] reduce log size in kafka event queue (#506) discard fa757195b32 [CARMEL-4929] Support spark driver runs as Unmanaged AM (#524) discard 0c7fd6efceb [CARMEL-4880][FOLLOW-UP] Fix incorrect query state after being cancel… (#520) discard e3d6c4f76f9 [CARMEL-4900] Sql failed because of FetchFailedException (#533) discard 868c6e3ae34 [CARMEL-4886][follow up] fix npe when not get session info (#525) discard 6ebffcd331b [CARMEL-4981] Traffic shape handler thread leak in ut (#529) discard 262a0538b38 [CARMEL-4963] Support monitoring on active taskset (#512) discard bc57147175f [CARMEL-4921] Correct the skew flag when the replacement of shuffle reader skipped (#523) discard dee3d23caec [CAREML-4766] Fix slow issue when shuffle tcp connection only shared in stage (#518) discard b8a7bcfca38 [CARMEL-4188] Support sinking operation log to MySQL (#504) discard 9aa85d27e1e [CARMEL-4944] Avoid unnecessary RPC call to BlockManagerMaster when submit Jobs/Stages (#519) discard d047cd48eed [CARMEL-4944] Construct the taskMetrics early and offload the work from DAGScheduler event process loop (#516) discard fde10f2388f [CARMEL-4965] maxNumConcurrentTasks should be defined to lazy (#513) discard 49207ca9e95 [CARMEL-4955] Fix 'Automatically cleanup broadcast' flaky test in ContextCleanerSuite (#507) discard 8d5bdefc6ab [CARMEL-4944] Improve Spark DAGScheduler performance by offload heavy… (#505) discard 93d8c755725 [SPARK-35273][SQL] CombineFilters support non-deterministic expressions (#509) discard c8644981cd6 [CARMEL-4941] IllegalStateException in spark ui (#497) discard a19286a5d50 [CARMEL-4886] Fix npe when not get session info (#498) discard 995b5b64e92 [HOT-FIX] Fix scala style issue (#503) discard f36613a8d62 [CARMEL-4898] make producer multi-thread (#495) discard d622b8beb01 [CARMEL-4230] add error code to error msg (#487) discard d02442e3560 [CARMEL-4880] Fix the InterruptedException error for Cancelled query (#480) discard 0b236fa4154 [CARMEL-4916] Refine top running API to reduce potential connection leak (#494) discard 5594781393f [CARMEL-4833] Query failed due to scala.MatchError (#468) discard 7a47f76bff4 [CARMEL-4101] Close InputStream when getting all indexes (#496) discard 1c1d58cdab3 [SPARK-32234][SQL] Spark sql commands are failing on selecting the orc tables discard be8236fc54f [SPARK-34897][SQL] Support reconcile schemas based on index after nested column pruning discard f8d3726df2d [SPARK-35096][SQL] SchemaPruning should adhere spark.sql.caseSensitive config discard 9b0464568ec Backport [SPARK-34949][CORE] Prevent BlockManager reregister when Executor is shutting down (#493) discard 7ec70559244 [CARMEL-4838] Shuffle throttling tracing and tcp connection isolation (#456) discard 60d597ad6c5 [CARMEL-4914] Add a callback map to ensure the session resources closed (#488) discard 23254594e85 [CARMEL-4823] Critical queue events analysis and monitoring (#483) discard 29bbab142c8 [CARMEL-2483] Choose non-bucket side as the build side of NestLoopBroadCastJoin (#476) discard 2d1ebdfca19 [CARMEL-4884] Pypass the data columns match exception for legacy empty delta table insertion (#478) discard 770fa22a598 [CARMEL-4896] Add a dedicated event queue for query log to avoid losing query log (#482) discard b14b063d25f [CARMEL-4907] OptimizeSkewedInsert should work for delta table (#486) discard e42d1fd3aa2 [CARMEL-4874] Role grant with admin option failing on spark 3.0 (#473) discard 673bb063109 [CARMEL-4905] Remove the leaked live entities with a TTL map (#484) discard 4e1f5ec2b8b [CARMEL-4824] Cache GetLocationsAndStatus to reduce overhead of seria… (#459) discard c5a7df2ed44 [CARMEL-4895] ExternalCatalogListener make SparkSession leak (#481) discard 565bcae11a4 [CARMEL-4887] Consider driver slow impact when expire executor (#475) discard 4f9845d3708 [CARMEL-4867] Add DDLUtils.DDL_TIME into tables missed in 3.0 which may be used in Delta (#470) discard b74db14e9fc [CARMEL-4881] Fix the slowness of makeOffer in TaskScheduler (#474) discard 90c745bdb72 Backport [SPARK-34087][SQL] Fix memory leak of ExecutionListenerBus (#469) discard a6420a6fbb1 [CARMEL-3408][FollowUP] fix missing metrics (#467) discard 6f86978518c [CARMEL-4841] Fix the issue of qualify window function with group by (#462) discard 4f6c94317a3 [CARMEL-4842][CARMEL-4238][FOLLOWUP] craete table stored as parquet not work when disable non-parquet (#460) discard 1777fba64ea [CARMEL-4847] Allow create managed table with existing files (#463) discard 0a9e82ffcea [CARMEL-4816] Remove external heavy accumulator from event listener bus (#455) discard 25eaf530b55 [CARMEL-4750] Support create table like delta table (#458) discard 8760b7c7783 [CARMEL-4839] Refactor DynamicPartitionPruningSuite for DPP on AE (#457) discard 46183455f6b [CARMEL-4780] Allow disabling logs for task status update (#1748) (#449) discard 1241ab912a6 [CARMEL-2421] HostToRack cache in TaskScheduler (#1096) (#450) discard 4b38d9bc616 [CARMEL-4815] Stop JVM if spark context stopped (#447) discard 08a6e1f89f8 [CARMEL-4821] Support bad disk detection using Exception (#446) discard 31b15fdaec1 [CARMEL-1379][FOLLOW-UP] Fix ConcurrentHashMap misusage and remove some logs (#431) discard 9acdc8b734c [CARMEL-4814] Use proper file system (#444) discard efbec18ec35 [CARMEL-4834] Remove assert partition numbers in ParquetFileFormat discard 2f5053598e8 [CARMEL-4832] Backport SPARK-32884 and SPARK-32855 Improve dynamic partition pruning evaluation (#441) discard 638818a0503 [CARMEL-4812] Allow run bin/spark-sql (#440) discard dabdf3ca036 [CARMEL-3408] ListenerBus should send event through kafka in multi-th… (#427) discard e5da672aa61 [CARMEL-4806] Support check query DAG by query id (#439) discard 376f8d8fd2b [CARMEL-4805] Simplify time calculation in ListenerBus (#438) discard 61916bf3b6c [CARMEL-4787] Misleading error message for make offers in 3.0 (#437) discard e5ddf15042f CARMEL-4685: Allow admin_readonly role unblock user (#435) discard 127c4339b66 [CARMEL-4238][FOLLOWUP] Friend error message of blocking non-parquet (#434) discard 71880b18fb4 [CARMEL-4796] Fix ACL issue for CTAS of delta table with subquery (#436) discard 97b3a003843 [CARMEL-4784][CARMEL-4767] Fix create and grant role issue (#433) discard 873248121d2 [CARMEL-4701] Remove heavy monitor lock in CatalogManager (#409) discard c6b92d24fa9 [CARMEL-4759] disable set location (#425) discard bcc56b80598 [CARMEL-4624] Unblock users for zeta scheduler sessions (#406) discard 1ad883fc61c [CARMEL-4743] Support monitoring on zombie tasks (#416) discard ff7619bffcf [CARMEL-4768][CARMEL-4775] Fix NPE issue when `show grant` (#426) discard 0d4ef45e9ec CARMEL-4773: Fix role name with upper case (#423) discard 4904685635e [CARMEL-4361][SPARK-34772][SQL] RebaseDateTime loadRebaseRecords should use Spark classloader instead of context (#421) discard 14c15a4d4e0 [CARMEL-4728] Allow executing AlterTableDropPartition inside DeltaCommand (#411) discard cf688f233f2 [CARMEL-4770] Executor is removed long time ago but the Spark UI still show it is active (#422) discard 5f9771d3a49 [CARMEL-4769] Disable logSlowEvent in 3.0 (#420) discard fb91f2d08c5 [CARMEL-4753][backport][CARMEL-1671] Add Queue Info into Metrics in Spark Driver (#418) discard cfb1dee6632 [CARMEL-4729] Restrict the paths used as table location (#412) discard 7217f14cdb4 [CARMEL-4739] Avoid to use SNAPSHOT to improve download jar from eBay central repository (#414) discard d9b76d9a738 [CARMEL-3836][FOLLOW-UP] Bucket table scan outputpartitioning may be incorrect (#408) discard 554c7df3246 [CARMEL-4660] Limit max cube expression size to 12 (#407) discard 782be3a9067 [CARMEL-4679] Expose runtime metrics of query (#403) discard 7f7998efd3e [CARMEL-4674] Correct the total cores number leveraging ExecutorAllocationManager (#404) discard c555ac6a5ef [CARMEL-4664] Lazy listing support for only have dynamic partition filter (#399) discard 4a32994667c [CARMEL-4680][SPARK-30115][SQL] Improve limit only query (#405) discard ba18f5e8b0e [CARMEL-4352] forbid to set spark.sql.autoBroadcastJoinThreshold (#402) discard f1af811d46a [CARMEL-4537] Manage and expose broadcast info in spark driver (#378) discard b12bb8a36b3 [CARMEL-4425] scala.MatchError when spilling large result set (#394) discard fea03cafc82 [CARMEL-4661] Shuffle client produce too many logs (#396) discard 36138afcb8e [CARMEL-4569][DLETA] Invalidate the delta table cache when drop table (#384) discard db3c9f929fd CARMEL-4662 Download query state is incorrect. (#398) discard f2c3aa915cc [CARMEL-4667] Configurable the max length of abbreviate plan string (#400) discard b36bb22e5e0 [CARMEL-4646] [FOLLOWUP] SQL level cache for HiveExternalCatalog (#397) discard 4d6b293c6aa [SPARK-34212][SQL][FOLLOWUP] Parquet vectorized reader can read decimal fields with a larger precision (#392) discard 97c4da845fa [CARMEL-4238][CARMEL-728] Block non-parquet/orc file format data consuming with configuration (#362) discard cfa5ff9ce92 [CARMEL-4415] Task result too large make the driver full gc (#373) discard 1bf62fe2681 [CARMEL-4510] Limit ExpressionSet size to avoid driver OOM (#376) discard eb5081edaa5 [CARMEL-4185] Too many rpc fail message in driver log (#370) discard f1d0950cf73 [CARMEL-4172] Restrict max threads of spark ui (#382) discard e19afc29d3a [CARMEL-4653] Backport [SPARK-34129][SQL] Add table name to LogicalRelation.simpleString discard 719160e3999 [CARMEL-4229] Fast fail if query takes much time for analysis (#366) discard db6bd321204 [CARMEL-4202] Max task time check should only apply to active stages (#381) discard 7032cdf396c [CARMEL-4646] SQL level cache for HiveExternalCatalog (#390) discard 82158eb1fd9 [CARMEL-4652] Backport [SPARK-34515][SQL] Fix NPE if InSet contains null value during getPartitionsByFilter (#391) discard 608846639d5 [CARMEL-4011] Provide specific reason when call cancalJobGroup (#1613) (#385) discard 6a50ac192f3 [CARMEL-4344] add max split limitations for cartesian rdd (#374) discard 6f17008a6a6 [CARMEL-4208] Session may leak when open session with wrong database (#371) discard a9f83562f53 [CARMEL-4572] Backport[SPARK-34212][SQL] Fix incorrect decimal reading from Parquet files discard 6cb4aff2cd3 [CARMEL-4643] Backport [CARMEL-536][CARMEL-3295] cache broadcast configuration(#388) discard 3a3084be37b [CARMEL-4210] Improve UserResourceConsumption in TaskSetManag (#351) discard 47c6cf42a91 [CARMEL-3462][FOLLOWUP] Another lazy listing support for spark 3.0 (#386) discard 4701df54959 [CARMEL-4644] Backport [SPARK-24994] Add UnwrapCastInBinaryComparison optimizer to simplify literal types (#387) discard edce2f85357 [CARMEL-3462][FOLLOWUP] Fix failed test discard 3e8efc70e18 [CARMEL-4517] Possible memory leak for ResultTask (#377) discard 0d36230d89f [CARMEL-4637] Backport [SPARK-32788] non-partitioned table scan should not have partition filter (#380) discard 71e0dd735be [CARMEL-4214] Fix cancel/closed query marked as failed in query log (#349) discard 18575aa861d [CARMEL-4112] Spark may launch too many jobs for a query with a limit… (#346) discard d96d144649b [CARMEL-4258] Add max split number per skewed partition (#344) discard 00fcfa90090 [CARMEL-3462] Another lazy listing support for spark 3.0 (#365) discard 18898f5c132 [CARMEL-4598] Refine Context Cleaner to improve the clean performance (#369) discard fd647741c63 [CARMEL-4349] Configurations set via JDBC connection string cannot be tracked (#367) discard 1620e07eb8f CARMEL-4512] Add session and query info when expose top resource usage (#368) discard d0c44dea518 [CARMEL-4339] Revert "[SPARK-31147][SQL] Forbid CHAR type in non-Hive-Serde tables" (#317) discard e57e72ec371 [CARMEL-4620] Backport [SPARK-32717] Add a AQEOptimizer for AdaptiveS… (#364) discard 37617b6ddc7 [CARMEL-4616] Upgrade Avro to 1.10.1-carmel1-SNAPSHOT (#363) discard d1265496cdc [CARMEL-2402] Task thread/FileScan threads are hung for ever for waiting for BlockInfoManager write lock (#262) discard 6590b1c3af8 [CARMEL-4497] Backport [SPARK-33612] and [SPARK-33621] (#359) discard 827e10d5139 [CARMEL-4614] Infer filters from DPP (#361) discard 0527452cbb1 [SPARK-33306] Timezone is needed when cast date to string (#360) discard e59d53445d5 [CARMEL-4449] Refactor code for compact table command (#358) discard 826bd65706a [CARMEL-4592] Prevent user from repairing a table with too many partitions (#353) discard c30cd21209b [CARMEL-4560] Support creating delta table on existing data (#352) discard b29decc56cc [CARMEL-3675][FOLLOW-UP] Add metric of total task number and shuffle read size (#348) discard 87b7510a0f8 [CARMEL-4287][CARMEL-4288] Detect Join/Window function data skew discard e6b5a471595 [CARMEL-4604] Backport [SPARK-34342] Format DateLiteral and TimestampLiteral toString discard 1e79ea819ad [CARMEL-4333] Runtime filter for spark 3.0 (#354) discard 87f44a2cc51 [CARMEL-636] Invalid url viewfs in Hive UDF (#333) discard dff3ed9623f [CARMEL-2543][FOLLOW-UP] Fix memory leak (#345) discard c2a0bd41909 [CARMEL-3594][CARMEL-3652][CARMEL-4519] Improve broadcast stability (#330) discard b7c88987de3 [CARMEL-3279] Add config to set max result size in session level (#332) discard 735aa81fb34 [CARMEL-4571] Backport [SPARK-32817] DPP throws error when broadcast side is empty (#340) discard ff6ce3254a7 [CARMEL-4568] Backport [SPARK-26345] Parquet support Column indexes (#339) discard d455863f63f [CARMEL-4308] Another support Dynamic Partition Pruning in Adaptive Execution (#334) discard d52e9ef8ce3 [CARMEL-2631] Add root cause information to query log when query error (#338) discard df1ee2abbad [CARMEL-4387] Fix toDnf java.lang.StackOverflowError (#336) discard 494e5609cd8 [CARMEL-3278] Fix AvroTypeException issue (#331) discard f1884e0e406 [CARMEL-2814] enhance log to find sql with stage easily (#328) discard a23ae52808f [CARMEL-4225][SPARK-32680][SQL][3.0] Don't Preprocess V2 CTAS with Unresolved Query (#329) discard dcf11fe7dc2 [CARMEL-4463] Backport [SPARK-32816] Planner error when aggregating multiple distinct DECIMAL columns (#313) discard a74906b91b4 [CARMEL-4499] Backport [SPARK-28067][SPARK-32018] Fix UnsafeRow set overflowed decimal (#325) discard dba6e1d5986 [CARMEL-4470][SPARK-29166][FOLLOWUP] Correct the max total file limitation checker (#320) discard edc0c5c4eb1 [CARMEL-3913] qualify row_number() over [window] = 1 can stop early (#240) discard f0393d552cb [CARMEL-4490] Backport [SPARK-34119] Keep necessary stats after partition pruning (#323) discard 8daa7907e04 [CARMEL-4487] Backport [SPARK-32761] Planner error when aggregating multiple distinct Constant columns (#322) discard 0e51280217a [CARMEL-4477] Backport [SPARK-34081][SQL] Only pushdown LeftSemi/LeftAnti over Aggregate if join can be planned as broadcast join discard 615eaf416a2 [CARMEL-4475] Backport [SPARK-34031][SQL] Union operator missing rowCount when CBO enabled discard f4516d551de CARMEL-4193 spark3.0 should not allow create table with location (#254) discard 06e8e39e73b [CARMEL-4468] Backport SPARK-34110 Upgrade Zookeeper to 3.6.2 (#314) discard 0f4c4adb547 [CARMEL-4351][DELTA] Delta UI collapse tables (#312) discard 9646485df9d [CARMEL-4455] Backport [SPARK-33428] Conv UDF use BigInt to avoid Long value overflow discard e7439b05e9c [CARMEL-4286][CARMEL-4380][FOLLOWUP] Simplify conditional in predicate should consider deterministic discard 1b3a95d55d1 [CARMEL-4409] Make subtract date backward compatible with spark2.3 (#302) discard 271a6a9286f [CARMEL-4419] Broadcast exchange job leak when statement canceled (#306) discard 5a03e5b6c60 [SPARK-33954][SQL] Some operator missing rowCount when enable CBO (#308) discard a38a6d57f73 [CARMEL-3685][FOLLOWUP] Add SQL rewrite suggestion in exception message for QUALIFY keyword (#304) discard ba19d183c2a [CARMEL-4122] Backport [SPARK-27194][SPARK-29302][SQL] Fix commit collision in dynamic partition overwrite mode (#300) discard 119731b9d55 [CARMEL-456] condense precision for decimal expression to improve aggregate performance (#298) discard 111377cce18 [CARMEL-2758][CARMEL-3256] Adjust the max file number when AE enabled (#301) discard 6aa74a4c7a0 [MINOR] mute annoying logs (#303) discard 6b1c0fb04b1 [CARMEL-1772] Make the same SQL could share the result of running SQL… (#297) discard 7465dbda6fb [CARMEL-4308] Support Dynamic Partition Pruning in Adaptive Execution (#282) discard 93f6a564bbf [CARMEL-3462] Lazy Listing Support (#284) discard 14886e8ae6c [CARMEL-4393][SPARK-34000] fix NoSuchElementException in stageAttemptToNumSpeculativeTasks (#299) discard 7758f64d4e7 [CARMEL-4366] Fix the indexing pruning issue due to empty iterator (#295) discard 708aa8fdac5 [CARMEL-3014] Fix query failed with exception 'zip RDDs with unequal numbers of partitions (#290) discard 5d0914ea705 [MINOR] Mute annoying logs (#292) discard 9d207f327eb CARMEL-4207 State in query log incorrect (#287) discard f4ebeb35fa7 CARMEL-4154 spark3.0 have no queue info in queryLog exception (#294) discard e567be3abb3 [CARMEL-1626] view output resolved from sql text instead of dataType stored in view definition (#1023) (#293) discard e2faf1a56c1 [CARMEL-4286][CARMEL-4380] Backport SPARK-33910 Simplify/Optimize conditional expressions (#291) discard 7811c34d1f2 [CARMEL-3356][FOLLOW-UP] Order by may introduce additional shuffle (#289) discard 195104cfce1 [CARMEL-2475] Long running query in cac queue, table partitioned by Id (#278) discard 46b5e49dc65 [CARMEL-4348] UnsupportedOperationException in Spark 3.0 (#288) discard ed36c361de7 [CARMEL-3836][FOLLOW-UP] Bucket table scan outputpartitioning may be incorrect (#286) discard 69e139edb5e [CARMEL-4175][DELTA] Add a conf to allow creating V2Table with sort columns (#285) discard 85a9b9d861e [CARMEL-4276][FOLLOWUP] Refactor the ResolveInsertInto to support user specified columns (#283) discard 5bad04ad89e [CARMEL-4207] State in query log incorrect (#247) discard 094b6339e6d [CARMEL-4323] AQE should work for basic insert/update/delete operations (#280) discard a705a31f0af [CARMEL-4333] Runtime filter (#275) discard 337a1e225bb [CARMEL-4321] Backport [SPARK-33273] Fix a race condition in subquery execution discard 8f966305bd5 [CARMEL-4317] Backport [SPARK-33582][SPARK-33581] Hive Metastore support filter by not-equals (#277) discard 7c1889d9cee [CARMEL-4277] Fix the incorrect error message for scan limit (#270) discard 0df38c51b2e [CARMEL-4276][DELTA] ResolveInsertInto should consider the specified columns (#274) discard d1c4d0e90a8 [CARMEL-4149] Fix make offer issues and add more log (#271) discard 48d8522e54d [CARMEL-4273]Always call alterTableStats to create a hive metastore e… (#268) discard e2043ac9fb6 [SPARK-32792][SQL] Improve InSet filter pushdown (#273) discard dc78eae7870 [CARMEL-3375] Introduce spark.sql.arithmeticOperations.failOnOverFlow (#142) discard 645184b3a25 [CARMEL-4270][DELTA] Add SkipOptimizingSubquery node for Delta command (#269) discard 38d10cb4e84 [CARMEL-4237][FOLLOWUP] Fix bug in rest api for refresh metadata (#267) discard 6448af7ae22 [CARMEL-4109][DELTA] Assignment should only be resolved with target table (#266) discard 591d04e57f2 [CARMEL-4251] Backport [SPARK-33617] Add default parallelism configuration for Spark SQL queries (#265) discard d524c259292 [CARMEL-1376][CARMEL-4216] Limit the max numbers of tasks that one stage could generat (#252) discard 117ea1d2135 [CARMEL-4136] backport [SPARK-33183][SPARK-33472] Fix Optimizer rule EliminateSorts and add a physical rule to remove redundant sorts (#261) discard 0d5550d7879 [CARMEL-2353] Read broadcast take too much time (#256) discard 66474aebed3 [CARMEL-3719] OOM in sort merge join (#125) discard d8129bb9415 [CARMEL-4237][FOLLOWUP] Fix bug in rest api for refresh metadata (#263) discard e7546dfb5f2 [CARMEL-3965] Add another max result size threshold for non-spillable result (#260) discard 87e425f1c7e [CARMEL-3130] Better fetch fail handling and MapOutputTracker improvement (#257) discard d0d0195497a [CARMEL-4239] Backport [SPARK-32276] Remove redundant sorts before re… (#259) discard fc75a3e048f [CARMEL-2996] check if file exists earlier before response in external shuffle service to decrease channel (#182) discard 41cc3437456 [CARMEL-4237] Fix bug in rest api for refresh metadata (#258) discard e40f92954ff [CARMEL-2976] More logs to track shuffle output information (#255) discard 9315340bcd1 [SPARK-33026] Add numRows to metric of BroadcastExchangeExec (#158) discard fc6835d0ac7 [CARMEL-3831] Enable kill executor in UI (#170) discard 6870e40de02 catch hdfs token exception (#115) discard b63b10f1bf9 [CARMEL-3490] Provide json support for filter function (#253) discard 8ab5e748ae7 [CARMEL-4014] Fix query failed with exception 'zip RDDs with unequal numbers of partitions' (#250) discard 4b3d3aaa55c [CARMEL-2462] Prevent user from dropping a table with too many partitions (#139) discard cb2c4ba23a1 [CARMEL-1379][FOLLOW-UP] Remove session after session closed (#246) discard 342e7c215a1 [CARMEL-3245] More general Skew Join pattern match (#230) discard d45cf1eb4cc [CARMEL-3356] Support some AE optimizations even it may introduce new… (#95) discard 1e495e73c56 [CARMEL-4213][SPARK-30768][SQL] Constraints inferred from inequality attributes (#249) discard 323037f89b9 [CARMEL-2445][FOLLOWUP] Remove duplicate setNumOutputRows (#245) discard 7c99bf18e95 [CARMEL-3282] admin readonly (#244) discard 8474661370d [CARMEL-4058] Workload Circuit Breaker - Block user if task max time … (#239) discard 02f13d78bde [CARMEL-2543] Do not submit task on empty partitions (#202) discard d1f6f684028 [CARMEL-4003] Configure query audit log with log4j (#223) discard 23d6f975e93 [CARMEL-3438] Add expansion check in SortMergeJoinExec (#242) discard 0a5fa4f44f7 [CARMEL-4144] Backport [SPARK-32879][SPARK-33140][SPARK-32056]SPARK-33432][SPARK-33389] (#229) discard 0b96aced21f [CARMEL-3694] compressionFactor takes effect with stats (#227) discard c00e06d8d20 [CARMEL-4194] Backport [SPARK-33477][SQL] Hive Metastore support filter by date type (#241) discard 41023b6c961 [CARMEL-1379][FOLLOW-UP] Fix download NPE (#238) discard fb189a9b0f6 [CARMEL-4017] Restrict on total broadcast size per query (#232) discard a374aa2b4b6 CARMEL-3905 too many RPC calls to hadoop nn when reading index (#1611) (#237) discard 553afb1ce85 [CARMEL-3728] Check memory in TaskReaper Support (#128) discard 68d7eaddb36 [CARMEL-3821] Expose basic queue usage info to zeta (#224) discard 5bda62abcda [CARMEL-3888] Zombie tasks can not be cancelled and killed, those tasks might consume and waste system resources (#225) discard 227cb39833f [CARMEL-3765] Check BDP 'Top Resource Usage Jobs' Active Task accuracy (#233) discard 8115a6737fb [CARMEL-4140] Implement built-in LikeAny and LikeAll (#213) discard 644109039fb [CARMEL-4138] Port latest download to 3.0 (#210) discard 1eb1bfb1e68 [CARMEL-3638] Work queue metrics (#231) discard 81d0a5425a6 [CARMEL-3940] Add session start/close information to query log (#222) discard 0a012451786 [CARMEL-4164] Fix query failed with IllegalArgumentException after Rebucket enabled (#236) discard e9cbab90c51 [CARMEL-3285] Forbid creating function with a static configuration (#234) discard 512c51b54d3 [CARMEL-2445] Upgrade Thrift to Support Modified Rows and Query Progress (#219) discard 9301134e4e6 [CARMEL-3784][DELTA] Update external heavy accumulators before they entering into listener event loop (#145) discard 7e538219ac3 [CARMEL-3673][FOLLOW-UP] remove flaky UT (#235) discard 2a0f1bb346f [MINOR] Fix the dependency of viewpoint (#197) discard 9b91aa2d8ed [CARMEL-3716] Use partitions instead of getPartitions to avoid re-fetch partition (#226) discard ea7c6086dee [CARMEL-3697] Task Summary Support in TaskScheduler (#228) discard f65adc3385d [CARMEL-3657] Add query metrics in query log (#218) discard 5e5702da2d9 [CARMEL-3673] Limit the max shuffle data size in a single query (#220) discard 02e06ae9d72 [CARMEL-2969] Auto retry on FileNotFoundException (#215) discard 60fb774a167 [CARMEL-4133] Backport [SPARK-33473] Extend interpreted subexpression elimination to other interpreted projections (#216) discard caa3aac162e [CARMEL-4139] Support GROUP BY use Separate columns and CUBE/ROLLUP discard 9a7c9d02846 [CARMEL-4141] Backport [SPARK-32948][SQL] Optimize to_json and from_json expression chain (#217) discard d502a60cd5f [CARMEL-3225] Support AE skew join with Union (#214) discard 0c2070cff9b [CARMEL-4132] Backport [SPARK-33427] Add subexpression elimination fo… (#211) discard 40bd447b655 [CARMEL-4137] Backport [SPARK-27421][SQL] Fix filter for int column and value class java.lang.String when pruning partition column (#209) discard 3a3656de53e [CARMEL-4126] Backport [SPARK-32332] Support columnar exchanges (#208) discard a11591bc588 [CARMEL-4123]Backport [SPARK-32430] Extend SparkSessionExtensions to inject rules into AQE query stage preparation (#207) discard d3d218b94d0 [CARMEL-4118] Backport [SPARK-33416][SQL] Avoid Hive metastore stack overflow when InSet predicate have many values discard f72d7c39f12 [CARMEL-4114] Bckport [SPARK-33458][SQL] Hive partition pruning support Contains, StartsWith and EndsWith predicate discard 05ac4f395ec [CARMEL-4114] Backport [SPARK-33385][SQL] Support bucket pruning for IsNaN discard b109b4e31bb [CARMEL-4113] Backport [SPARK-33372][SQL] Fix InSet bucket pruning discard 76790453bf6 [CARMEL-3836] Add config to dynamically determine whether to apply bu… (#190) discard 44e54647970 [CARMEL-4099] Backport [SPARK-32740] Refactor common partitioning/distribution logic to BaseAggregateExec (#201) discard 813f33ff865 [CARMEL-4094] Backport [SPARK-33302] Failed to push down filters through Expand (#198) discard 38fbcd80d17 [CARMEL-4100] Backport [SPARK-32031] Fix the wrong references of PartialMerge/Final AggregateExpression (#200) discard edd6f98ff2e [CARMEL-4086] Backport [SPARK-33337][SQL] Support subexpression elimination in branches of conditional expressions discard 55cd8d1310f [CARMEL-3917] Backport [SPARK-31253] Add metrics to AQE shuffle reader (#193) discard 68b8217f658 [CARMEL-3945] Extend Show Session Command to include queue info (#189) discard 9d777e423bf [CARMEL-4056]Backport [SPARK-33290][SQL] REFRESH TABLE should invalid… (#192) discard bdd337c52db [CARMEL-4035] fix for missing output for qualify (#191) discard 738cd6af078 CARMEL-3863 only allow admin to create/rebuild index (#188) discard c3ae3bc3787 [CARMEL-3985] Improve CoarseGrainedSchedulerBackend performance when there are lots of events Support (#178) discard c94b6b25d0c [CARMEL-3851] Upload not work in delta table (#187) discard 3533fdc9298 [CARMEL-3474] BuiltIn UDF java_method() is dangerous (#173) discard 1da3aaa1819 CARMEL-3834 make mysql-connector included in the package (#155) discard a8158dbf14b [CARMEL-3862][FOLLOWUP] Make HiveClientImpl share nothing (#181) discard 0d724433a72 [CARMEL-3853][DELTA] Rowcount for insert, update & delete (#177) discard ea72be6a708 [CARMEL-3513] fix flaky CompactTableSuite (#185) discard 5df38860dd5 [CARMEL-3941] Backport [SPARK-29606][SQL] Improve EliminateOuterJoin performance discard b4a0e1d87de [CARMEL-3986] Avoid sending many heartbeats frequently after recovering from cpu pressure Support (#179) discard 6b66a100e84 [CARMEL-3494] Parquet read performance improvement (#123) discard 4d3f133d995 [CARMEL-3983] CollectAsIterator when listing files in job Support (#175) discard 1bff7c00290 [CARMEL-3962]Add api to export driver thread dump (#171) discard 372a98b5e99 [CARMEL-3963] Thread dump when driver crash due to RpcTimeout in DAGScheduler (#172) discard 7a2a964191a [CARMEL-3150] Integration with admin and metadata cache notification (#21) discard df12d154a03 [CARMEL-3862][SPARK-32118][SQL] Use fine-grained read write lock for each database in HiveExternalCatalog (#166) discard b471b5a4ca0 [CARMEL-3950][CARMEL-3916][CARMEL-1297] Add precheck for data writing command before AQE (#168) discard 46b7d5cd7b3 [CARMEL-3833] enable rule stats on UI (#163) discard a87c1fd7d38 [CARMEL-3847] Rename temporary table name should case insensitive (#169) discard de02dd64cff [CARMEL-3797]Backport [SPARK-32730][SQL] Improve LeftSemi and Existence SortMergeJoin right side buffering discard c7db97f538c [CARMEL-3871] Missing local number files discard 4f27c0f9d9c [CARMEL-3912] Create temporary table without using should use parquet by default (#160) discard 47dd50ff69d [CARMEL-3751][FOLLOWUP][DELTA] Check ACL for creating data source V2 table (#164) discard d3422b16208 [CARMEL-3499][FOLLOWUP] Change vCores to 1 in ContainerRequest for hadoop limit (#159) discard 6799b2747d2 [SPARK-33018][SQL] Fix estimate statistics issue if child has 0 bytes (#157) discard c7e7b8945d2 [CARMEL-3856] Backport [SPARK-32594] Insert wrong dates to Hive tables (#156) discard 7ca49df7414 [SPARK-32914][SQL] Avoid constructing dataType multiple times (#151) discard 119ba4fc71b [CARMEL-3796] Backport [SPARK-32629][SQL] Track metrics of BitSet/OpenHashSet in full outer SHJ (#148) discard 8f5c5745dee [CARMEL-3463] Transparent Decryption Support (#70) discard 101a0b38c65 [CARMEL-3668] Implement the equivalent TD functions for get the current authorized user and session user Support (#153) discard afe897297e0 [CARMEL-1571][FOLLOWUP] Partially revert CARMEL-1571 (#154) discard 71e4fdc658a [CARMEL-3812][SPARK-33014][SQL] Support multiple bucket columns in DataSourceV2 table (#150) discard 119a70f8db8 [CARMEL-3158][FOLLOW-UP] Fix metric key not found exceeption discard 0094014e424 [CARMEL-3572] reduce information in SparkListenerThriftServerQueryExit event (#138) discard a2bd457acdd [CARMEL-1571] Disable select from PATH by default (#79) discard 6d49b429bb6 [CARMEL-1206][SPARK-29166][SQL] Add parameters to limit the number of dynamic partitions for data source table (#102) discard 1859606f3fd [CARMEL-3803] Backport [SPARK-32767] Bucket join should work if spark.sql.shuffle.partitions larger than bucket number discard b7a6b8907db [CARMEL-3791] Backport [SPARK-32705][SQL] Fix serialization issue for EmptyHashedRelation (#147) discard 640ad31642d [CARMEL-3790] Backport [SPARK-32649][SQL] Optimize BHJ/SHJ inner/semi join with empty hashed relation (#146) discard 204a64c3d21 [CARMEL-3789] Backport [SPARK-32678][SQL] Rename EmptyHashedRelationWithAllNullKeys and simplify NAAJ generated code (#144) discard 7b7fbffe907 [CARMEL-3170][FOLLOWUP][DELTA] Add a unit test for creating bucket table (#143) discard 7f3543f8bf4 [CARMEL-3170][FOLLOWUP][DELTA] Delta table metrics (#137) discard 2dea9061189 [CARMEL-3170][FOLLOWUP] Move DeltaSQLQuerySuite to slow test (#140) discard c8f9b5c268e [CARMEL-3731]Backport [SPARK-32399] Support full outer join in shuffled hash join (#136) discard d60d7b5a32e [CARMEL-3158] HDFS cache and other remote HDFS support changes (#117) discard 67bf5420bfd [CARMEL-3513] Add compact table command (#22) discard 8ba435471e0 [CARMEL-3741] Backport [SPARK-32706][SQL] Improve cast string to decimal type (#133) discard 8f787c07820 [CARMEL-3752] Backport [SPARK-31511] Make BytesToBytesMap iterator() thread-safe (#135) discard cb2b033b8fa [CARMEL-3751][DELTA] ACL for delta (#134) discard 839062c80cb [CARMEL-3556] Backport [SPARK-32573] Anti Join Improvement with EmptyHashedRelation and EmptyHashedRelationWithAllNullKeys (#131) discard ac6ad14b680 [CARMEL-3729] Do not re-register if executor is recently removed Support (#129) discard 05035e40cb5 [CARMEL-3170][DELTA] Delta supports SparkSQL 3.0 (#130) discard a254d0a1b5c [CARMEL-3687] view database discard 82fdbc40049 [CARMEL-3739]Backport [SPARK-31865][SQL] Fix complex AQE query stage … (#132) discard 747e8dc4f72 CARML-3709 Add more info to query log (#121) discard 4adeaed04cf [CARMEL-3727]Keep necessary stats after partition pruning (#127) discard f26a760a6a1 [CARMEL-3725]Add stats including row count for more operators (#126) discard 96e5972deb2 [CARMEL-3172] Upload Support (#62) discard 4b05d88ffaa [CARMEL-3717] Update histogram height value if the number of rows expanded for Join operation (#124) discard d28531765ad [CARMEL-3698] implement deny for auth (#118) discard 780816ab719 [CARMEL-3685] implement qualify (#112) discard a5bfe3bf9a1 [CARMEL-3713]Backport [SPARK-32615] Fix AQE aggregateMetrics java.util.NoSuchElementException (#122) discard 614fa94a8a1 [CARMEL-3601] Fix remote table path based on database location (#68) discard 105a94e6ceb [CARMEL-3710]Backport [SPARK-31967] Loading jobs UI page takes 40 seconds (#120) discard a228a49d5e9 [CARMEL-3667] Show Session Command Support (#105) discard 20617eefa67 [CARMEL-3580] Backport [CARMEL-1603]ViewPoint - RunTime Query Glance (#59) discard a40575297a6 [CARMEL-3666] Zombie Stage/Tasks Check Support (#104) discard beb8f1952db [CARMEL-3700][FOLLOWUP] Add ExtendedSQLTest to ParquetV1FilterSuite and ParquetV2FilterSuite discard e6942fc609c [CARMEL-3362] Enable Query Log in spark3.0 (#25) discard b6feb5a46af [CARMEL-3700] Port community spark/master/dev to support test with sbt (#116) discard 7716b6cbe00 [CARMEL-1599][SPARK-30069][CORE][YARN] Clean up non-shuffle disk bloc… (#80) discard a2ad88b36c3 add executor id and detail info into task thread name (#111) discard 6c46f9a8d2d [MINOR] Fix sbt build missing httpclient issue: (#107) discard 92bbafdda56 [CARMEL-3671][FOLLOWUP] Make spark.sql.hive.metastore.jars support hdfs path (#113) discard 844898b6ec4 [CARMEL-3602][FOLLOWUP]HiveThriftServer kerberos login should use the… (#109) discard b2edbe6f423 [CARMEL-3671] Make spark.sql.hive.metastore.jars support hdfs path (#108) discard 3c7967e2153 [CARMEL-3682] Add missing dependency for log4j-extras (#110) discard 07b46333007 Revert "[CARMEL-2462] Prevent user from dropping a table with too many partit… (#72)" (#106) discard b519327ecb9 [CARMEL-2813] index upgrade 3.0 - index metrics improvement discard 0c18c1a1a71 [CARMEL-3645] Scan size limit support (#91) discard 7aa305b9f62 [CARMEL-2462] Prevent user from dropping a table with too many partit… (#72) discard abb9da52c46 [CARMEL-2480] Add restriction on the bucket number of a DDL (#71) discard a3d02bc859e [CARMEL-1106][CARMEL-1191] Tune target shuffle partition size to acce… (#89) discard b7616699f13 [CARMEL-2718] Need respect case sensitive configuration when parsing bucket information (#98) discard 79b6fbf2334 [CARMEL-1379] User Level Resource Management Phase (#100) discard 8182cdaf74d [CARMEL-1732] Create temporary table should not check the database privilege (#84) discard e5be66bc11a [CARMEL-3602] [FOLLOWUP]HiveThriftServer kerberos login should use the value from spark.kerberos.keytab (#101) discard 07ca2ae2e4b [CARMEL-3343] Clear the physical plan after the stage it belongs to finished (#92) discard bb99536e637 [CARMEL-3236] Prevent a skewed join from producing too many splits (#97) discard 1c1bd059ac3 [CARMEL-667] Bucket pruning may fail to apply for IN clause (#99) discard d58a34d0b0a [NIMOR] Fix sbt build issue: (#56) discard 49eeb1471ef [MINOR] Fix HiveThriftServer2 NPE issue (#94) discard 8ebbf539e0e [CARMEL-1199] Optimize executeTake() in SparkPlan when submit multiple jobs and but progress slowly discard 5d890840887 [CARMEL-3339] Support optimize skewed join with only one query stage (#90) discard 2768742375f [CARMEL-3602][FOLLOWUP] HiveThriftServer kerberos login should use the value from spark conf (#93) discard b1be56ce184 [CARMEL-2593] Make RowEncoder checkOverflow configurable (#85) discard f42b52b7c1a [CARMEL-3525] support spillable large result set (#31) discard ec49233a9c9 [CARMEL-3166] dpp to support AE and enhancement (#77) discard a3a9dd7f34a [CARMEL-1433] Add max expansion rate for stage with shuffle write (#87) discard 7902bc35276 [CARMEL-1631] Make executor metrics sources configurable (#73) discard b3dd4460754 [CARMEL-655] Limit return count for a single query to avoid driver OOM (#46) discard 2cc8578ccb0 [CARMEL-1661] backport and enhance [SPARK-14922][SPARK-17732][SPARK-23866] Support partition filter in ALTER TABLE DROP PARTITION and batch dropping PARTITIONS (#76) discard 6613ceb8e63 [CARMEL-3590] zeta support in viewpoint server (#64) discard b74dc7adbf2 [CARMEL-3545] Shuffle memory usage improvement - Fix NPE issue (#88) discard 8c22d1d6635 [CARMEL-3157] Index upgrade to 3.0 (#52) discard 40789c4917a [CARMEL-3152] driver scheduling enhancement (#58) discard e1ffe931f91 [CARMEL-2335] Add new operator InsertIntoDataSource for delta tables (#78) discard 2b96b3fafed [CARMEL-2897] API for adding Spark listener to custom a queue (#86) discard a889b0f143a [SPARK-32663][CORE] Avoid individual closing of pooled TransportClient (#82) discard 3a22daec0cb CARMEL-3571 Support ViewPointServer in Spark Command Builder (#54) discard 861f5043d7f [CARMEL-3620][SPARK-32659][SQL] Fix the data issue when pruning DPP on non-atomic type (#75) discard a923d889541 [CARMEL-1298][SQL] Set CREATE TABLE LIKE command file format configurable (#74) discard 45c0f60deb1 [CARMEL-3564] Disable LOAD DATA INPATH & INSERT INTO PATH discard ac70e7b2d08 [CARMEL-3523] Auto repartition when writing to a table discard 474ac1cf34c [CARMEL-3167] download support (#53) discard 9484b60c50d [CARMEL-484][FOLLOW-UP] Add more metrics (#66) discard 36a6446f08b [CARMEL-3548] Shuffle OOM error handling enhancement and slow rpc monitoring (#37) discard 258a5572ef5 [CARMEL-3602] HiveThriftServer kerberos login should use the value from spark.kerberos.keytab (#69) discard 49d71a6b917 [CARMEL-3359][SPARK-31377][SPARK-31389][SPARK-32421][SQL] Add code-gen for shuffled hash join (#65) discard 8d0dea0d9b8 [CARMEL-640] add max memory limit for result tasks (#45) discard 7c460f1fea6 [CARMEL-3535] Create temporary table like and analyze a temporary table (#61) discard 232343ed176 [CARMEL-3592][SPARK-32420] Add handling for unique key in non-codegen hash join discard bfc42d64f50 [CARMEL-3586][SPARK-32383] Preserve hash join (BHJ and SHJ) stream side ordering discard 4bcf1783584 [CARMEL-3545] Shuffle memory usage improvement (#33) discard d4d52c303b2 [CARMEL-484] Collect and expose metrics in HiveExternalCatalog to monitor the performance of HiveMetastore (#43) discard 886badefe13 [CARMEL-3529] Backport [SPARK-32437] Improve MapStatus deserialization speed with RoaringBitmap 0.9.0 (#26) discard fabbde4ab56 [CARMEL-3524] Create ReadWriteLock for each database in HiveExternalC… (#24) discard c173b47953c [CARMEL-3374] Support recursive SQL query discard 89958d571e3 [CARMEL-3574][SPARK-32330] Preserve shuffled hash join build side partitioning discard 15a9ffd1ae9 [CARMEL-3579] Backport SPARK-27142 Provide REST API for SQL information discard 49f199f5e8a [CARMEL-3372] Support column list specification in insert into statement discard 3024e43a5cf [CARMEL-3555][SPARK-32290] NotInSubquery SingleColumn Optimize discard cebd93a2280 [CARMEL-3510][CARMEL-3511] Try to replace ShuffleExchangeExec with CoalesceExec or RebucketingExec discard c1a98863dec [CARMEL-3573][SPARK-31869] BroadcastHashJoinExec can utilize the build side for its output partitioning discard 1d832f298be [CARMEL-3568][SPARK-31719] Refactor JoinSelection discard 39376528ea5 [CARMEL-3431] write to a bucket table discard c795881efe1 [CARMEL-3433] Support bucket join if bucket columns are subset of join keys discard 3e50a8a7320 [CARMEL-3558] Possible deadlock for multi-thread scan when task is killed (#38) discard 2db2ee97711 [CARMEL-3569][SPARK-30842] Adjust abstraction structure for join operators (#44) discard 174366c085a [CARMEL-3553] Add upload and download api to thrift protocol discard b1fc9069ff7 CARMEL-3537 add ut for spark-26859 (#30) discard 129e30de377 [CARMEL-3519] Viewpoint Support (#27) discard 613513800c5 [CARMEL-3560] Increase retry times to avoid flaky test discard 0ceddbf4c20 [CARMEL-3559][SPARK-25154] Support NOT IN sub-queries inside nested OR conditions discard 7663d1a9477 [CARMEL-3451][SPARK-31705][SPARK-28169][SPARK-32302] Extract additional OR clause from Join condition and push down to query discard 2a73041682e [CARMEL-3533] Make the Column Batch number configurable in VectorizedParquetRecordReader discard 13d2f2e0923 [CARMEL-3532] Enable multi-thread file scan discard e1ee4351a3f [CARMEL-3536] Ignore HiveExternalCatalogVersionsSuite (#29) discard ad6d8000a6a Backport [SPARK-30616][SQL] Introduce TTL config option for SQL Metadata Cache (#14) discard 5d43bfee5fd [SPARK-30525][SQL] HiveTableScanExec do not need to prune partitions again after pushing down to SessionCatalog for partition pruning discard 73824be915e [SPARK-32302][SQL] Partially push down disjunctive predicates through Join/Partitions discard 9059eafeb63 [SPARK-28169][SQL] Convert scan predicate condition to CNF discard 57f1613366b [SPARK-31705][SQL][FOLLOWUP] Avoid the unnecessary CNF computation for full-outer joins discard 1fa88fcdc61 [SPARK-31705][SQL] Push more possible predicates through Join via CNF conversion discard 9a229155919 [CARMEL-3477] Backport [SPARK-31047][SQL] Improve file listing for ViewFileSystem (#15) discard 104a62bb53a [CARMEL-3381] Consolidate avro format support in Hermes (#19) discard 2ca1fbffa98 [SPARK-31793][SQL] Reduce the memory usage in file scan location metadata (#8) discard dcd6c6a409e [CARMEL-3500] Catch potential exception for renew kerberos ticket (#18) discard 88e9d8b7440 [SPARK-29231][SQL] Constraints should be inferred from cast equality constraint (#13) discard e933ef6b02a [CARMEL-3499] Change vCores to 1 in ContainerRequest for hadoop limit (#17) discard 171868b0162 [CARMEL-3154] authorization (#4) discard 81d384a4c40 [SPARK-30724][SQL] Support 'LIKE ANY' and 'LIKE ALL' operators (#5) discard 91f1a3acd0e [SPARK-31220][SQL] repartition obeys initialPartitionNum when adaptiveExecutionEnabled (#7) discard adbb0533a07 [SPARK-28481][SQL] More expressions should extend NullIntolerant (#9) discard c77e263d3e9 [CARMEL-2691] Support % as well in the pattern spec for show like table 'xxx' (#10) discard 1930e50cfff [CARMEL-2691] Support % as well in the pattern spec for show like table 'xxx' (#6) discard 3e23e3d24e8 [CARMEL-3156][SPARK-32064][SQL] Supporting create temporary table (#3) discard f6570569dd3 Merge pull request #2 from xuluan/am-onyarn discard d1515a1c88d register thrift server to zookeeper discard 42beda25d54 ignore some test cases to enable ci discard 753245ec22c Delete 1.txt discard dc955ad7e6f ignore some ut discard cfa382f53fe test discard 5c309cc3e83 [CARMEL-3186] Run spark thrift server in yarn cluster mode for spark3 --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org