Script 'mail_helper' called by obssrc Hello community, here is the log from the commit of package redis for openSUSE:Leap:16.0 checked in at 2025-07-08 11:31:53 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Leap:16.0/redis (Old) and /work/SRC/openSUSE:Leap:16.0/.redis.new.7373 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "redis" Tue Jul 8 11:31:53 2025 rev:2 rq:1291047 version:8.0.3 Changes: -------- --- /work/SRC/openSUSE:Leap:16.0/redis/redis.changes 2025-07-03 10:54:17.216174949 +0200 +++ /work/SRC/openSUSE:Leap:16.0/.redis.new.7373/redis.changes 2025-07-08 11:31:53.919916292 +0200 @@ -1,0 +2,13 @@ +Sun Jul 6 14:46:25 UTC 2025 - Илья Индиго <i...@ilya.top> + +- Updated to 8.0.3 + * https://github.com/redis/redis/releases/tag/8.0.3 + * Fixed out-of-bounds write in HyperLogLog commands (CVE-2025-32023). + * Fixed retry accepting other connections even if the accepted + connection reports an error (CVE-2025-48367). + * Fixed a short read may lead to an exit() on a replica. + * Fixed db->expires is not defragmented. + * Added new WITHATTRIBS to return the JSON attribute associated + with an element. + +------------------------------------------------------------------- Old: ---- redis-8.0.2.tar.gz New: ---- redis-8.0.3.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ redis.spec ++++++ --- /var/tmp/diff_new_pack.9bl9ye/_old 2025-07-08 11:31:54.355934198 +0200 +++ /var/tmp/diff_new_pack.9bl9ye/_new 2025-07-08 11:31:54.355934198 +0200 @@ -20,7 +20,7 @@ %define _log_dir %{_localstatedir}/log/%{name} %define _conf_dir %{_sysconfdir}/%{name} Name: redis -Version: 8.0.2 +Version: 8.0.3 Release: 0 Summary: Persistent key-value database License: AGPL-3.0-only ++++++ redis-8.0.2.tar.gz -> redis-8.0.3.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/00-RELEASENOTES new/redis-8.0.3/00-RELEASENOTES --- old/redis-8.0.2/00-RELEASENOTES 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/00-RELEASENOTES 2025-07-06 13:59:42.000000000 +0200 @@ -19,6 +19,26 @@ #TSn = Time Series (https://github.com/RedisTimeSeries/RedisTimeSeries) #PRn = Probabilistic (https://github.com/RedisBloom/RedisBloom) +================================================================================ +Redis 8.0.3 Released Sun 6 Jul 2025 12:00:00 IST +================================================================================ + +Update urgency: `SECURITY`: There are security fixes in the release. + +### Security fixes + +* (CVE-2025-32023) Fix out-of-bounds write in `HyperLogLog` commands +* (CVE-2025-48367) Retry accepting other connections even if the accepted connection reports an error + +### New Features + +- #14065 `VSIM`: Add new `WITHATTRIBS` to return the JSON attribute associated with an element + +### Bug fixes + +- #14085 A short read may lead to an exit() on a replica +- #14092 db->expires is not defragmented + ================================================================================ Redis 8.0.2 Released Tue 27 May 2025 12:00:00 IST @@ -128,6 +148,8 @@ - #13966, #13932 `CLUSTER SLOTS` - TLS port update not reflected in CLUSTER SLOTS - #13958 `XTRIM`, `XADD` - incorrect lag due to trimming stream - #13931 `HGETEX` - wrong order of keyspace notifications +- #JS1337 - JSON - `JSON.DEL` emits no `DEL` notification when removing the entire value (MOD-9117) +- #TS1742 - Time Series - `TS.INFO` - `duplicatePolicy` is `nil` when set to the default value (MOD-5423) (edited) ========================================================== diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/modules/vector-sets/README.md new/redis-8.0.3/modules/vector-sets/README.md --- old/redis-8.0.2/modules/vector-sets/README.md 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/modules/vector-sets/README.md 2025-07-06 13:59:42.000000000 +0200 @@ -66,7 +66,7 @@ **VSIM: return elements by vector similarity** - VSIM key [ELE|FP32|VALUES] <vector or element> [WITHSCORES] [COUNT num] [EF search-exploration-factor] [FILTER expression] [FILTER-EF max-filtering-effort] [TRUTH] [NOTHREAD] + VSIM key [ELE|FP32|VALUES] <vector or element> [WITHSCORES] [WITHATTRIBS] [COUNT num] [EF search-exploration-factor] [FILTER expression] [FILTER-EF max-filtering-effort] [TRUTH] [NOTHREAD] The command returns similar vectors, for simplicity (and verbosity) in the following example, instead of providing a vector using FP32 or VALUES (like in `VADD`), we will ask for elements having a vector similar to a given element already in the sorted set: @@ -98,8 +98,14 @@ The `NOTHREAD` option forces the command to execute the search on the data structure in the main thread. Normally `VSIM` spawns a thread instead. This may be useful for benchmarking purposes, or when we work with extremely small vector sets and don't want to pay the cost of spawning a thread. It is possible that in the future this option will be automatically used by Redis when we detect small vector sets. Note that this option blocks the server for all the time needed to complete the command, so it is a source of potential latency issues: if you are in doubt, never use it. +The `WITHSCORES` option returns, for each returned element, a floating point number representing how near the element is from the query, as a similarity between 0 and 1, where 0 means the vectors are opposite, and 1 means they are pointing exactly in the same direction (maximum similarity). + +The `WITHATTRIBS` option returns, for each element, the JSON attribute associated with the element, or NULL for the elements missing an attribute. + For `FILTER` and `FILTER-EF` options, please check the filtered search section of this documentation. +Note that when `WITHSCORES` and `WITHATTRIBS` are provided at the same time, the RESP2 reply guarantees that the returned elements are always in the sequence *ele*,*score*,*attribs*, while RESP3 replies will be in the form *ele > score|attrib* when just one is provided, or *ele -> [score,attrib]* when both are provided, that is, when both options are used and RESP3 is used the score and attribute will be a two-items array associated to the element key. + **VDIM: return the dimension of the vectors inside the vector set** VDIM keyname diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/modules/vector-sets/commands.json new/redis-8.0.3/modules/vector-sets/commands.json --- old/redis-8.0.2/modules/vector-sets/commands.json 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/modules/vector-sets/commands.json 2025-07-06 13:59:42.000000000 +0200 @@ -291,6 +291,12 @@ "name": "withscores", "type": "pure-token", "optional": true + }, + { + "token": "WITHATTRIBS", + "name": "withattribs", + "type": "pure-token", + "optional": true } ] }, diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/modules/vector-sets/hnsw.c new/redis-8.0.3/modules/vector-sets/hnsw.c --- old/redis-8.0.2/modules/vector-sets/hnsw.c 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/modules/vector-sets/hnsw.c 2025-07-06 13:59:42.000000000 +0200 @@ -45,6 +45,7 @@ #include <float.h> /* for INFINITY if not in math.h */ #include <assert.h> #include "hnsw.h" +#include "mixer.h" #if 0 #define debugmsg printf @@ -2127,6 +2128,7 @@ * The function returns NULL both on out of memory and if the remaining * parameters length does not match the number of links or other items * to load. */ +#define HNSW_SER_WORSTLINK_MISSING UINT32_MAX hnswNode *hnsw_insert_serialized(HNSW *index, void *vector, uint64_t *params, uint32_t params_len, void *value) { if (params_len < 2) return NULL; @@ -2158,6 +2160,13 @@ uint32_t num_links = params[param_idx++]; uint32_t max_links = params[param_idx++]; + /* Sanity check: links should be less than max links and + * in general a reasonable amount. */ + if (num_links > max_links || max_links > HNSW_MAX_M*4) { + hnsw_node_free(node); + return NULL; + } + /* If max_links is larger than current allocation, reallocate. * It could happen in select_neighbors() that we over-allocate the * node under very unlikely to happen conditions. */ @@ -2185,6 +2194,10 @@ * fit more than 2^32 nodes in a 32 bit system. */ for (uint32_t j = 0; j < num_links; j++) node->layers[i].links[j] = (hnswNode*)params[param_idx++]; + + /* XXX: fix me, we need to store the worst link info in a + * backward compatible way. */ + node->layers[i].worst_idx = HNSW_SER_WORSTLINK_MISSING; } /* Get l2 and quantization range. */ @@ -2221,13 +2234,28 @@ return id; } +/* Helper for duplicated link detection in hnsw_deserialize_index(). */ +static int qsort_compare_pointers(const void *aptr, const void *bptr) { + uintptr_t a = *((uintptr_t*)aptr); + uintptr_t b = *((uintptr_t*)bptr); + if (a > b) return 1; + if (a < b) return -1; + return 0; +} + /* Fix pointers of neighbors nodes: after loading the serialized nodes, the * neighbors links are just IDs (casted to pointers), instead of the actual * pointers. We need to resolve IDs into pointers. * + * The two integers salt0 and salt1 are used to make the internal state + * of the function unguessable to an external attacker, in order to protect + * from corruptions. Show be two random numbers from /dev/urandom if possible + * otherwise can be just 0,0 if the application is not security critical and + * never processes untrusted inputs. + * * Return 0 on error (out of memory or some ID that can't be resolved), 1 on * success. */ -int hnsw_deserialize_index(HNSW *index) { +int hnsw_deserialize_index(HNSW *index, uint64_t salt0, uint64_t salt1) { /* We will use simple linear probing, so over-allocating is a good * idea: anyway this flat array of pointers will consume a fraction * of the memory of the loaded index. */ @@ -2253,12 +2281,60 @@ node = node->next; } - /* Second pass: fix pointers of all the neighbors links. */ + /* Second pass: fix pointers of all the neighbors links. + * As we scan and fix the links, we also compute the accumulator + * register "reciprocal", that is used in order to guarantee that all + * the links are reciprocal. + * + * This is how it works, we hash (using a strong hash function) the + * following key for each link that we see from A to B (or vice versa): + * + * hash(salt || A || B || link-level) + * + * We always sort A and B, so the same link from A to B and from B to A + * will hash the same. The we xor the result into the 128 bit accumulator. + * If each link has its own backlink, the accumulator is guaranteed to + * be zero at the end. + * + * Collisions are extremely unlikely to happen, and an external attacker + * can't easily control the hash function output, since the salt is + * unknown, and also there would be to control the pointers. + * + * This algorithm is O(1) for each node so it is basically free for + * us, as we scan the list of nodes, and runs on constant and very + * small memory. */ + uint64_t accumulator[2] = {0,0}; + node = index->head; // Rewind. while(node) { + uint64_t this_node_id = node->id; for (uint32_t i = 0; i <= node->level; i++) { + // Check if there are duplicated links: those are + // also corruptions of the on-disk serialization format. + if (node->layers[i].num_links > 0) { + qsort(node->layers[i].links, node->layers[i].num_links, + sizeof(void*), qsort_compare_pointers); + for (uint32_t j = 0; j < node->layers[i].num_links-1; j++) { + if (node->layers[i].links[j] == node->layers[i].links[j+1]) + goto corrupted; + } + } + + // Resolve pointers. for (uint32_t j = 0; j < node->layers[i].num_links; j++) { uint64_t linked_id = (uint64_t) node->layers[i].links[j]; + + // We can't link to our own node. + if (linked_id == this_node_id) goto corrupted; + + // Compute accumulator for reciprocal links check. + uint64_t mixed_h1, mixed_h2; + secure_pair_mixer_128(salt0, salt1, this_node_id, linked_id, (uint64_t)i, &mixed_h1, &mixed_h2); + + accumulator[0] ^= mixed_h1; + accumulator[1] ^= mixed_h2; + + // Fix links. uint64_t bucket = hnsw_hash_node_id(linked_id) & (table_size-1); hnswNode *neighbor = NULL; for (uint64_t k = 0; k < table_size; k++) { @@ -2268,19 +2344,37 @@ } bucket = (bucket+1) & (table_size-1); } - if (neighbor == NULL) { + + /* The neighbor must exist and also exist at the right + * level. */ + if (neighbor == NULL || neighbor->level < i) { /* Unresolved link! Either a bug in this code * or broken serialization data. */ - hfree(table); - return 0; + goto corrupted; } node->layers[i].links[j] = neighbor; } + + /* The worst link information was missing from older + * serialization formats. Compute it on the fly if needed. */ + if (node->layers[i].worst_idx == HNSW_SER_WORSTLINK_MISSING) { + hnsw_update_worst_neighbor(index,node,i); + } } node = node->next; } + + /* Check that links are reciprocal, otherwise fail. */ + if (accumulator[0] || accumulator[1]) goto corrupted; + + /* Everything fine. Return success. */ hfree(table); return 1; + +corrupted: + /* Some corruption error detected. */ + hfree(table); + return 0; } /* ================================ Iterator ================================ */ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/modules/vector-sets/hnsw.h new/redis-8.0.3/modules/vector-sets/hnsw.h --- old/redis-8.0.2/modules/vector-sets/hnsw.h 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/modules/vector-sets/hnsw.h 2025-07-06 13:59:42.000000000 +0200 @@ -158,7 +158,7 @@ hnswSerNode *hnsw_serialize_node(HNSW *index, hnswNode *node); void hnsw_free_serialized_node(hnswSerNode *sn); hnswNode *hnsw_insert_serialized(HNSW *index, void *vector, uint64_t *params, uint32_t params_len, void *value); -int hnsw_deserialize_index(HNSW *index); +int hnsw_deserialize_index(HNSW *index, uint64_t salt0, uint64_t salt1); // Helper function in case the user wants to directly copy // the vector bytes. diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/modules/vector-sets/mixer.h new/redis-8.0.3/modules/vector-sets/mixer.h --- old/redis-8.0.2/modules/vector-sets/mixer.h 1970-01-01 01:00:00.000000000 +0100 +++ new/redis-8.0.3/modules/vector-sets/mixer.h 2025-07-06 13:59:42.000000000 +0200 @@ -0,0 +1,106 @@ +/* Redis implementation for vector sets. The data structure itself + * is implemented in hnsw.c. + * + * Copyright (c) 2009-Present, Redis Ltd. + * All rights reserved. + * + * Licensed under your choice of (a) the Redis Source Available License 2.0 + * (RSALv2); or (b) the Server Side Public License v1 (SSPLv1); or (c) the + * GNU Affero General Public License v3 (AGPLv3). + * Originally authored by: Salvatore Sanfilippo. + * + * ============================================================================= + * + * Mixing function for HNSW link integrity verification + * Designed to resist collision attacks when salts are unknown. + */ + +#include <stdint.h> +#include <string.h> + +static inline uint64_t ROTL64(uint64_t x, int r) { + return (x << r) | (x >> (64 - r)); +} + +// Use more rounds and stronger constants +#define MIX_PRIME_1 0xFF51AFD7ED558CCDULL +#define MIX_PRIME_2 0xC4CEB9FE1A85EC53ULL +#define MIX_PRIME_3 0x9E3779B97F4A7C15ULL +#define MIX_PRIME_4 0xBF58476D1CE4E5B9ULL +#define MIX_PRIME_5 0x94D049BB133111EBULL +#define MIX_PRIME_6 0x2B7E151628AED2A7ULL + +/* Mixer design goals: + * 1. Thorough mixing of the level parameter. + * 2. Enough rounds of mixing. + * 3. Cross-influence between h1 and h2. + * 4. Domain separation to prevent related-key attacks. + */ +void secure_pair_mixer_128(uint64_t salt0, uint64_t salt1, + uint64_t id1_in, uint64_t id2_in, uint64_t level, + uint64_t* out_h1, uint64_t* out_h2) { + // Order independence (A -> B links should hash as B -> A links). + uint64_t id_a = (id1_in < id2_in) ? id1_in : id2_in; + uint64_t id_b = (id1_in < id2_in) ? id2_in : id1_in; + + // Domain separation: mix salts with a constant to prevent + // related-key attacks. + uint64_t h1 = salt0 ^ 0xDEADBEEFDEADBEEFULL; + uint64_t h2 = salt1 ^ 0xCAFEBABECAFEBABEULL; + + // First, thoroughly mix the level into both accumulators + // This prevents predictable level values from being a weakness + uint64_t level_mix = level; + level_mix *= MIX_PRIME_5; + level_mix ^= level_mix >> 32; + level_mix *= MIX_PRIME_6; + + h1 ^= level_mix; + h2 ^= ROTL64(level_mix, 31); + + // Mix in id_a with strong diffusion. + h1 ^= id_a; + h1 *= MIX_PRIME_1; + h1 = ROTL64(h1, 23); + h1 *= MIX_PRIME_2; + + // Mix in id_b. + h2 ^= id_b; + h2 *= MIX_PRIME_3; + h2 = ROTL64(h2, 29); + h2 *= MIX_PRIME_4; + + // Three rounds of cross-mixing for better security. + for (int i = 0; i < 3; i++) { + // Cross-influence. + uint64_t tmp = h1; + h1 += h2; + h2 += tmp; + + // Mix h1. + h1 ^= ROTL64(h1, 31); + h1 *= MIX_PRIME_1; + h1 ^= salt0; + + // Mix h2. + h2 ^= ROTL64(h2, 37); + h2 *= MIX_PRIME_2; + h2 ^= salt1; + } + + // Finalization with avalanche rounds. + h1 ^= h1 >> 33; + h1 *= MIX_PRIME_3; + h1 ^= h1 >> 29; + h1 *= MIX_PRIME_4; + h1 ^= h1 >> 32; + + h2 ^= h2 >> 33; + h2 *= MIX_PRIME_5; + h2 ^= h2 >> 29; + h2 *= MIX_PRIME_6; + h2 ^= h2 >> 32; + + *out_h1 = h1; + *out_h2 = h2; +} diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/modules/vector-sets/test.py new/redis-8.0.3/modules/vector-sets/test.py --- old/redis-8.0.2/modules/vector-sets/test.py 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/modules/vector-sets/test.py 2025-07-06 13:59:42.000000000 +0200 @@ -94,6 +94,7 @@ self.test_key = f"test:{self.__class__.__name__.lower()}" # Primary Redis instance self.redis = redis.Redis(port=primary_port) + self.redis3 = redis.Redis(port=primary_port,protocol=3) # Replica Redis instance self.replica = redis.Redis(port=replica_port) # Replication status @@ -184,6 +185,7 @@ # Create test instance with specified ports test_instance = obj() test_instance.redis = redis.Redis(port=primary_port) + test_instance.redis3 = redis.Redis(port=primary_port,protocol=3) test_instance.replica = redis.Redis(port=replica_port) test_instance.primary_port = primary_port test_instance.replica_port = replica_port diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/modules/vector-sets/tests/with.py new/redis-8.0.3/modules/vector-sets/tests/with.py --- old/redis-8.0.2/modules/vector-sets/tests/with.py 1970-01-01 01:00:00.000000000 +0100 +++ new/redis-8.0.3/modules/vector-sets/tests/with.py 2025-07-06 13:59:42.000000000 +0200 @@ -0,0 +1,214 @@ +from test import TestCase, generate_random_vector +import struct +import json +import random + +class VSIMWithAttribs(TestCase): + def getname(self): + return "VSIM WITHATTRIBS/WITHSCORES functionality testing" + + def setup(self): + super().setup() + self.dim = 8 + self.count = 20 + + # Create vectors with attributes + for i in range(self.count): + vec = generate_random_vector(self.dim) + vec_bytes = struct.pack(f'{self.dim}f', *vec) + + # Item name + name = f"{self.test_key}:item:{i}" + + # Add to Redis + self.redis.execute_command('VADD', self.test_key, 'FP32', vec_bytes, name) + + # Create and add attribute + if i % 5 == 0: + # Every 5th item has no attribute (for testing NULL responses) + continue + + category = random.choice(["electronics", "furniture", "clothing"]) + price = random.randint(50, 1000) + attrs = {"category": category, "price": price, "id": i} + + self.redis.execute_command('VSETATTR', self.test_key, name, json.dumps(attrs)) + + def is_numeric(self, value): + """Check if a value can be converted to float""" + try: + if isinstance(value, (int, float)): + return True + if isinstance(value, bytes): + float(value.decode('utf-8')) + return True + if isinstance(value, str): + float(value) + return True + return False + except (ValueError, TypeError): + return False + + def test(self): + # Create query vector + query_vec = generate_random_vector(self.dim) + + # Test 1: VSIM with no additional options (should be same for RESP2 and RESP3) + cmd_args = ['VSIM', self.test_key, 'VALUES', self.dim] + cmd_args.extend([str(x) for x in query_vec]) + cmd_args.extend(['COUNT', 5]) + + results_resp2 = self.redis.execute_command(*cmd_args) + results_resp3 = self.redis3.execute_command(*cmd_args) + + # Both should return simple arrays of item names + assert len(results_resp2) == 5, f"RESP2: Expected 5 results, got {len(results_resp2)}" + assert len(results_resp3) == 5, f"RESP3: Expected 5 results, got {len(results_resp3)}" + assert all(isinstance(item, bytes) for item in results_resp2), "RESP2: Results should be byte strings" + assert all(isinstance(item, bytes) for item in results_resp3), "RESP3: Results should be byte strings" + + # Test 2: VSIM with WITHSCORES only + cmd_args = ['VSIM', self.test_key, 'VALUES', self.dim] + cmd_args.extend([str(x) for x in query_vec]) + cmd_args.extend(['COUNT', 5, 'WITHSCORES']) + + results_resp2 = self.redis.execute_command(*cmd_args) + results_resp3 = self.redis3.execute_command(*cmd_args) + + # RESP2: Should be a flat array alternating item, score + assert len(results_resp2) == 10, f"RESP2: Expected 10 elements (5 items × 2), got {len(results_resp2)}" + for i in range(0, len(results_resp2), 2): + assert isinstance(results_resp2[i], bytes), f"RESP2: Item at {i} should be bytes" + assert self.is_numeric(results_resp2[i+1]), f"RESP2: Score at {i+1} should be numeric" + score = float(results_resp2[i+1]) if isinstance(results_resp2[i+1], bytes) else results_resp2[i+1] + assert 0 <= score <= 1, f"RESP2: Score {score} should be between 0 and 1" + + # RESP3: Should be a dict/map with items as keys and scores as DIRECT values (not arrays) + assert isinstance(results_resp3, dict), f"RESP3: Expected dict, got {type(results_resp3)}" + assert len(results_resp3) == 5, f"RESP3: Expected 5 entries, got {len(results_resp3)}" + for item, score in results_resp3.items(): + assert isinstance(item, bytes), f"RESP3: Key should be bytes" + # Score should be a direct value, NOT an array + assert not isinstance(score, list), f"RESP3: With single WITH option, value should not be array" + assert self.is_numeric(score), f"RESP3: Score should be numeric, got {type(score)}" + score_val = float(score) if isinstance(score, bytes) else score + assert 0 <= score_val <= 1, f"RESP3: Score {score_val} should be between 0 and 1" + + # Test 3: VSIM with WITHATTRIBS only + cmd_args = ['VSIM', self.test_key, 'VALUES', self.dim] + cmd_args.extend([str(x) for x in query_vec]) + cmd_args.extend(['COUNT', 5, 'WITHATTRIBS']) + + results_resp2 = self.redis.execute_command(*cmd_args) + results_resp3 = self.redis3.execute_command(*cmd_args) + + # RESP2: Should be a flat array alternating item, attribute + assert len(results_resp2) == 10, f"RESP2: Expected 10 elements (5 items × 2), got {len(results_resp2)}" + for i in range(0, len(results_resp2), 2): + assert isinstance(results_resp2[i], bytes), f"RESP2: Item at {i} should be bytes" + attr = results_resp2[i+1] + assert attr is None or isinstance(attr, bytes), f"RESP2: Attribute at {i+1} should be None or bytes" + if attr is not None: + # Verify it's valid JSON + json.loads(attr) + + # RESP3: Should be a dict/map with items as keys and attributes as DIRECT values (not arrays) + assert isinstance(results_resp3, dict), f"RESP3: Expected dict, got {type(results_resp3)}" + assert len(results_resp3) == 5, f"RESP3: Expected 5 entries, got {len(results_resp3)}" + for item, attr in results_resp3.items(): + assert isinstance(item, bytes), f"RESP3: Key should be bytes" + # Attribute should be a direct value, NOT an array + assert not isinstance(attr, list), f"RESP3: With single WITH option, value should not be array" + assert attr is None or isinstance(attr, bytes), f"RESP3: Attribute should be None or bytes" + if attr is not None: + # Verify it's valid JSON + json.loads(attr) + + # Test 4: VSIM with both WITHSCORES and WITHATTRIBS + cmd_args = ['VSIM', self.test_key, 'VALUES', self.dim] + cmd_args.extend([str(x) for x in query_vec]) + cmd_args.extend(['COUNT', 5, 'WITHSCORES', 'WITHATTRIBS']) + + results_resp2 = self.redis.execute_command(*cmd_args) + results_resp3 = self.redis3.execute_command(*cmd_args) + + # RESP2: Should be a flat array with pattern: item, score, attribute + assert len(results_resp2) == 15, f"RESP2: Expected 15 elements (5 items × 3), got {len(results_resp2)}" + for i in range(0, len(results_resp2), 3): + assert isinstance(results_resp2[i], bytes), f"RESP2: Item at {i} should be bytes" + assert self.is_numeric(results_resp2[i+1]), f"RESP2: Score at {i+1} should be numeric" + score = float(results_resp2[i+1]) if isinstance(results_resp2[i+1], bytes) else results_resp2[i+1] + assert 0 <= score <= 1, f"RESP2: Score {score} should be between 0 and 1" + attr = results_resp2[i+2] + assert attr is None or isinstance(attr, bytes), f"RESP2: Attribute at {i+2} should be None or bytes" + + # RESP3: Should be a dict where each value is a 2-element array [score, attribute] + assert isinstance(results_resp3, dict), f"RESP3: Expected dict, got {type(results_resp3)}" + assert len(results_resp3) == 5, f"RESP3: Expected 5 entries, got {len(results_resp3)}" + for item, value in results_resp3.items(): + assert isinstance(item, bytes), f"RESP3: Key should be bytes" + # With BOTH options, value MUST be an array + assert isinstance(value, list), f"RESP3: With both WITH options, value should be a list, got {type(value)}" + assert len(value) == 2, f"RESP3: Value should have 2 elements [score, attr], got {len(value)}" + + score, attr = value + assert self.is_numeric(score), f"RESP3: Score should be numeric" + score_val = float(score) if isinstance(score, bytes) else score + assert 0 <= score_val <= 1, f"RESP3: Score {score_val} should be between 0 and 1" + assert attr is None or isinstance(attr, bytes), f"RESP3: Attribute should be None or bytes" + + # Test 5: Verify consistency - same items returned in same order + cmd_args = ['VSIM', self.test_key, 'VALUES', self.dim] + cmd_args.extend([str(x) for x in query_vec]) + cmd_args.extend(['COUNT', 5, 'WITHSCORES', 'WITHATTRIBS']) + + results_resp2 = self.redis.execute_command(*cmd_args) + results_resp3 = self.redis3.execute_command(*cmd_args) + + # Extract items from RESP2 (every 3rd element starting from 0) + items_resp2 = [results_resp2[i] for i in range(0, len(results_resp2), 3)] + + # Extract items from RESP3 (keys of the dict) + items_resp3 = list(results_resp3.keys()) + + # Verify same items returned + assert set(items_resp2) == set(items_resp3), "RESP2 and RESP3 should return the same items" + + # Build a mapping from items to scores and attributes for comparison + data_resp2 = {} + for i in range(0, len(results_resp2), 3): + item = results_resp2[i] + score = float(results_resp2[i+1]) if isinstance(results_resp2[i+1], bytes) else results_resp2[i+1] + attr = results_resp2[i+2] + data_resp2[item] = (score, attr) + + data_resp3 = {} + for item, value in results_resp3.items(): + score = float(value[0]) if isinstance(value[0], bytes) else value[0] + attr = value[1] + data_resp3[item] = (score, attr) + + # Verify scores and attributes match for each item + for item in data_resp2: + score_resp2, attr_resp2 = data_resp2[item] + score_resp3, attr_resp3 = data_resp3[item] + + assert abs(score_resp2 - score_resp3) < 0.0001, \ + f"Scores for {item} don't match: RESP2={score_resp2}, RESP3={score_resp3}" + assert attr_resp2 == attr_resp3, \ + f"Attributes for {item} don't match: RESP2={attr_resp2}, RESP3={attr_resp3}" + + # Test 6: Test ordering of WITHSCORES and WITHATTRIBS doesn't matter + cmd_args1 = ['VSIM', self.test_key, 'VALUES', self.dim] + cmd_args1.extend([str(x) for x in query_vec]) + cmd_args1.extend(['COUNT', 3, 'WITHSCORES', 'WITHATTRIBS']) + + cmd_args2 = ['VSIM', self.test_key, 'VALUES', self.dim] + cmd_args2.extend([str(x) for x in query_vec]) + cmd_args2.extend(['COUNT', 3, 'WITHATTRIBS', 'WITHSCORES']) # Reversed order + + results1_resp3 = self.redis3.execute_command(*cmd_args1) + results2_resp3 = self.redis3.execute_command(*cmd_args2) + + # Both should return the same structure + assert results1_resp3 == results2_resp3, "Order of WITH options shouldn't matter" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/modules/vector-sets/vset.c new/redis-8.0.3/modules/vector-sets/vset.c --- old/redis-8.0.2/modules/vector-sets/vset.c 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/modules/vector-sets/vset.c 2025-07-06 13:59:42.000000000 +0200 @@ -801,8 +801,8 @@ * handles the HNSW locking explicitly. */ void VSIM_execute(RedisModuleCtx *ctx, struct vsetObject *vset, float *vec, unsigned long count, float epsilon, unsigned long withscores, - unsigned long ef, exprstate *filter_expr, unsigned long filter_ef, - int ground_truth) + unsigned long withattribs, unsigned long ef, exprstate *filter_expr, + unsigned long filter_ef, int ground_truth) { /* In our scan, we can't just collect 'count' elements as * if count is small we would explore the graph in an insufficient @@ -837,28 +837,52 @@ } /* Return results */ - if (withscores) + int resp3 = RedisModule_GetContextFlags(ctx) & REDISMODULE_CTX_FLAGS_RESP3; + int reply_with_map = resp3 && (withscores || withattribs); + + if (reply_with_map) RedisModule_ReplyWithMap(ctx, REDISMODULE_POSTPONED_LEN); else RedisModule_ReplyWithArray(ctx, REDISMODULE_POSTPONED_LEN); - long long arraylen = 0; + long long arraylen = 0; for (unsigned int i = 0; i < found && i < count; i++) { if (distances[i] > epsilon) break; struct vsetNodeVal *nv = neighbors[i]->value; RedisModule_ReplyWithString(ctx, nv->item); arraylen++; + + /* If the user asked for multiple properties at the same time using + * the RESP3 protocol, we wrap the value of the map into an N-items + * array. Two for now, since we have just two properties that can be + * requested. + * + * So in the case of RESP2 we will just have the flat reply: + * item, score, attribute. For RESP3 instead item -> [score, attribute] + */ + if (resp3 && withscores && withattribs) + RedisModule_ReplyWithArray(ctx,2); + if (withscores) { /* The similarity score is provided in a 0-1 range. */ RedisModule_ReplyWithDouble(ctx, 1.0 - distances[i]/2.0); } + if (withattribs) { + /* Return the attributes as well, if any. */ + if (nv->attrib) + RedisModule_ReplyWithString(ctx, nv->attrib); + else + RedisModule_ReplyWithNull(ctx); + } } hnsw_release_read_slot(vset->hnsw,slot); - if (withscores) + if (reply_with_map) { RedisModule_ReplySetMapLength(ctx, arraylen); - else - RedisModule_ReplySetArrayLength(ctx, arraylen); + } else { + int items_per_ele = 1+withattribs+withscores; + RedisModule_ReplySetArrayLength(ctx, arraylen * items_per_ele); + } RedisModule_Free(vec); RedisModule_Free(neighbors); @@ -878,10 +902,11 @@ unsigned long count = (unsigned long)targ[3]; float epsilon = *((float*)targ[4]); unsigned long withscores = (unsigned long)targ[5]; - unsigned long ef = (unsigned long)targ[6]; - exprstate *filter_expr = targ[7]; - unsigned long filter_ef = (unsigned long)targ[8]; - unsigned long ground_truth = (unsigned long)targ[9]; + unsigned long withattribs = (unsigned long)targ[6]; + unsigned long ef = (unsigned long)targ[7]; + exprstate *filter_expr = targ[8]; + unsigned long filter_ef = (unsigned long)targ[9]; + unsigned long ground_truth = (unsigned long)targ[10]; RedisModule_Free(targ[4]); RedisModule_Free(targ); @@ -894,7 +919,7 @@ RedisModuleCtx *ctx = RedisModule_GetThreadSafeContext(bc); // Run the query. - VSIM_execute(ctx, vset, vec, count, epsilon, withscores, ef, filter_expr, filter_ef, ground_truth); + VSIM_execute(ctx, vset, vec, count, epsilon, withscores, withattribs, ef, filter_expr, filter_ef, ground_truth); pthread_rwlock_unlock(&vset->in_use_lock); // Cleanup. @@ -904,7 +929,7 @@ return NULL; } -/* VSIM key [ELE|FP32|VALUES] <vector or ele> [WITHSCORES] [COUNT num] [EPSILON eps] [EF exploration-factor] [FILTER expression] [FILTER-EF exploration-factor] */ +/* VSIM key [ELE|FP32|VALUES] <vector or ele> [WITHSCORES] [WITHATTRIBS] [COUNT num] [EPSILON eps] [EF exploration-factor] [FILTER expression] [FILTER-EF exploration-factor] */ int VSIM_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { RedisModule_AutoMemory(ctx); @@ -914,6 +939,7 @@ /* Defaults */ int withscores = 0; + int withattribs = 0; long long count = VSET_DEFAULT_COUNT; /* New default value */ long long ef = 0; /* Exploration factor (see HNSW paper) */ double epsilon = 2.0; /* Max cosine distance */ @@ -1017,6 +1043,9 @@ if (!strcasecmp(opt, "WITHSCORES")) { withscores = 1; j++; + } else if (!strcasecmp(opt, "WITHATTRIBS")) { + withattribs = 1; + j++; } else if (!strcasecmp(opt, "TRUTH")) { ground_truth = 1; j++; @@ -1097,7 +1126,7 @@ * free slot if all the HNSW_MAX_THREADS slots are used. */ RedisModuleBlockedClient *bc = RedisModule_BlockClient(ctx,NULL,NULL,NULL,0); pthread_t tid; - void **targ = RedisModule_Alloc(sizeof(void*)*10); + void **targ = RedisModule_Alloc(sizeof(void*)*11); targ[0] = bc; targ[1] = vset; targ[2] = vec; @@ -1105,10 +1134,11 @@ targ[4] = RedisModule_Alloc(sizeof(float)); *((float*)targ[4]) = epsilon; targ[5] = (void*)(unsigned long)withscores; - targ[6] = (void*)(unsigned long)ef; - targ[7] = (void*)filter_expr; - targ[8] = (void*)(unsigned long)filter_ef; - targ[9] = (void*)(unsigned long)ground_truth; + targ[6] = (void*)(unsigned long)withattribs; + targ[7] = (void*)(unsigned long)ef; + targ[8] = (void*)filter_expr; + targ[9] = (void*)(unsigned long)filter_ef; + targ[10] = (void*)(unsigned long)ground_truth; RedisModule_BlockedClientMeasureTimeStart(bc); vset->thread_creation_pending++; if (pthread_create(&tid,NULL,VSIM_thread,targ) != 0) { @@ -1116,10 +1146,10 @@ RedisModule_AbortBlock(bc); RedisModule_Free(targ[4]); RedisModule_Free(targ); - VSIM_execute(ctx, vset, vec, count, epsilon, withscores, ef, filter_expr, filter_ef, ground_truth); + VSIM_execute(ctx, vset, vec, count, epsilon, withscores, withattribs, ef, filter_expr, filter_ef, ground_truth); } } else { - VSIM_execute(ctx, vset, vec, count, epsilon, withscores, ef, filter_expr, filter_ef, ground_truth); + VSIM_execute(ctx, vset, vec, count, epsilon, withscores, withattribs, ef, filter_expr, filter_ef, ground_truth); } return REDISMODULE_OK; @@ -1853,7 +1883,10 @@ RedisModule_Free(vector); RedisModule_Free(params); } - if (!hnsw_deserialize_index(vset->hnsw)) goto ioerr; + + uint64_t salt[2]; + RedisModule_GetRandomBytes((unsigned char*)salt,sizeof(salt)); + if (!hnsw_deserialize_index(vset->hnsw, salt[0], salt[1])) goto ioerr; return vset; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/src/ae_evport.c new/redis-8.0.3/src/ae_evport.c --- old/redis-8.0.2/src/ae_evport.c 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/src/ae_evport.c 2025-07-06 13:59:42.000000000 +0200 @@ -216,7 +216,9 @@ * the fact that our caller has already updated the mask in the eventLoop. */ - fullmask = eventLoop->events[fd].mask; + /* We always remove the specified events from the current mask, + * regardless of whether eventLoop->events[fd].mask has been updated yet. */ + fullmask = eventLoop->events[fd].mask & ~mask; if (fullmask == AE_NONE) { /* * We're removing *all* events, so use port_dissociate to remove the diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/src/anet.c new/redis-8.0.3/src/anet.c --- old/redis-8.0.2/src/anet.c 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/src/anet.c 2025-07-06 13:59:42.000000000 +0200 @@ -787,3 +787,27 @@ if (stat(filepath, &sb) == -1) return 0; return S_ISFIFO(sb.st_mode); } + +/* This function must be called after accept4() fails. It returns 1 if 'err' + * indicates accepted connection faced an error, and it's okay to continue + * accepting next connection by calling accept4() again. Other errors either + * indicate programming errors, e.g. calling accept() on a closed fd or indicate + * a resource limit has been reached, e.g. -EMFILE, open fd limit has been + * reached. In the latter case, caller might wait until resources are available. + * See accept4() documentation for details. */ +int anetAcceptFailureNeedsRetry(int err) { + if (err == ECONNABORTED) + return 1; + +#if defined(__linux__) + /* For details, see 'Error Handling' section on + * https://man7.org/linux/man-pages/man2/accept.2.html */ + if (err == ENETDOWN || err == EPROTO || err == ENOPROTOOPT || + err == EHOSTDOWN || err == ENONET || err == EHOSTUNREACH || + err == EOPNOTSUPP || err == ENETUNREACH) + { + return 1; + } +#endif + return 0; +} diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/src/anet.h new/redis-8.0.3/src/anet.h --- old/redis-8.0.2/src/anet.h 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/src/anet.h 2025-07-06 13:59:42.000000000 +0200 @@ -53,5 +53,6 @@ int anetSetSockMarkId(char *err, int fd, uint32_t id); int anetGetError(int fd); int anetIsFifo(char *filepath); +int anetAcceptFailureNeedsRetry(int err); #endif diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/src/aof.c new/redis-8.0.3/src/aof.c --- old/redis-8.0.2/src/aof.c 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/src/aof.c 2025-07-06 13:59:42.000000000 +0200 @@ -2579,9 +2579,9 @@ serverLog(LL_NOTICE, "Successfully created the temporary AOF base file %s", tmpfile); sendChildCowInfo(CHILD_INFO_TYPE_AOF_COW_SIZE, "AOF rewrite"); - exitFromChild(0); + exitFromChild(0, 0); } else { - exitFromChild(1); + exitFromChild(1, 0); } } else { /* Parent */ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/src/childinfo.c new/redis-8.0.3/src/childinfo.c --- old/redis-8.0.2/src/childinfo.c 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/src/childinfo.c 2025-07-06 13:59:42.000000000 +0200 @@ -94,7 +94,7 @@ if (write(server.child_info_pipe[1], &data, wlen) != wlen) { /* Failed writing to parent, it could have been killed, exit. */ serverLog(LL_WARNING,"Child failed reporting info to parent, exiting. %s", strerror(errno)); - exitFromChild(1); + exitFromChild(1, 0); } } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/src/cluster_legacy.c new/redis-8.0.3/src/cluster_legacy.c --- old/redis-8.0.2/src/cluster_legacy.c 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/src/cluster_legacy.c 2025-07-06 13:59:42.000000000 +0200 @@ -1253,6 +1253,8 @@ while(max--) { cfd = anetTcpAccept(server.neterr, fd, cip, sizeof(cip), &cport); if (cfd == ANET_ERR) { + if (anetAcceptFailureNeedsRetry(errno)) + continue; if (errno != EWOULDBLOCK) serverLog(LL_VERBOSE, "Error accepting cluster node: %s", server.neterr); diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/src/defrag.c new/redis-8.0.3/src/defrag.c --- old/redis-8.0.2/src/defrag.c 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/src/defrag.c 2025-07-06 13:59:42.000000000 +0200 @@ -1187,7 +1187,7 @@ static doneStatus defragStageExpiresKvstore(void *ctx, monotime endtime) { defragKeysCtx *defrag_keys_ctx = ctx; redisDb *db = &server.db[defrag_keys_ctx->dbid]; - if (db->keys != defrag_keys_ctx->kvstate.kvs) { + if (db->expires != defrag_keys_ctx->kvstate.kvs) { /* There has been a change of the kvs (flushdb, swapdb, etc.). Just complete the stage. */ return DEFRAG_DONE; } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/src/eval.c new/redis-8.0.3/src/eval.c --- old/redis-8.0.2/src/eval.c 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/src/eval.c 2025-07-06 13:59:42.000000000 +0200 @@ -920,7 +920,7 @@ if (ldb.forked) { writeToClient(c,0); serverLog(LL_NOTICE,"Lua debugging session child exiting"); - exitFromChild(0); + exitFromChild(0, 0); } else { serverLog(LL_NOTICE, "Redis synchronous debugging eval session ended"); diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/src/hyperloglog.c new/redis-8.0.3/src/hyperloglog.c --- old/redis-8.0.2/src/hyperloglog.c 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/src/hyperloglog.c 2025-07-06 13:59:42.000000000 +0200 @@ -579,6 +579,7 @@ struct hllhdr *hdr, *oldhdr = (struct hllhdr*)sparse; int idx = 0, runlen, regval; uint8_t *p = (uint8_t*)sparse, *end = p+sdslen(sparse); + int valid = 1; /* If the representation is already the right one return ASAP. */ hdr = (struct hllhdr*) sparse; @@ -598,16 +599,27 @@ while(p < end) { if (HLL_SPARSE_IS_ZERO(p)) { runlen = HLL_SPARSE_ZERO_LEN(p); + if ((runlen + idx) > HLL_REGISTERS) { /* Overflow. */ + valid = 0; + break; + } idx += runlen; p++; } else if (HLL_SPARSE_IS_XZERO(p)) { runlen = HLL_SPARSE_XZERO_LEN(p); + if ((runlen + idx) > HLL_REGISTERS) { /* Overflow. */ + valid = 0; + break; + } idx += runlen; p += 2; } else { runlen = HLL_SPARSE_VAL_LEN(p); regval = HLL_SPARSE_VAL_VALUE(p); - if ((runlen + idx) > HLL_REGISTERS) break; /* Overflow. */ + if ((runlen + idx) > HLL_REGISTERS) { /* Overflow. */ + valid = 0; + break; + } while(runlen--) { HLL_DENSE_SET_REGISTER(hdr->registers,idx,regval); idx++; @@ -618,7 +630,7 @@ /* If the sparse representation was valid, we expect to find idx * set to HLL_REGISTERS. */ - if (idx != HLL_REGISTERS) { + if (!valid || idx != HLL_REGISTERS) { sdsfree(dense); return C_ERR; } @@ -915,27 +927,40 @@ void hllSparseRegHisto(uint8_t *sparse, int sparselen, int *invalid, int* reghisto) { int idx = 0, runlen, regval; uint8_t *end = sparse+sparselen, *p = sparse; + int valid = 1; while(p < end) { if (HLL_SPARSE_IS_ZERO(p)) { runlen = HLL_SPARSE_ZERO_LEN(p); + if ((runlen + idx) > HLL_REGISTERS) { /* Overflow. */ + valid = 0; + break; + } idx += runlen; reghisto[0] += runlen; p++; } else if (HLL_SPARSE_IS_XZERO(p)) { runlen = HLL_SPARSE_XZERO_LEN(p); + if ((runlen + idx) > HLL_REGISTERS) { /* Overflow. */ + valid = 0; + break; + } idx += runlen; reghisto[0] += runlen; p += 2; } else { runlen = HLL_SPARSE_VAL_LEN(p); regval = HLL_SPARSE_VAL_VALUE(p); + if ((runlen + idx) > HLL_REGISTERS) { /* Overflow. */ + valid = 0; + break; + } idx += runlen; reghisto[regval] += runlen; p++; } } - if (idx != HLL_REGISTERS && invalid) *invalid = 1; + if ((!valid || idx != HLL_REGISTERS) && invalid) *invalid = 1; } /* ========================= HyperLogLog Count ============================== @@ -1204,22 +1229,34 @@ } else { uint8_t *p = hll->ptr, *end = p + sdslen(hll->ptr); long runlen, regval; + int valid = 1; p += HLL_HDR_SIZE; i = 0; while(p < end) { if (HLL_SPARSE_IS_ZERO(p)) { runlen = HLL_SPARSE_ZERO_LEN(p); + if ((runlen + i) > HLL_REGISTERS) { /* Overflow. */ + valid = 0; + break; + } i += runlen; p++; } else if (HLL_SPARSE_IS_XZERO(p)) { runlen = HLL_SPARSE_XZERO_LEN(p); + if ((runlen + i) > HLL_REGISTERS) { /* Overflow. */ + valid = 0; + break; + } i += runlen; p += 2; } else { runlen = HLL_SPARSE_VAL_LEN(p); regval = HLL_SPARSE_VAL_VALUE(p); - if ((runlen + i) > HLL_REGISTERS) break; /* Overflow. */ + if ((runlen + i) > HLL_REGISTERS) { /* Overflow. */ + valid = 0; + break; + } while(runlen--) { if (regval > max[i]) max[i] = regval; i++; @@ -1227,7 +1264,7 @@ p++; } } - if (i != HLL_REGISTERS) return C_ERR; + if (!valid || i != HLL_REGISTERS) return C_ERR; } return C_OK; } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/src/module.c new/redis-8.0.3/src/module.c --- old/redis-8.0.2/src/module.c 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/src/module.c 2025-07-06 13:59:42.000000000 +0200 @@ -11445,7 +11445,7 @@ */ int RM_ExitFromChild(int retcode) { sendChildCowInfo(CHILD_INFO_TYPE_MODULE_COW_SIZE, "Module fork"); - exitFromChild(retcode); + exitFromChild(retcode, 0); return REDISMODULE_OK; } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/src/rdb.c new/redis-8.0.3/src/rdb.c --- old/redis-8.0.2/src/rdb.c 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/src/rdb.c 2025-07-06 13:59:42.000000000 +0200 @@ -1658,7 +1658,7 @@ if (retval == C_OK) { sendChildCowInfo(CHILD_INFO_TYPE_RDB_COW_SIZE, "RDB"); } - exitFromChild((retval == C_OK) ? 0 : 1); + exitFromChild((retval == C_OK) ? 0 : 1, 0); } else { /* Parent */ if (childpid == -1) { @@ -3978,7 +3978,7 @@ UNUSED(dummy); } zfree(conns); - exitFromChild((retval == C_OK) ? 0 : 1); + exitFromChild((retval == C_OK) ? 0 : 1, 0); } else { /* Parent */ if (childpid == -1) { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/src/rio.h new/redis-8.0.3/src/rio.h --- old/redis-8.0.2/src/rio.h 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/src/rio.h 2025-07-06 13:59:42.000000000 +0200 @@ -23,7 +23,6 @@ #define RIO_FLAG_READ_ERROR (1<<0) #define RIO_FLAG_WRITE_ERROR (1<<1) -#define RIO_FLAG_ABORT (1<<2) #define RIO_TYPE_FILE (1<<0) #define RIO_TYPE_BUFFER (1<<1) @@ -103,7 +102,7 @@ * if needed. */ static inline size_t rioWrite(rio *r, const void *buf, size_t len) { - if (r->flags & (RIO_FLAG_WRITE_ERROR | RIO_FLAG_ABORT)) return 0; + if (r->flags & (RIO_FLAG_WRITE_ERROR)) return 0; while (len) { size_t bytes_to_write = (r->max_processing_chunk && r->max_processing_chunk < len) ? r->max_processing_chunk : len; if (r->update_cksum) r->update_cksum(r,buf,bytes_to_write); @@ -119,7 +118,7 @@ } static inline size_t rioRead(rio *r, void *buf, size_t len) { - if (r->flags & (RIO_FLAG_READ_ERROR | RIO_FLAG_ABORT)) return 0; + if (r->flags & (RIO_FLAG_READ_ERROR)) return 0; while (len) { size_t bytes_to_read = (r->max_processing_chunk && r->max_processing_chunk < len) ? r->max_processing_chunk : len; if (r->read(r,buf,bytes_to_read) == 0) { @@ -142,8 +141,10 @@ return r->flush(r); } +/* Abort RIO asynchronously by setting read and write error flags. Subsequent + * rioRead()/rioWrite() calls will fail, letting the caller terminate safely. */ static inline void rioAbort(rio *r) { - r->flags |= RIO_FLAG_ABORT; + r->flags |= (RIO_FLAG_READ_ERROR | RIO_FLAG_WRITE_ERROR); } /* This function allows to know if there was a read error in any past @@ -159,7 +160,7 @@ } static inline void rioClearErrors(rio *r) { - r->flags &= ~(RIO_FLAG_READ_ERROR|RIO_FLAG_WRITE_ERROR|RIO_FLAG_ABORT); + r->flags &= ~(RIO_FLAG_READ_ERROR|RIO_FLAG_WRITE_ERROR); } void rioInitWithFile(rio *r, FILE *fp); diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/src/server.c new/redis-8.0.3/src/server.c --- old/redis-8.0.2/src/server.c 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/src/server.c 2025-07-06 13:59:42.000000000 +0200 @@ -257,11 +257,19 @@ /* After an RDB dump or AOF rewrite we exit from children using _exit() instead of * exit(), because the latter may interact with the same file objects used by * the parent process. However if we are testing the coverage normal exit() is - * used in order to obtain the right coverage information. */ -void exitFromChild(int retcode) { + * used in order to obtain the right coverage information. + * There is a caveat for when we exit due to a signal. + * In this case we want the function to be async signal safe, so we can't use exit() + */ +void exitFromChild(int retcode, int from_signal) { #ifdef COVERAGE_TEST - exit(retcode); + if (!from_signal) { + exit(retcode); + } else { + _exit(retcode); + } #else + UNUSED(from_signal); _exit(retcode); #endif } @@ -6727,7 +6735,10 @@ UNUSED(sig); int level = server.in_fork_child == CHILD_TYPE_MODULE? LL_VERBOSE: LL_WARNING; serverLogRawFromHandler(level, "Received SIGUSR1 in child, exiting now."); - exitFromChild(SERVER_CHILD_NOERROR_RETVAL); + /* We don't want to perform any IO in the child when the parent is terminating us. + * We don't know what our stack trace is, it is possible that we were called during an IO operation + * If we were to do another IO operation, we might end up in a deadlock */ + exitFromChild(SERVER_CHILD_NOERROR_RETVAL, 1); } void setupChildSignalHandlers(void) { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/src/server.h new/redis-8.0.3/src/server.h --- old/redis-8.0.2/src/server.h 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/src/server.h 2025-07-06 13:59:42.000000000 +0200 @@ -2731,7 +2731,7 @@ void getRandomHexChars(char *p, size_t len); void getRandomBytes(unsigned char *p, size_t len); uint64_t crc64(uint64_t crc, const unsigned char *s, uint64_t l); -void exitFromChild(int retcode); +void exitFromChild(int retcode, int from_signal); long long redisPopcount(void *s, long count); int redisSetProcTitle(char *title); int validateProcTitleTemplate(const char *template); diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/src/socket.c new/redis-8.0.3/src/socket.c --- old/redis-8.0.2/src/socket.c 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/src/socket.c 2025-07-06 13:59:42.000000000 +0200 @@ -308,6 +308,8 @@ while(max--) { cfd = anetTcpAccept(server.neterr, fd, cip, sizeof(cip), &cport); if (cfd == ANET_ERR) { + if (anetAcceptFailureNeedsRetry(errno)) + continue; if (errno != EWOULDBLOCK) serverLog(LL_WARNING, "Accepting client connection: %s", server.neterr); diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/src/tls.c new/redis-8.0.3/src/tls.c --- old/redis-8.0.2/src/tls.c 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/src/tls.c 2025-07-06 13:59:42.000000000 +0200 @@ -771,6 +771,8 @@ while(max--) { cfd = anetTcpAccept(server.neterr, fd, cip, sizeof(cip), &cport); if (cfd == ANET_ERR) { + if (anetAcceptFailureNeedsRetry(errno)) + continue; if (errno != EWOULDBLOCK) serverLog(LL_WARNING, "Accepting client connection: %s", server.neterr); diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/src/unix.c new/redis-8.0.3/src/unix.c --- old/redis-8.0.2/src/unix.c 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/src/unix.c 2025-07-06 13:59:42.000000000 +0200 @@ -102,6 +102,8 @@ while(max--) { cfd = anetUnixAccept(server.neterr, fd); if (cfd == ANET_ERR) { + if (anetAcceptFailureNeedsRetry(errno)) + continue; if (errno != EWOULDBLOCK) serverLog(LL_WARNING, "Accepting client connection: %s", server.neterr); diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/src/version.h new/redis-8.0.3/src/version.h --- old/redis-8.0.2/src/version.h 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/src/version.h 2025-07-06 13:59:42.000000000 +0200 @@ -1,2 +1,2 @@ -#define REDIS_VERSION "8.0.2" -#define REDIS_VERSION_NUM 0x00080002 +#define REDIS_VERSION "8.0.3" +#define REDIS_VERSION_NUM 0x00080003 diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/tests/integration/replication-rdbchannel.tcl new/redis-8.0.3/tests/integration/replication-rdbchannel.tcl --- old/redis-8.0.2/tests/integration/replication-rdbchannel.tcl 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/tests/integration/replication-rdbchannel.tcl 2025-07-06 13:59:42.000000000 +0200 @@ -248,9 +248,8 @@ populate 10000 master 10000 ;# 10k keys of 10k, means 100mb $replica config set loading-process-events-interval-bytes 262144 ;# process events every 256kb of rdb or command stream - # Start write traffic writing at most 5mbps - - set load_handle [start_write_load $master_host $master_port 100 "key1" 10000 2] + # Start write traffic + set load_handle [start_write_load $master_host $master_port 100 "key1" 5000 4] set prev_used [s 0 used_memory] @@ -300,8 +299,8 @@ assert_lessthan [expr $peak_master_used_mem - $prev_used - $backlog_size] 1000000 assert_lessthan $peak_master_rpl_buf [expr {$backlog_size + 1000000}] assert_lessthan $peak_master_slave_buf_size 1000000 - # buffers in the replica are more than 10mb - assert_morethan $peak_replica_buf_size 10000000 + # buffers in the replica are more than 5mb + assert_morethan $peak_replica_buf_size 5000000 stop_write_load $load_handle } @@ -434,11 +433,10 @@ fail "Replica did not start loading" } - # Generate some traffic for backlog ~2mb + # Generate replication traffic of ~20mb to disconnect the slave on obuf limit populate 20 master 1000000 -1 - set res [wait_for_log_messages -1 {"*Client * closed * for overcoming of output buffer limits.*"} $loglines 1000 10] - set loglines [lindex $res 1] + wait_for_log_messages -1 {"*Client * closed * for overcoming of output buffer limits.*"} $loglines 1000 10 $replica config set key-load-delay 0 # Wait until replica loads RDB diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/tests/unit/hyperloglog.tcl new/redis-8.0.3/tests/unit/hyperloglog.tcl --- old/redis-8.0.2/tests/unit/hyperloglog.tcl 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/tests/unit/hyperloglog.tcl 2025-07-06 13:59:42.000000000 +0200 @@ -137,6 +137,61 @@ set e } {*WRONGTYPE*} + test {Corrupted sparse HyperLogLogs doesn't cause overflow and out-of-bounds with XZERO opcode} { + r del hll + + # Create a sparse-encoded HyperLogLog header + set header "HYLL" + set payload [binary format c12 {1 0 0 0 0 0 0 0 0 0 0 0}] + set pl [binary format a4a12 $header $payload] + + # Create an XZERO opcode with the maximum run length of 16384(2^14) + set runlen [expr 16384 - 1] + set chunk [binary format cc [expr {0b01000000 | ($runlen >> 8)}] [expr {$runlen & 0xff}]] + # Fill the HLL with more than 131072(2^17) XZERO opcodes to make the total + # run length exceed 4GB, will cause an integer overflow. + set repeat [expr 131072 + 1000] + for {set i 0} {$i < $repeat} {incr i} { + append pl $chunk + } + + # Create a VAL opcode with a value that will cause out-of-bounds. + append pl [binary format c 0b11111111] + r set hll $pl + + # This should not overflow and out-of-bounds. + assert_error {*INVALIDOBJ*} {r pfcount hll hll} + assert_error {*INVALIDOBJ*} {r pfdebug getreg hll} + r ping + } + + test {Corrupted sparse HyperLogLogs doesn't cause overflow and out-of-bounds with ZERO opcode} { + r del hll + + # Create a sparse-encoded HyperLogLog header + set header "HYLL" + set payload [binary format c12 {1 0 0 0 0 0 0 0 0 0 0 0}] + set pl [binary format a4a12 $header $payload] + + # # Create an ZERO opcode with the maximum run length of 64(2^6) + set chunk [binary format c [expr {0b00000000 | 0x3f}]] + # Fill the HLL with more than 33554432(2^17) ZERO opcodes to make the total + # run length exceed 4GB, will cause an integer overflow. + set repeat [expr 33554432 + 1000] + for {set i 0} {$i < $repeat} {incr i} { + append pl $chunk + } + + # Create a VAL opcode with a value that will cause out-of-bounds. + append pl [binary format c 0b11111111] + r set hll $pl + + # This should not overflow and out-of-bounds. + assert_error {*INVALIDOBJ*} {r pfcount hll hll} + assert_error {*INVALIDOBJ*} {r pfdebug getreg hll} + r ping + } + test {Corrupted dense HyperLogLogs are detected: Wrong length} { r del hll r pfadd hll a b c diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/tests/unit/memefficiency.tcl new/redis-8.0.3/tests/unit/memefficiency.tcl --- old/redis-8.0.2/tests/unit/memefficiency.tcl 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/tests/unit/memefficiency.tcl 2025-07-06 13:59:42.000000000 +0200 @@ -520,10 +520,10 @@ r config resetstat # TODO: Lower the threshold after defraging the ebuckets. # Now just to ensure that the reference is updated correctly. - r config set active-defrag-threshold-lower 12 + r config set active-defrag-threshold-lower 10 r config set active-defrag-cycle-min 65 r config set active-defrag-cycle-max 75 - r config set active-defrag-ignore-bytes 1500kb + r config set active-defrag-ignore-bytes 1000kb r config set maxmemory 0 r config set hash-max-listpack-value 512 r config set hash-max-listpack-entries 10 @@ -558,7 +558,7 @@ puts "frag [s allocator_frag_ratio]" puts "frag_bytes [s allocator_frag_bytes]" } - assert_lessthan [s allocator_frag_ratio] 1.05 + assert_lessthan [s allocator_frag_ratio] 1.1 # Delete all the keys to create fragmentation for {set i 0} {$i < $n} {incr i} { @@ -591,7 +591,7 @@ } # wait for the active defrag to stop working - wait_for_defrag_stop 500 100 1.5 + wait_for_defrag_stop 500 100 1.1 # test the fragmentation is lower after 120 ;# serverCron only updates the info once in 100ms @@ -732,15 +732,15 @@ r config set active-defrag-cycle-max 75 r config set active-defrag-ignore-bytes 2mb r config set maxmemory 0 - r config set list-max-ziplist-size 5 ;# list of 500k items will have 100k quicklist nodes + r config set list-max-ziplist-size 1 ;# list of 100k items will have 100k quicklist nodes # create big keys with 10k items set rd [redis_deferring_client] set expected_frag 1.5 # add a mass of list nodes to two lists (allocations are interlaced) - set val [string repeat A 100] ;# 5 items of 100 bytes puts us in the 640 bytes bin, which has 32 regs, so high potential for fragmentation - set elements 500000 + set val [string repeat A 500] ;# 1 item of 500 bytes puts us in the 640 bytes bin, which has 32 regs, so high potential for fragmentation + set elements 100000 for {set j 0} {$j < $elements} {incr j} { $rd lpush biglist1 $val $rd lpush biglist2 $val @@ -811,9 +811,9 @@ assert {$max_latency <= 30} } - # in extreme cases of stagnation, we see over 20m misses before the tests aborts with "defrag didn't stop", - # in normal cases we only see 100k misses out of 500k elements - assert {$misses < $elements} + # in extreme cases of stagnation, we see over 5m misses before the tests aborts with "defrag didn't stop", + # in normal cases we only see 100k misses out of 100k elements + assert {$misses < $elements * 2} } # verify the data isn't corrupted or changed set newdigest [debug_digest] diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/redis-8.0.2/tests/unit/moduleapi/testrdb.tcl new/redis-8.0.3/tests/unit/moduleapi/testrdb.tcl --- old/redis-8.0.2/tests/unit/moduleapi/testrdb.tcl 2025-05-27 14:39:18.000000000 +0200 +++ new/redis-8.0.3/tests/unit/moduleapi/testrdb.tcl 2025-07-06 13:59:42.000000000 +0200 @@ -116,6 +116,11 @@ $master config set dynamic-hz no $replica config set dynamic-hz no set start [clock clicks -milliseconds] + # Generate small keys + for {set k 0} {$k < 20000} {incr k} { + r testrdb.set.key keysmall$k [string repeat A [expr {int(rand()*100)}]] + } + # Generate larger keys for {set k 0} {$k < 30} {incr k} { r testrdb.set.key key$k [string repeat A [expr {int(rand()*1000000)}]] } ++++++ redis.hashes ++++++ --- /var/tmp/diff_new_pack.9bl9ye/_old 2025-07-08 11:31:55.011961141 +0200 +++ /var/tmp/diff_new_pack.9bl9ye/_new 2025-07-08 11:31:55.015961305 +0200 @@ -191,4 +191,10 @@ hash redis-8.0.2.tar.gz sha256 e9296b67b54c91befbcca046d67071c780a1f7c9f9e1ae5ed94773c3bb9b542f http://download.redis.io/releases/redis-8.0.2.tar.gz hash redis-7.4.4.tar.gz sha256 985c465146453f4d79912e70b2dc516577a1667cbf9b0420a0c87878fcc6f32f http://download.redis.io/releases/redis-7.4.4.tar.gz hash redis-7.2.9.tar.gz sha256 2343cc49db3beb9d2925a44e13032805a608821a58f25bd874c84881115a20b7 http://download.redis.io/releases/redis-7.2.9.tar.gz +hash redis-8.0.2.tar.gz sha256 e9296b67b54c91befbcca046d67071c780a1f7c9f9e1ae5ed94773c3bb9b542f http://download.redis.io/releases/redis-8.0.2.tar.gz +hash redis-8.2-rc1.tar.gz sha256 bf27a934e741dfaa74e629e545a0a237dd8037c0444dec5f3e0bdd23a8a7e9c1 http://download.redis.io/releases/redis-8.2-rc1.tar.gz +hash redis-6.2.19.tar.gz sha256 73be4202261c2e2e3534ec2c3dcfbb338cceff40481ecf46c3578cb9e5fdea74 http://download.redis.io/releases/redis-6.2.19.tar.gz +hash redis-7.2.10.tar.gz sha256 e576ad54bc53770649c556933ecd555b975e3dac422e46356102436a437b43c7 http://download.redis.io/releases/redis-7.2.10.tar.gz +hash redis-7.4.5.tar.gz sha256 00bb280528f5d7934bec8ab309b8125088c209131e10609cb1563b91365633bb http://download.redis.io/releases/redis-7.4.5.tar.gz +hash redis-8.0.3.tar.gz sha256 33f37290b00b14e9a884dd4dcba335febd63ea16c51609d34fa41e031ad587df http://download.redis.io/releases/redis-8.0.3.tar.gz
