Awesome! That's great to hear! Pls feel free to let me know if you have any questions!
Thanks, Cheng On Tue, Oct 25, 2022 at 10:36 AM Aaron Ploetz <aaronplo...@gmail.com> wrote: > Works! > > So I was running on my *local*, and all of my attempts to add to the > denylist were failing because the denylist_consistency_level was set to > QUORUM: > > WARN [main] 2022-10-25 11:57:27,238 NoSpamLogger.java:108 - Attempting to > load denylist and not enough nodes are available for a QUORUM refresh. > Reload the denylist when unavailable nodes are recovered to ensure your > denylist remains in sync. > INFO [main] 2022-10-25 11:57:27,238 PartitionDenylist.java:163 - > Insufficient nodes while loading partition denylist cache. Scheduled retry > in 5 seconds. > > Adjusted that setting to ONE, and now everything is working. > > Thanks again, Cheng! > > Aaron > > > On Fri, Oct 21, 2022 at 9:48 AM Aaron Ploetz <aaronplo...@gmail.com> > wrote: > >> Awesome. Thank you, Cheng! I’ll give this a shot and let you know. >> >> Thanks, >> >> Aaron >> >> >> On Oct 21, 2022, at 12:45 AM, Cheng Wang <che...@netflix.com> wrote: >> >> >> Hi Aaron, >> >> After reading through the code, I finally figured out the issue. So back >> to your original question where you failed to run >> $>run denylistKey stackoverflow weather_sensor_data "'Minneapolis, >> MN',202210" >> #IllegalArgumentException: Operation denylistKey with 4 parameters >> doesn't exist in bean org.apache.cassandra.db:type=StorageProxy >> >> It's not a Cassandra issue since it failed at the JMX parser stage, even >> before it goes to the Cassandra internal StorageProxy::denylistKey method. >> Yes, you got the right gist. It's because of the extra space between >> "Minneapolis" and "MN". To avoid the error, you need to add "\\" before the >> white space as below : >> >> $>run denylistKey "stackoverflow" "weather_sensor_data" "Minneapolis,\\ >> MN:202210" >> >> #calling operation denylistKey of mbean >> org.apache.cassandra.db:type=StorageProxy with params [stackoverflow, >> weather_sensor_data, Minneapolis, MN:202210] >> >> #operation returns: >> >> true >> >> And yes, the composite key is delimited by ":" . You can find more >> examples from the test code as it used composite keys >> >> https://github.com/apache/cassandra/blob/trunk/test/unit/org/apache/cassandra/service/PartitionDenylistTest.java >> >> And then when you select the key, it will get denylisted: >> >> cqlsh> select ks_name, table_name, key from >> system_distributed.partition_denylist; >> >> >> ks_name | table_name | key >> >> >> ---------------+---------------------+------------------------------------------------------ >> >> stackoverflow | weather_sensor_data | >> 0x000f4d696e6e6561706f6c69732c204d4e000004000315e200 >> >> >> (2 rows) >> >> cqlsh> select * from stackoverflow.weather_sensor_data where city = >> 'Minneapolis, MN' AND month = 202210; >> >> InvalidRequest: Error from server: code=2200 [Invalid query] >> message="Unable to read denylisted partition >> [0xDecoratedKey(8132598509541575594, >> 000f4d696e6e6561706f6c69732c204d4e000004000315e200)] in >> stackoverflow/weather_sensor_data" >> >> cqlsh> >> >> >> For your second question why even a single partition key doesn't work, my >> apologies my previous answer on direct insert to the s >> ystem_distributed.partition_denylist like >> insert into system_distributed.partition_denylist (ks_name, table_name, >> key) values ('stackoverflow', 'weather_sensor_data', >> textAsBlob('''Minneapolis, MN'', 202210')); >> is wrong. It's because the way Cassandra stores the key in the system >> table is it converts the string into a hexadecimal literal. You can find >> the source code here >> >> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/schema/PartitionDenylist.java >> And check out the function addKeyToDenylist >> final String insert = String.format("INSERT INTO system_distributed. >> partition_denylist (ks_name, table_name, key) VALUES ('%s', '%s', 0x%s)", >> keyspace, table, ByteBufferUtil.bytesToHex(key)); >> >> You can see that it's different from the way of textAsBlob. A better (yet >> general) explanation is here >> >> https://stackoverflow.com/questions/31994880/why-cassandra-cql-query-with-hexadecimal-literal-works-but-textasbinary-does-not >> >> So... the short answer is, I get back my previous answer. My apologies! >> It's not recommended to directly insert rows into >> the system_distributed.partition_denylist table.You'd better go with the >> JMX interface since it will go to the StorageProxy which handles more logic >> like the hex string conversion and other error handling stuff. >> >> Hope the answer is helpful! Pls feel free to let me know if you have any >> questions! >> >> Thanks, >> Cheng >> >> >> On Thu, Oct 20, 2022 at 7:07 AM Aaron Ploetz <aaronplo...@gmail.com> >> wrote: >> >>> No worries, Cheng! >>> >>> So I actually pivoted a little and adjusted my example table to use a >>> single integer-based partition key. >>> >>> aaron@cqlsh:stackoverflow> SELECT ks_name, table_name, blobAsint(key) >>> FROM system_distributed.partition_denylist WHERE ks_name='stackoverflow' >>> AND table_name='weather_sensor_data_by_month'; >>> >>> ks_name | table_name | system.blobasint(key) >>> ---------------+------------------------------+----------------------- >>> stackoverflow | weather_sensor_data_by_month | 202210 >>> >>> But even with the yaml changes, it still allows me to SELECT that >>> partition. >>> >>> aaron@cqlsh:stackoverflow> SELECT * FROM weather_sensor_data_by_month >>> WHERE month=202210 LIMIT 1; >>> >>> month | city | recorded_time | temp >>> --------+----------------+---------------------------------+------ >>> 202210 | Minneapolis,MN | 2022-10-17 11:30:00.000000+0000 | 1 >>> >>> Just wondering what I should do to get it to deny access? >>> >>> Thanks, >>> >>> Aaron >>> >>> >>> >>> On Wed, Oct 19, 2022 at 5:34 PM Cheng Wang <che...@netflix.com> wrote: >>> >>>> Hi Aaron, >>>> >>>> Sorry for the late reply, was dealing with a production issue (maybe >>>> another topic for Cassandra Summit :-)). Are you running on your local >>>> machine? Then yes, you do need to enable the config for all the following >>>> >>>> enable_partition_denylist: true >>>> >>>> enable_denylist_writes: true >>>> enable_denylist_reads: true >>>> enable_denylist_range_reads: true >>>> >>>> I am not sure enable_partition_denylist will enable for the rest of >>>> three @Jordan West <jord...@netflix.com> maybe you can confirm for >>>> that? But it's better to enable for all just to be safe. >>>> I will play by my side and get back to you soon about the >>>> composite keys. >>>> >>>> Thanks >>>> Cheng >>>> >>>> On Wed, Oct 19, 2022 at 1:42 PM Aaron Ploetz <aaronplo...@gmail.com> >>>> wrote: >>>> >>>>> Just checking, but for this to work, do I have to mess with these >>>>> settings in the YAML at all? >>>>> >>>>> partition_denylist_enabled: true >>>>> denylist_reads_enabled: true >>>>> >>>>> They're commented out by default. >>>>> >>>>> Thanks, >>>>> >>>>> Aaron >>>>> >>>>> >>>>> On Mon, Oct 17, 2022 at 4:53 PM Aaron Ploetz <aaronplo...@gmail.com> >>>>> wrote: >>>>> >>>>>> Thanks for the help with the INSERT, Cheng! I'm further along than >>>>>> before. But it still must not be matching up quite right, because I can >>>>>> still select that partition. >>>>>> >>>>>> I have several different combinations of the two keys (and I removed >>>>>> the space) of "Minneapolis,MN" and 202210. Here's what I've got out >>>>>> there >>>>>> so far: >>>>>> >>>>>> aaron@cqlsh:stackoverflow> select ks_name, table_name, >>>>>> blobAsText(key) from system_distributed.partition_denylist; >>>>>> >>>>>> ks_name | table_name | system.blobastext(key) >>>>>> ---------------+---------------------+-------------------------- >>>>>> stackoverflow | weather_sensor_data | 'Minneapolis,MN', 202210 >>>>>> stackoverflow | weather_sensor_data | 'Minneapolis,MN',202210 >>>>>> stackoverflow | weather_sensor_data | 'Minneapolis,MN':202210 >>>>>> stackoverflow | weather_sensor_data | Minneapolis,MN, 202210 >>>>>> stackoverflow | weather_sensor_data | Minneapolis,MN202210 >>>>>> stackoverflow | weather_sensor_data | Minneapolis,MN:202210 >>>>>> >>>>>> (6 rows) >>>>>> >>>>>> aaron@cqlsh:stackoverflow> SELECT * FROM weather_sensor_data WHERE >>>>>> city='Minneapolis,MN' AND month=202210; >>>>>> >>>>>> city | month | recorded_time | temp >>>>>> ----------------+--------+---------------------------------+------ >>>>>> Minneapolis,MN | 202210 | 2022-10-17 11:30:00.000000+0000 | 1 >>>>>> Minneapolis,MN | 202210 | 2022-10-17 11:25:00.000000+0000 | 1 >>>>>> Minneapolis,MN | 202210 | 2022-10-17 11:20:00.000000+0000 | 1 >>>>>> Minneapolis,MN | 202210 | 2022-10-17 11:15:00.000000+0000 | 1 >>>>>> Minneapolis,MN | 202210 | 2022-10-17 11:10:00.000000+0000 | 2 >>>>>> Minneapolis,MN | 202210 | 2022-10-17 11:05:00.000000+0000 | 2 >>>>>> Minneapolis,MN | 202210 | 2022-10-17 11:00:00.000000+0000 | 2 >>>>>> >>>>>> (7 rows) >>>>>> >>>>>> As you can see, I can still select the partition. I was really >>>>>> hoping one of those combinations would do it. >>>>>> >>>>>> Looking at the StorageProxyTest.java in the project, I saw that it >>>>>> was delimited by a colon ":", which is why I tried that, too. >>>>>> >>>>>> Still looking for the right way to enter both of those keys. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Aaron >>>>>> >>>>>> >>>>>> On Mon, Oct 17, 2022 at 4:40 PM Cheng Wang <che...@netflix.com> >>>>>> wrote: >>>>>> >>>>>>> Another approach is, instead of using $$, you can put additional >>>>>>> pair of single quote around the 'Minneapolis, MN' >>>>>>> >>>>>>> cqlsh> insert into system_distributed.partition_denylist (ks_name, >>>>>>> table_name, key) values ('stackoverflow', 'weather_sensor_data', >>>>>>> textAsBlob('''Minneapolis, MN'', 202210')); >>>>>>> >>>>>>> cqlsh> select ks_name, table_name, blobAsText(key) from >>>>>>> system_distributed.partition_denylist; >>>>>>> >>>>>>> >>>>>>> ks_name | table_name | system.blobastext(key) >>>>>>> >>>>>>> ---------------+---------------------+--------------------------- >>>>>>> >>>>>>> stackoverflow | weather_sensor_data | 'Minneapolis, MN', 202210 >>>>>>> >>>>>>> On Mon, Oct 17, 2022 at 2:30 PM Cheng Wang <che...@netflix.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Aaron, >>>>>>>> >>>>>>>> Yes, you can directly insert into the >>>>>>>> system_distributed.partition_denylist >>>>>>>> instead of using JMX. Jordan wrote a blog post for denylist >>>>>>>> >>>>>>>> https://cassandra.apache.org/_/blog/Apache-Cassandra-4.1-Denylisting-Partitions.html >>>>>>>> >>>>>>>> And the syntax error, one way around is to put $$ around like below >>>>>>>> >>>>>>>> cqlsh> insert into system_distributed.partition_denylist (ks_name, >>>>>>>> table_name, key) values ('stackoverflow', 'weather_sensor_data', >>>>>>>> textAsBlob($$'Minneapolis, MN', 202210$$)); >>>>>>>> >>>>>>>> There is post about this >>>>>>>> >>>>>>>> https://docs.datastax.com/en/cql-oss/3.3/cql/cql_reference/escape_char_r.html#:~:text=Column%20names%20that%20contain%20characters,using%20a%20single%20quotation%20mark >>>>>>>> . >>>>>>>> >>>>>>>> and then you can verify the insert by doing >>>>>>>> >>>>>>>> cqlsh> select ks_name, table_name, blobAsText(key) from >>>>>>>> system_distributed.partition_denylist; >>>>>>>> >>>>>>>> >>>>>>>> ks_name | table_name | system.blobastext(key) >>>>>>>> >>>>>>>> ---------------+---------------------+--------------------------- >>>>>>>> >>>>>>>> stackoverflow | weather_sensor_data | 'Minneapolis, MN', 202210 >>>>>>>> Pls let me know if it works for you. >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Oct 17, 2022 at 1:35 PM Aaron Ploetz <aaronplo...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I have this table definition: >>>>>>>>> >>>>>>>>> CREATE TABLE stackoverflow.weather_sensor_data ( >>>>>>>>> city text, >>>>>>>>> month int, >>>>>>>>> recorded_time timestamp, >>>>>>>>> temp float, >>>>>>>>> PRIMARY KEY ((city, month), recorded_time) >>>>>>>>> ) WITH CLUSTERING ORDER BY (recorded_time DESC) >>>>>>>>> >>>>>>>>> Sample data looks like this: >>>>>>>>> >>>>>>>>> > SELECT * FROM weather_sensor_data WHERE city='Minneapolis, MN' >>>>>>>>> AND month=202111; >>>>>>>>> >>>>>>>>> city | month | recorded_time | temp >>>>>>>>> -----------------+--------+---------------------------------+------ >>>>>>>>> Minneapolis, MN | 202111 | 2021-11-01 08:35:00.000000+0000 | 3 >>>>>>>>> Minneapolis, MN | 202111 | 2021-11-01 08:30:00.000000+0000 | 3 >>>>>>>>> Minneapolis, MN | 202111 | 2021-11-01 08:25:00.000000+0000 | 2 >>>>>>>>> Minneapolis, MN | 202111 | 2021-11-01 08:20:00.000000+0000 | 2 >>>>>>>>> Minneapolis, MN | 202111 | 2021-11-01 08:15:00.000000+0000 | 2 >>>>>>>>> >>>>>>>>> (5 rows) >>>>>>>>> >>>>>>>>> Using JMX Term, I've tried to denylist that partition, but I must >>>>>>>>> have the syntax for composite keys incorrect: >>>>>>>>> >>>>>>>>> $>bean org.apache.cassandra.db:type=StorageProxy >>>>>>>>> $>run denylistKey stackoverflow weather_sensor_data "'Minneapolis, >>>>>>>>> MN',202210" >>>>>>>>> #IllegalArgumentException: Operation denylistKey with 4 parameters >>>>>>>>> doesn't exist in bean org.apache.cassandra.db:type=StorageProxy >>>>>>>>> >>>>>>>>> Obviously, it's reading the space between "Minneapolis," and "MN" >>>>>>>>> as a delimiter. What's the right way to handle commas, spaces, and >>>>>>>>> composite keys for this? >>>>>>>>> >>>>>>>>> Also, is there another way to accomplish this without using JMX? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Aaron >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>
