Fixes for UDF NPE during restart.
CASSANDRA-18739 describes a reproducible NPE on restart with some UDFs. The solution outlined in that ticket was not used and a much simpler solution provided by Stefan Miklosovic was implemented. There are 2 pull requests open for Cassandra 4.0 and 4.1 that have the fairly simple fix as well as a test case to show that there was a problem and that the fix works. Can someone take a look at the issue and pull requests. Stefan is working on a solution for v5 that we may want to get in ASAP. v4.0 - https://github.com/apache/cassandra/pull/2584 v4.1 - https://github.com/apache/cassandra/pull/2585 Thank you, Claude
Re: UDF: adding custom jar to classpath
OK, that seems clear now :) I understood from our conversations that "enable_user_defined_functions_threads: false" would disable the UDF' specific class loader but it seems I understood wrongly, so the only way to use custom packages in UDF is to modify source code. Many thanks! Le mer. 6 avr. 2022 à 16:35, bened...@apache.org a écrit : > The property you are setting permits some kinds of privilege escalation, > but by default classes outside of those pre-defined by the whitelist are > not permitted. This is imposed here: > https://github.com/apache/cassandra/blob/210793f943dc522161fd26b6192f38a5c83fa131/src/java/org/apache/cassandra/cql3/functions/UDFunction.java#L168 > > > > You will need to modify the source code to e.g. add additional > allowedPatterns, or perhaps to permit additional patterns to be configured > at startup. > > > > *From: *Sébastien Rebecchi > *Date: *Wednesday, 6 April 2022 at 15:15 > *To: *dev@cassandra.apache.org , > e.dimitr...@gmail.com > *Cc: *ble...@apache.org > *Subject: *Re: UDF: adding custom jar to classpath > > Hi Ekaterina, > > > > I use 4.0.1. > > But as I said I added a jar in classpath (/usr/share/cassandra/lib/ folder > on every node) and I see that the jar is loaded in the classpath from the > Cassandra command line. And I have "enable_user_defined_functions: true" > and "enable_user_defined_functions_threads: false" in cassandra.yaml. > > So I don't see what is missing or not done properly. > > > > Best regards, > > Sébastien. > > > > Le mer. 6 avr. 2022 à 16:03, Ekaterina Dimitrova > a écrit : > > Hi Sebastian, > > Do you use the latest 4.0.3 version? Those options were added in 4.0.2 I > believe, so if you try them with an earlier version - below message is what > you would get as they didn’t exist. > > > > Best regards, > > Ekaterina > > > > On Wed, 6 Apr 2022 at 9:53, Sébastien Rebecchi > wrote: > > Hi Benjamin, Hi everybody, > > > > I found in the documentation that we should add "allow_insecure_udfs: > true" and optionally "allow_extra_insecure_udfs: true" so that > "enable_user_defined_functions_threads: false" is really taken into account > (I understood like that). That would explain why my UDF still does not run > even with "enable_user_defined_functions_threads: false". Found in > https://github.com/apache/cassandra/blob/cassandra-4.0/NEWS.txt > > > > So I tried to add "allow_insecure_udfs: true" and > "allow_extra_insecure_udfs: true" in cassandra.yaml, but then Cassandra > failed to restart and I got that error in logs "Exception > (org.apache.cassandra.exceptions.ConfigurationException) encountered during > startup: Invalid yaml. Please remove properties [allow_insecure_udfs, > allow_extra_insecure_udfs] from your cassandra.yaml". > > > > Should I understand that we can activate that 2 extra confs only by > changing source code? That would be really disappointing :( And if no, then > how to activate all UDF possibilities from cassandra.yaml please? > > > > Thanks in advance, > > > > Sébastien. > > > > > > Le mar. 5 avr. 2022 à 10:36, Benjamin Lerer a écrit : > > Unfortunately, I do not have much time for doing some digging. Sorry for > that :-( > > You should look at JavaBasedUDFunction and UDFExecutorServic. > > > > Le lun. 4 avr. 2022 à 17:25, Sébastien Rebecchi > a écrit : > > Hi! > > Do you have any more ideas for me? > > Cordially, > > Sébastien. > > > > Le lun. 28 mars 2022 à 16:39, Sébastien Rebecchi > a écrit : > > Unfortunately, it is not working even with > "enable_user_defined_functions_threads: false" in cassandra.yaml :/ > > Is there any way to check the running configuration? > > > > Le lun. 28 mars 2022 à 15:35, Benjamin Lerer a écrit : > > I do not think that allowing to customize UDF classes whitelist has been > discussed before. Feel free to open a JIRA ticket :-) > > I have some plans to revisit how we securise UDFs as the current threading > approach has some impact in terms of latency. That can be a good > opportunity to look into providing more flexibility. > > > > Le lun. 28 mars 2022 à 15:00, Sébastien Rebecchi > a écrit : > > Thanks you very much! I will try that. > > As you know, would it be a long-terms solution? Or is there any plan to > add the possibility to customize UDF classes whitelist? > > > > Le lun. 28 mars 2022 à 14:31, Benjamin Lerer a écrit : > > Is there a way to customize that default behaviour? > > > > Looking a
Re: UDF: adding custom jar to classpath
The property you are setting permits some kinds of privilege escalation, but by default classes outside of those pre-defined by the whitelist are not permitted. This is imposed here: https://github.com/apache/cassandra/blob/210793f943dc522161fd26b6192f38a5c83fa131/src/java/org/apache/cassandra/cql3/functions/UDFunction.java#L168 You will need to modify the source code to e.g. add additional allowedPatterns, or perhaps to permit additional patterns to be configured at startup. From: Sébastien Rebecchi Date: Wednesday, 6 April 2022 at 15:15 To: dev@cassandra.apache.org , e.dimitr...@gmail.com Cc: ble...@apache.org Subject: Re: UDF: adding custom jar to classpath Hi Ekaterina, I use 4.0.1. But as I said I added a jar in classpath (/usr/share/cassandra/lib/ folder on every node) and I see that the jar is loaded in the classpath from the Cassandra command line. And I have "enable_user_defined_functions: true" and "enable_user_defined_functions_threads: false" in cassandra.yaml. So I don't see what is missing or not done properly. Best regards, Sébastien. Le mer. 6 avr. 2022 à 16:03, Ekaterina Dimitrova mailto:e.dimitr...@gmail.com>> a écrit : Hi Sebastian, Do you use the latest 4.0.3 version? Those options were added in 4.0.2 I believe, so if you try them with an earlier version - below message is what you would get as they didn’t exist. Best regards, Ekaterina On Wed, 6 Apr 2022 at 9:53, Sébastien Rebecchi mailto:srebec...@kameleoon.com>> wrote: Hi Benjamin, Hi everybody, I found in the documentation that we should add "allow_insecure_udfs: true" and optionally "allow_extra_insecure_udfs: true" so that "enable_user_defined_functions_threads: false" is really taken into account (I understood like that). That would explain why my UDF still does not run even with "enable_user_defined_functions_threads: false". Found in https://github.com/apache/cassandra/blob/cassandra-4.0/NEWS.txt So I tried to add "allow_insecure_udfs: true" and "allow_extra_insecure_udfs: true" in cassandra.yaml, but then Cassandra failed to restart and I got that error in logs "Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: Invalid yaml. Please remove properties [allow_insecure_udfs, allow_extra_insecure_udfs] from your cassandra.yaml". Should I understand that we can activate that 2 extra confs only by changing source code? That would be really disappointing :( And if no, then how to activate all UDF possibilities from cassandra.yaml please? Thanks in advance, Sébastien. Le mar. 5 avr. 2022 à 10:36, Benjamin Lerer mailto:ble...@apache.org>> a écrit : Unfortunately, I do not have much time for doing some digging. Sorry for that :-( You should look at JavaBasedUDFunction and UDFExecutorServic. Le lun. 4 avr. 2022 à 17:25, Sébastien Rebecchi mailto:srebec...@kameleoon.com>> a écrit : Hi! Do you have any more ideas for me? Cordially, Sébastien. Le lun. 28 mars 2022 à 16:39, Sébastien Rebecchi mailto:srebec...@kameleoon.com>> a écrit : Unfortunately, it is not working even with "enable_user_defined_functions_threads: false" in cassandra.yaml :/ Is there any way to check the running configuration? Le lun. 28 mars 2022 à 15:35, Benjamin Lerer mailto:ble...@apache.org>> a écrit : I do not think that allowing to customize UDF classes whitelist has been discussed before. Feel free to open a JIRA ticket :-) I have some plans to revisit how we securise UDFs as the current threading approach has some impact in terms of latency. That can be a good opportunity to look into providing more flexibility. Le lun. 28 mars 2022 à 15:00, Sébastien Rebecchi mailto:srebec...@kameleoon.com>> a écrit : Thanks you very much! I will try that. As you know, would it be a long-terms solution? Or is there any plan to add the possibility to customize UDF classes whitelist? Le lun. 28 mars 2022 à 14:31, Benjamin Lerer mailto:ble...@apache.org>> a écrit : Is there a way to customize that default behaviour? Looking at JavaBasedUDFunction quickly it seems that the ClassLoader is only used when you use the UDFExecutorService to execute your UDFs. You can try to disable it using "enable_user_defined_functions_threads: false" and see if it works. Now that also means that you have to ensure that only trusted persons can create UDF or UDA as it removes all safety mechanisms. Le lun. 28 mars 2022 à 13:23, Sébastien Rebecchi mailto:srebec...@kameleoon.com>> a écrit : Hi Benjamin, Thanks for the answer. Is there a way to customize that default behaviour? If no, could you indicate where to find this class loader in the github of Cassandra please? Le lun. 28 mars 2022 à 12:40, Benjamin Lerer mailto:ble...@apache.org>> a écrit : Hi Sébastien, Cassandra uses a special classloader for UDFs that limi
Re: UDF: adding custom jar to classpath
Hi Ekaterina, I use 4.0.1. But as I said I added a jar in classpath (/usr/share/cassandra/lib/ folder on every node) and I see that the jar is loaded in the classpath from the Cassandra command line. And I have "enable_user_defined_functions: true" and "enable_user_defined_functions_threads: false" in cassandra.yaml. So I don't see what is missing or not done properly. Best regards, Sébastien. Le mer. 6 avr. 2022 à 16:03, Ekaterina Dimitrova a écrit : > Hi Sebastian, > Do you use the latest 4.0.3 version? Those options were added in 4.0.2 I > believe, so if you try them with an earlier version - below message is what > you would get as they didn’t exist. > > Best regards, > Ekaterina > > On Wed, 6 Apr 2022 at 9:53, Sébastien Rebecchi > wrote: > >> Hi Benjamin, Hi everybody, >> >> I found in the documentation that we should add "allow_insecure_udfs: >> true" and optionally "allow_extra_insecure_udfs: true" so that >> "enable_user_defined_functions_threads: false" is really taken into account >> (I understood like that). That would explain why my UDF still does not run >> even with "enable_user_defined_functions_threads: false". Found in >> https://github.com/apache/cassandra/blob/cassandra-4.0/NEWS.txt >> >> So I tried to add "allow_insecure_udfs: true" and >> "allow_extra_insecure_udfs: true" in cassandra.yaml, but then Cassandra >> failed to restart and I got that error in logs "Exception >> (org.apache.cassandra.exceptions.ConfigurationException) encountered during >> startup: Invalid yaml. Please remove properties [allow_insecure_udfs, >> allow_extra_insecure_udfs] from your cassandra.yaml". >> >> Should I understand that we can activate that 2 extra confs only by >> changing source code? That would be really disappointing :( And if no, then >> how to activate all UDF possibilities from cassandra.yaml please? >> >> Thanks in advance, >> >> Sébastien. >> >> >> Le mar. 5 avr. 2022 à 10:36, Benjamin Lerer a écrit : >> >>> Unfortunately, I do not have much time for doing some digging. Sorry for >>> that :-( >>> You should look at JavaBasedUDFunction and UDFExecutorServic. >>> >>> Le lun. 4 avr. 2022 à 17:25, Sébastien Rebecchi >>> a écrit : >>> >>>> Hi! >>>> Do you have any more ideas for me? >>>> Cordially, >>>> Sébastien. >>>> >>>> Le lun. 28 mars 2022 à 16:39, Sébastien Rebecchi < >>>> srebec...@kameleoon.com> a écrit : >>>> >>>>> Unfortunately, it is not working even with >>>>> "enable_user_defined_functions_threads: false" in cassandra.yaml :/ >>>>> Is there any way to check the running configuration? >>>>> >>>>> Le lun. 28 mars 2022 à 15:35, Benjamin Lerer a >>>>> écrit : >>>>> >>>>>> I do not think that allowing to customize UDF classes whitelist has >>>>>> been discussed before. Feel free to open a JIRA ticket :-) >>>>>> I have some plans to revisit how we securise UDFs as the current >>>>>> threading approach has some impact in terms of latency. That can be a >>>>>> good >>>>>> opportunity to look into providing more flexibility. >>>>>> >>>>>> Le lun. 28 mars 2022 à 15:00, Sébastien Rebecchi < >>>>>> srebec...@kameleoon.com> a écrit : >>>>>> >>>>>>> Thanks you very much! I will try that. >>>>>>> As you know, would it be a long-terms solution? Or is there any plan >>>>>>> to add the possibility to customize UDF classes whitelist? >>>>>>> >>>>>>> Le lun. 28 mars 2022 à 14:31, Benjamin Lerer a >>>>>>> écrit : >>>>>>> >>>>>>>> Is there a way to customize that default behaviour? >>>>>>>> >>>>>>>> >>>>>>>> Looking at JavaBasedUDFunction quickly it seems that the >>>>>>>> ClassLoader is only used when you use the UDFExecutorService to >>>>>>>> execute your UDFs. You can try to disable it using >>>>>>>> "enable_user_defined_functions_threads: false" and see if it works. >>>>>>>> Now that also means that you have to ensure that only trusted >>>>>>>> persons can create UDF or UDA a
Re: UDF: adding custom jar to classpath
Hi Sebastian, Do you use the latest 4.0.3 version? Those options were added in 4.0.2 I believe, so if you try them with an earlier version - below message is what you would get as they didn’t exist. Best regards, Ekaterina On Wed, 6 Apr 2022 at 9:53, Sébastien Rebecchi wrote: > Hi Benjamin, Hi everybody, > > I found in the documentation that we should add "allow_insecure_udfs: > true" and optionally "allow_extra_insecure_udfs: true" so that > "enable_user_defined_functions_threads: false" is really taken into account > (I understood like that). That would explain why my UDF still does not run > even with "enable_user_defined_functions_threads: false". Found in > https://github.com/apache/cassandra/blob/cassandra-4.0/NEWS.txt > > So I tried to add "allow_insecure_udfs: true" and > "allow_extra_insecure_udfs: true" in cassandra.yaml, but then Cassandra > failed to restart and I got that error in logs "Exception > (org.apache.cassandra.exceptions.ConfigurationException) encountered during > startup: Invalid yaml. Please remove properties [allow_insecure_udfs, > allow_extra_insecure_udfs] from your cassandra.yaml". > > Should I understand that we can activate that 2 extra confs only by > changing source code? That would be really disappointing :( And if no, then > how to activate all UDF possibilities from cassandra.yaml please? > > Thanks in advance, > > Sébastien. > > > Le mar. 5 avr. 2022 à 10:36, Benjamin Lerer a écrit : > >> Unfortunately, I do not have much time for doing some digging. Sorry for >> that :-( >> You should look at JavaBasedUDFunction and UDFExecutorServic. >> >> Le lun. 4 avr. 2022 à 17:25, Sébastien Rebecchi >> a écrit : >> >>> Hi! >>> Do you have any more ideas for me? >>> Cordially, >>> Sébastien. >>> >>> Le lun. 28 mars 2022 à 16:39, Sébastien Rebecchi < >>> srebec...@kameleoon.com> a écrit : >>> >>>> Unfortunately, it is not working even with >>>> "enable_user_defined_functions_threads: false" in cassandra.yaml :/ >>>> Is there any way to check the running configuration? >>>> >>>> Le lun. 28 mars 2022 à 15:35, Benjamin Lerer a >>>> écrit : >>>> >>>>> I do not think that allowing to customize UDF classes whitelist has >>>>> been discussed before. Feel free to open a JIRA ticket :-) >>>>> I have some plans to revisit how we securise UDFs as the current >>>>> threading approach has some impact in terms of latency. That can be a good >>>>> opportunity to look into providing more flexibility. >>>>> >>>>> Le lun. 28 mars 2022 à 15:00, Sébastien Rebecchi < >>>>> srebec...@kameleoon.com> a écrit : >>>>> >>>>>> Thanks you very much! I will try that. >>>>>> As you know, would it be a long-terms solution? Or is there any plan >>>>>> to add the possibility to customize UDF classes whitelist? >>>>>> >>>>>> Le lun. 28 mars 2022 à 14:31, Benjamin Lerer a >>>>>> écrit : >>>>>> >>>>>>> Is there a way to customize that default behaviour? >>>>>>> >>>>>>> >>>>>>> Looking at JavaBasedUDFunction quickly it seems that the >>>>>>> ClassLoader is only used when you use the UDFExecutorService to >>>>>>> execute your UDFs. You can try to disable it using >>>>>>> "enable_user_defined_functions_threads: false" and see if it works. >>>>>>> Now that also means that you have to ensure that only trusted >>>>>>> persons can create UDF or UDA as it removes all safety mechanisms. >>>>>>> >>>>>>> Le lun. 28 mars 2022 à 13:23, Sébastien Rebecchi < >>>>>>> srebec...@kameleoon.com> a écrit : >>>>>>> >>>>>>>> Hi Benjamin, >>>>>>>> >>>>>>>> Thanks for the answer. >>>>>>>> Is there a way to customize that default behaviour? If no, could >>>>>>>> you indicate where to find this class loader in the github of Cassandra >>>>>>>> please? >>>>>>>> >>>>>>>> Le lun. 28 mars 2022 à 12:40, Benjamin Lerer a >>>>>>>> écrit : >>>>>>>> >>>>>>>>> Hi Sébast
Re: UDF: adding custom jar to classpath
Sorry, I found that allow_insecure_udfs and allow_extra_insecure_udfs have been introduced in 4.0.2 but I run 4.0.1, so that explains the error. So for 4.0.1 "enable_user_defined_functions_threads: false" should be enough, no advance on that I still don't know why my UDF does not compile then Le mer. 6 avr. 2022 à 15:52, Sébastien Rebecchi a écrit : > Hi Benjamin, Hi everybody, > > I found in the documentation that we should add "allow_insecure_udfs: > true" and optionally "allow_extra_insecure_udfs: true" so that > "enable_user_defined_functions_threads: false" is really taken into account > (I understood like that). That would explain why my UDF still does not run > even with "enable_user_defined_functions_threads: false". Found in > https://github.com/apache/cassandra/blob/cassandra-4.0/NEWS.txt > > So I tried to add "allow_insecure_udfs: true" and > "allow_extra_insecure_udfs: true" in cassandra.yaml, but then Cassandra > failed to restart and I got that error in logs "Exception > (org.apache.cassandra.exceptions.ConfigurationException) encountered during > startup: Invalid yaml. Please remove properties [allow_insecure_udfs, > allow_extra_insecure_udfs] from your cassandra.yaml". > > Should I understand that we can activate that 2 extra confs only by > changing source code? That would be really disappointing :( And if no, then > how to activate all UDF possibilities from cassandra.yaml please? > > Thanks in advance, > > Sébastien. > > > Le mar. 5 avr. 2022 à 10:36, Benjamin Lerer a écrit : > >> Unfortunately, I do not have much time for doing some digging. Sorry for >> that :-( >> You should look at JavaBasedUDFunction and UDFExecutorServic. >> >> Le lun. 4 avr. 2022 à 17:25, Sébastien Rebecchi >> a écrit : >> >>> Hi! >>> Do you have any more ideas for me? >>> Cordially, >>> Sébastien. >>> >>> Le lun. 28 mars 2022 à 16:39, Sébastien Rebecchi < >>> srebec...@kameleoon.com> a écrit : >>> >>>> Unfortunately, it is not working even with >>>> "enable_user_defined_functions_threads: false" in cassandra.yaml :/ >>>> Is there any way to check the running configuration? >>>> >>>> Le lun. 28 mars 2022 à 15:35, Benjamin Lerer a >>>> écrit : >>>> >>>>> I do not think that allowing to customize UDF classes whitelist has >>>>> been discussed before. Feel free to open a JIRA ticket :-) >>>>> I have some plans to revisit how we securise UDFs as the current >>>>> threading approach has some impact in terms of latency. That can be a good >>>>> opportunity to look into providing more flexibility. >>>>> >>>>> Le lun. 28 mars 2022 à 15:00, Sébastien Rebecchi < >>>>> srebec...@kameleoon.com> a écrit : >>>>> >>>>>> Thanks you very much! I will try that. >>>>>> As you know, would it be a long-terms solution? Or is there any plan >>>>>> to add the possibility to customize UDF classes whitelist? >>>>>> >>>>>> Le lun. 28 mars 2022 à 14:31, Benjamin Lerer a >>>>>> écrit : >>>>>> >>>>>>> Is there a way to customize that default behaviour? >>>>>>> >>>>>>> >>>>>>> Looking at JavaBasedUDFunction quickly it seems that the >>>>>>> ClassLoader is only used when you use the UDFExecutorService to >>>>>>> execute your UDFs. You can try to disable it using >>>>>>> "enable_user_defined_functions_threads: false" and see if it works. >>>>>>> Now that also means that you have to ensure that only trusted >>>>>>> persons can create UDF or UDA as it removes all safety mechanisms. >>>>>>> >>>>>>> Le lun. 28 mars 2022 à 13:23, Sébastien Rebecchi < >>>>>>> srebec...@kameleoon.com> a écrit : >>>>>>> >>>>>>>> Hi Benjamin, >>>>>>>> >>>>>>>> Thanks for the answer. >>>>>>>> Is there a way to customize that default behaviour? If no, could >>>>>>>> you indicate where to find this class loader in the github of Cassandra >>>>>>>> please? >>>>>>>> >>>>>>>> Le lun. 28 mars 2022 à 12:40, Benjamin Lerer a >>&g
Re: UDF: adding custom jar to classpath
Hi Benjamin, Hi everybody, I found in the documentation that we should add "allow_insecure_udfs: true" and optionally "allow_extra_insecure_udfs: true" so that "enable_user_defined_functions_threads: false" is really taken into account (I understood like that). That would explain why my UDF still does not run even with "enable_user_defined_functions_threads: false". Found in https://github.com/apache/cassandra/blob/cassandra-4.0/NEWS.txt So I tried to add "allow_insecure_udfs: true" and "allow_extra_insecure_udfs: true" in cassandra.yaml, but then Cassandra failed to restart and I got that error in logs "Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: Invalid yaml. Please remove properties [allow_insecure_udfs, allow_extra_insecure_udfs] from your cassandra.yaml". Should I understand that we can activate that 2 extra confs only by changing source code? That would be really disappointing :( And if no, then how to activate all UDF possibilities from cassandra.yaml please? Thanks in advance, Sébastien. Le mar. 5 avr. 2022 à 10:36, Benjamin Lerer a écrit : > Unfortunately, I do not have much time for doing some digging. Sorry for > that :-( > You should look at JavaBasedUDFunction and UDFExecutorServic. > > Le lun. 4 avr. 2022 à 17:25, Sébastien Rebecchi > a écrit : > >> Hi! >> Do you have any more ideas for me? >> Cordially, >> Sébastien. >> >> Le lun. 28 mars 2022 à 16:39, Sébastien Rebecchi >> a écrit : >> >>> Unfortunately, it is not working even with >>> "enable_user_defined_functions_threads: false" in cassandra.yaml :/ >>> Is there any way to check the running configuration? >>> >>> Le lun. 28 mars 2022 à 15:35, Benjamin Lerer a >>> écrit : >>> >>>> I do not think that allowing to customize UDF classes whitelist has >>>> been discussed before. Feel free to open a JIRA ticket :-) >>>> I have some plans to revisit how we securise UDFs as the current >>>> threading approach has some impact in terms of latency. That can be a good >>>> opportunity to look into providing more flexibility. >>>> >>>> Le lun. 28 mars 2022 à 15:00, Sébastien Rebecchi < >>>> srebec...@kameleoon.com> a écrit : >>>> >>>>> Thanks you very much! I will try that. >>>>> As you know, would it be a long-terms solution? Or is there any plan >>>>> to add the possibility to customize UDF classes whitelist? >>>>> >>>>> Le lun. 28 mars 2022 à 14:31, Benjamin Lerer a >>>>> écrit : >>>>> >>>>>> Is there a way to customize that default behaviour? >>>>>> >>>>>> >>>>>> Looking at JavaBasedUDFunction quickly it seems that the ClassLoader >>>>>> is only used when you use the UDFExecutorService to execute your >>>>>> UDFs. You can try to disable it using >>>>>> "enable_user_defined_functions_threads: false" and see if it works. >>>>>> Now that also means that you have to ensure that only trusted persons >>>>>> can create UDF or UDA as it removes all safety mechanisms. >>>>>> >>>>>> Le lun. 28 mars 2022 à 13:23, Sébastien Rebecchi < >>>>>> srebec...@kameleoon.com> a écrit : >>>>>> >>>>>>> Hi Benjamin, >>>>>>> >>>>>>> Thanks for the answer. >>>>>>> Is there a way to customize that default behaviour? If no, could you >>>>>>> indicate where to find this class loader in the github of Cassandra >>>>>>> please? >>>>>>> >>>>>>> Le lun. 28 mars 2022 à 12:40, Benjamin Lerer a >>>>>>> écrit : >>>>>>> >>>>>>>> Hi Sébastien, >>>>>>>> >>>>>>>> Cassandra uses a special classloader for UDFs that limit which >>>>>>>> classes can be used. >>>>>>>> You cannot rely on non-JDK classes for UDFs and some of the JDK >>>>>>>> packages like the IO package for example cannot be used. >>>>>>>> The goal is simply to ensure that UDFs cannot compromise the server >>>>>>>> security. >>>>>>>> >>>>>>>> Le lun. 28 mars 2022 à 11:31, Sébastien Rebecchi < >>>>>>>> srebec...@
Re: UDF: adding custom jar to classpath
Unfortunately, I do not have much time for doing some digging. Sorry for that :-( You should look at JavaBasedUDFunction and UDFExecutorServic. Le lun. 4 avr. 2022 à 17:25, Sébastien Rebecchi a écrit : > Hi! > Do you have any more ideas for me? > Cordially, > Sébastien. > > Le lun. 28 mars 2022 à 16:39, Sébastien Rebecchi > a écrit : > >> Unfortunately, it is not working even with >> "enable_user_defined_functions_threads: false" in cassandra.yaml :/ >> Is there any way to check the running configuration? >> >> Le lun. 28 mars 2022 à 15:35, Benjamin Lerer a >> écrit : >> >>> I do not think that allowing to customize UDF classes whitelist has been >>> discussed before. Feel free to open a JIRA ticket :-) >>> I have some plans to revisit how we securise UDFs as the current >>> threading approach has some impact in terms of latency. That can be a good >>> opportunity to look into providing more flexibility. >>> >>> Le lun. 28 mars 2022 à 15:00, Sébastien Rebecchi < >>> srebec...@kameleoon.com> a écrit : >>> >>>> Thanks you very much! I will try that. >>>> As you know, would it be a long-terms solution? Or is there any plan to >>>> add the possibility to customize UDF classes whitelist? >>>> >>>> Le lun. 28 mars 2022 à 14:31, Benjamin Lerer a >>>> écrit : >>>> >>>>> Is there a way to customize that default behaviour? >>>>> >>>>> >>>>> Looking at JavaBasedUDFunction quickly it seems that the ClassLoader >>>>> is only used when you use the UDFExecutorService to execute your >>>>> UDFs. You can try to disable it using >>>>> "enable_user_defined_functions_threads: false" and see if it works. >>>>> Now that also means that you have to ensure that only trusted persons >>>>> can create UDF or UDA as it removes all safety mechanisms. >>>>> >>>>> Le lun. 28 mars 2022 à 13:23, Sébastien Rebecchi < >>>>> srebec...@kameleoon.com> a écrit : >>>>> >>>>>> Hi Benjamin, >>>>>> >>>>>> Thanks for the answer. >>>>>> Is there a way to customize that default behaviour? If no, could you >>>>>> indicate where to find this class loader in the github of Cassandra >>>>>> please? >>>>>> >>>>>> Le lun. 28 mars 2022 à 12:40, Benjamin Lerer a >>>>>> écrit : >>>>>> >>>>>>> Hi Sébastien, >>>>>>> >>>>>>> Cassandra uses a special classloader for UDFs that limit which >>>>>>> classes can be used. >>>>>>> You cannot rely on non-JDK classes for UDFs and some of the JDK >>>>>>> packages like the IO package for example cannot be used. >>>>>>> The goal is simply to ensure that UDFs cannot compromise the server >>>>>>> security. >>>>>>> >>>>>>> Le lun. 28 mars 2022 à 11:31, Sébastien Rebecchi < >>>>>>> srebec...@kameleoon.com> a écrit : >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> I am trying to create a UDF based on custom methods. >>>>>>>> So I set enable_user_defined_functions to true and added a jar in >>>>>>>> "/usr/share/cassandra/lib/" folder on every node, restarted the nodes >>>>>>>> and I >>>>>>>> can see from the command line that the jar is indeed used (in the >>>>>>>> classpath >>>>>>>> with -cp). >>>>>>>> >>>>>>>> But when i create the UDF I got that error: >>>>>>>> >>>>>>>> CREATE OR REPLACE FUNCTION blobToJson (input blob) RETURNS NULL ON >>>>>>>> NULL INPUT RETURNS text LANGUAGE java AS 'return >>>>>>>> com.kameleoon.visit.Visit.writeToJson(com.kameleoon.visit.Visit.readFromByteBuffer(input));'; >>>>>>>> InvalidRequest: Error from server: code=2200 [Invalid query] >>>>>>>> message="Java source compilation failed: >>>>>>>> Line 1: com.kameleoon.visit.Visit cannot be resolved to a type >>>>>>>> Line 1: com.kameleoon.visit.Visit cannot be resolved to a type >>>>>>>> >>>>>>>> Of course the class com.kameleoon.visit.Visit does exist in the jar >>>>>>>> and the jar has read rights to every user (chmod 444). So I can not >>>>>>>> find >>>>>>>> the reason. >>>>>>>> >>>>>>>> versions are: [cqlsh 6.0.0 | Cassandra 4.0.1 | CQL spec 3.4.5 | >>>>>>>> Native protocol v5] >>>>>>>> >>>>>>>> Any help would be appreciated! >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> Sébastien. >>>>>>>> >>>>>>>
Re: UDF: adding custom jar to classpath
Hi! Do you have any more ideas for me? Cordially, Sébastien. Le lun. 28 mars 2022 à 16:39, Sébastien Rebecchi a écrit : > Unfortunately, it is not working even with > "enable_user_defined_functions_threads: false" in cassandra.yaml :/ > Is there any way to check the running configuration? > > Le lun. 28 mars 2022 à 15:35, Benjamin Lerer a écrit : > >> I do not think that allowing to customize UDF classes whitelist has been >> discussed before. Feel free to open a JIRA ticket :-) >> I have some plans to revisit how we securise UDFs as the current >> threading approach has some impact in terms of latency. That can be a good >> opportunity to look into providing more flexibility. >> >> Le lun. 28 mars 2022 à 15:00, Sébastien Rebecchi >> a écrit : >> >>> Thanks you very much! I will try that. >>> As you know, would it be a long-terms solution? Or is there any plan to >>> add the possibility to customize UDF classes whitelist? >>> >>> Le lun. 28 mars 2022 à 14:31, Benjamin Lerer a >>> écrit : >>> >>>> Is there a way to customize that default behaviour? >>>> >>>> >>>> Looking at JavaBasedUDFunction quickly it seems that the ClassLoader >>>> is only used when you use the UDFExecutorService to execute your UDFs. >>>> You can try to disable it using "enable_user_defined_functions_threads: >>>> false" and see if it works. >>>> Now that also means that you have to ensure that only trusted persons >>>> can create UDF or UDA as it removes all safety mechanisms. >>>> >>>> Le lun. 28 mars 2022 à 13:23, Sébastien Rebecchi < >>>> srebec...@kameleoon.com> a écrit : >>>> >>>>> Hi Benjamin, >>>>> >>>>> Thanks for the answer. >>>>> Is there a way to customize that default behaviour? If no, could you >>>>> indicate where to find this class loader in the github of Cassandra >>>>> please? >>>>> >>>>> Le lun. 28 mars 2022 à 12:40, Benjamin Lerer a >>>>> écrit : >>>>> >>>>>> Hi Sébastien, >>>>>> >>>>>> Cassandra uses a special classloader for UDFs that limit which >>>>>> classes can be used. >>>>>> You cannot rely on non-JDK classes for UDFs and some of the JDK >>>>>> packages like the IO package for example cannot be used. >>>>>> The goal is simply to ensure that UDFs cannot compromise the server >>>>>> security. >>>>>> >>>>>> Le lun. 28 mars 2022 à 11:31, Sébastien Rebecchi < >>>>>> srebec...@kameleoon.com> a écrit : >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I am trying to create a UDF based on custom methods. >>>>>>> So I set enable_user_defined_functions to true and added a jar in >>>>>>> "/usr/share/cassandra/lib/" folder on every node, restarted the nodes >>>>>>> and I >>>>>>> can see from the command line that the jar is indeed used (in the >>>>>>> classpath >>>>>>> with -cp). >>>>>>> >>>>>>> But when i create the UDF I got that error: >>>>>>> >>>>>>> CREATE OR REPLACE FUNCTION blobToJson (input blob) RETURNS NULL ON >>>>>>> NULL INPUT RETURNS text LANGUAGE java AS 'return >>>>>>> com.kameleoon.visit.Visit.writeToJson(com.kameleoon.visit.Visit.readFromByteBuffer(input));'; >>>>>>> InvalidRequest: Error from server: code=2200 [Invalid query] >>>>>>> message="Java source compilation failed: >>>>>>> Line 1: com.kameleoon.visit.Visit cannot be resolved to a type >>>>>>> Line 1: com.kameleoon.visit.Visit cannot be resolved to a type >>>>>>> >>>>>>> Of course the class com.kameleoon.visit.Visit does exist in the jar >>>>>>> and the jar has read rights to every user (chmod 444). So I can not find >>>>>>> the reason. >>>>>>> >>>>>>> versions are: [cqlsh 6.0.0 | Cassandra 4.0.1 | CQL spec 3.4.5 | >>>>>>> Native protocol v5] >>>>>>> >>>>>>> Any help would be appreciated! >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> Sébastien. >>>>>>> >>>>>>
Re: UDF: adding custom jar to classpath
Unfortunately, it is not working even with "enable_user_defined_functions_threads: false" in cassandra.yaml :/ Is there any way to check the running configuration? Le lun. 28 mars 2022 à 15:35, Benjamin Lerer a écrit : > I do not think that allowing to customize UDF classes whitelist has been > discussed before. Feel free to open a JIRA ticket :-) > I have some plans to revisit how we securise UDFs as the current threading > approach has some impact in terms of latency. That can be a good > opportunity to look into providing more flexibility. > > Le lun. 28 mars 2022 à 15:00, Sébastien Rebecchi > a écrit : > >> Thanks you very much! I will try that. >> As you know, would it be a long-terms solution? Or is there any plan to >> add the possibility to customize UDF classes whitelist? >> >> Le lun. 28 mars 2022 à 14:31, Benjamin Lerer a >> écrit : >> >>> Is there a way to customize that default behaviour? >>> >>> >>> Looking at JavaBasedUDFunction quickly it seems that the ClassLoader is >>> only used when you use the UDFExecutorService to execute your UDFs. You >>> can try to disable it using "enable_user_defined_functions_threads: false" >>> and see if it works. >>> Now that also means that you have to ensure that only trusted persons >>> can create UDF or UDA as it removes all safety mechanisms. >>> >>> Le lun. 28 mars 2022 à 13:23, Sébastien Rebecchi < >>> srebec...@kameleoon.com> a écrit : >>> >>>> Hi Benjamin, >>>> >>>> Thanks for the answer. >>>> Is there a way to customize that default behaviour? If no, could you >>>> indicate where to find this class loader in the github of Cassandra please? >>>> >>>> Le lun. 28 mars 2022 à 12:40, Benjamin Lerer a >>>> écrit : >>>> >>>>> Hi Sébastien, >>>>> >>>>> Cassandra uses a special classloader for UDFs that limit which classes >>>>> can be used. >>>>> You cannot rely on non-JDK classes for UDFs and some of the JDK >>>>> packages like the IO package for example cannot be used. >>>>> The goal is simply to ensure that UDFs cannot compromise the server >>>>> security. >>>>> >>>>> Le lun. 28 mars 2022 à 11:31, Sébastien Rebecchi < >>>>> srebec...@kameleoon.com> a écrit : >>>>> >>>>>> Hello, >>>>>> >>>>>> I am trying to create a UDF based on custom methods. >>>>>> So I set enable_user_defined_functions to true and added a jar in >>>>>> "/usr/share/cassandra/lib/" folder on every node, restarted the nodes >>>>>> and I >>>>>> can see from the command line that the jar is indeed used (in the >>>>>> classpath >>>>>> with -cp). >>>>>> >>>>>> But when i create the UDF I got that error: >>>>>> >>>>>> CREATE OR REPLACE FUNCTION blobToJson (input blob) RETURNS NULL ON >>>>>> NULL INPUT RETURNS text LANGUAGE java AS 'return >>>>>> com.kameleoon.visit.Visit.writeToJson(com.kameleoon.visit.Visit.readFromByteBuffer(input));'; >>>>>> InvalidRequest: Error from server: code=2200 [Invalid query] >>>>>> message="Java source compilation failed: >>>>>> Line 1: com.kameleoon.visit.Visit cannot be resolved to a type >>>>>> Line 1: com.kameleoon.visit.Visit cannot be resolved to a type >>>>>> >>>>>> Of course the class com.kameleoon.visit.Visit does exist in the jar >>>>>> and the jar has read rights to every user (chmod 444). So I can not find >>>>>> the reason. >>>>>> >>>>>> versions are: [cqlsh 6.0.0 | Cassandra 4.0.1 | CQL spec 3.4.5 | >>>>>> Native protocol v5] >>>>>> >>>>>> Any help would be appreciated! >>>>>> >>>>>> Thanks! >>>>>> >>>>>> Sébastien. >>>>>> >>>>>
Re: UDF: adding custom jar to classpath
I do not think that allowing to customize UDF classes whitelist has been discussed before. Feel free to open a JIRA ticket :-) I have some plans to revisit how we securise UDFs as the current threading approach has some impact in terms of latency. That can be a good opportunity to look into providing more flexibility. Le lun. 28 mars 2022 à 15:00, Sébastien Rebecchi a écrit : > Thanks you very much! I will try that. > As you know, would it be a long-terms solution? Or is there any plan to > add the possibility to customize UDF classes whitelist? > > Le lun. 28 mars 2022 à 14:31, Benjamin Lerer a écrit : > >> Is there a way to customize that default behaviour? >> >> >> Looking at JavaBasedUDFunction quickly it seems that the ClassLoader is >> only used when you use the UDFExecutorService to execute your UDFs. You >> can try to disable it using "enable_user_defined_functions_threads: false" >> and see if it works. >> Now that also means that you have to ensure that only trusted persons can >> create UDF or UDA as it removes all safety mechanisms. >> >> Le lun. 28 mars 2022 à 13:23, Sébastien Rebecchi >> a écrit : >> >>> Hi Benjamin, >>> >>> Thanks for the answer. >>> Is there a way to customize that default behaviour? If no, could you >>> indicate where to find this class loader in the github of Cassandra please? >>> >>> Le lun. 28 mars 2022 à 12:40, Benjamin Lerer a >>> écrit : >>> >>>> Hi Sébastien, >>>> >>>> Cassandra uses a special classloader for UDFs that limit which classes >>>> can be used. >>>> You cannot rely on non-JDK classes for UDFs and some of the JDK >>>> packages like the IO package for example cannot be used. >>>> The goal is simply to ensure that UDFs cannot compromise the server >>>> security. >>>> >>>> Le lun. 28 mars 2022 à 11:31, Sébastien Rebecchi < >>>> srebec...@kameleoon.com> a écrit : >>>> >>>>> Hello, >>>>> >>>>> I am trying to create a UDF based on custom methods. >>>>> So I set enable_user_defined_functions to true and added a jar in >>>>> "/usr/share/cassandra/lib/" folder on every node, restarted the nodes and >>>>> I >>>>> can see from the command line that the jar is indeed used (in the >>>>> classpath >>>>> with -cp). >>>>> >>>>> But when i create the UDF I got that error: >>>>> >>>>> CREATE OR REPLACE FUNCTION blobToJson (input blob) RETURNS NULL ON >>>>> NULL INPUT RETURNS text LANGUAGE java AS 'return >>>>> com.kameleoon.visit.Visit.writeToJson(com.kameleoon.visit.Visit.readFromByteBuffer(input));'; >>>>> InvalidRequest: Error from server: code=2200 [Invalid query] >>>>> message="Java source compilation failed: >>>>> Line 1: com.kameleoon.visit.Visit cannot be resolved to a type >>>>> Line 1: com.kameleoon.visit.Visit cannot be resolved to a type >>>>> >>>>> Of course the class com.kameleoon.visit.Visit does exist in the jar >>>>> and the jar has read rights to every user (chmod 444). So I can not find >>>>> the reason. >>>>> >>>>> versions are: [cqlsh 6.0.0 | Cassandra 4.0.1 | CQL spec 3.4.5 | Native >>>>> protocol v5] >>>>> >>>>> Any help would be appreciated! >>>>> >>>>> Thanks! >>>>> >>>>> Sébastien. >>>>> >>>>
Re: UDF: adding custom jar to classpath
Thanks you very much! I will try that. As you know, would it be a long-terms solution? Or is there any plan to add the possibility to customize UDF classes whitelist? Le lun. 28 mars 2022 à 14:31, Benjamin Lerer a écrit : > Is there a way to customize that default behaviour? > > > Looking at JavaBasedUDFunction quickly it seems that the ClassLoader is > only used when you use the UDFExecutorService to execute your UDFs. You > can try to disable it using "enable_user_defined_functions_threads: false" > and see if it works. > Now that also means that you have to ensure that only trusted persons can > create UDF or UDA as it removes all safety mechanisms. > > Le lun. 28 mars 2022 à 13:23, Sébastien Rebecchi > a écrit : > >> Hi Benjamin, >> >> Thanks for the answer. >> Is there a way to customize that default behaviour? If no, could you >> indicate where to find this class loader in the github of Cassandra please? >> >> Le lun. 28 mars 2022 à 12:40, Benjamin Lerer a >> écrit : >> >>> Hi Sébastien, >>> >>> Cassandra uses a special classloader for UDFs that limit which classes >>> can be used. >>> You cannot rely on non-JDK classes for UDFs and some of the JDK packages >>> like the IO package for example cannot be used. >>> The goal is simply to ensure that UDFs cannot compromise the server >>> security. >>> >>> Le lun. 28 mars 2022 à 11:31, Sébastien Rebecchi < >>> srebec...@kameleoon.com> a écrit : >>> >>>> Hello, >>>> >>>> I am trying to create a UDF based on custom methods. >>>> So I set enable_user_defined_functions to true and added a jar in >>>> "/usr/share/cassandra/lib/" folder on every node, restarted the nodes and I >>>> can see from the command line that the jar is indeed used (in the classpath >>>> with -cp). >>>> >>>> But when i create the UDF I got that error: >>>> >>>> CREATE OR REPLACE FUNCTION blobToJson (input blob) RETURNS NULL ON NULL >>>> INPUT RETURNS text LANGUAGE java AS 'return >>>> com.kameleoon.visit.Visit.writeToJson(com.kameleoon.visit.Visit.readFromByteBuffer(input));'; >>>> InvalidRequest: Error from server: code=2200 [Invalid query] >>>> message="Java source compilation failed: >>>> Line 1: com.kameleoon.visit.Visit cannot be resolved to a type >>>> Line 1: com.kameleoon.visit.Visit cannot be resolved to a type >>>> >>>> Of course the class com.kameleoon.visit.Visit does exist in the jar and >>>> the jar has read rights to every user (chmod 444). So I can not find the >>>> reason. >>>> >>>> versions are: [cqlsh 6.0.0 | Cassandra 4.0.1 | CQL spec 3.4.5 | Native >>>> protocol v5] >>>> >>>> Any help would be appreciated! >>>> >>>> Thanks! >>>> >>>> Sébastien. >>>> >>>
Re: UDF: adding custom jar to classpath
> > Is there a way to customize that default behaviour? Looking at JavaBasedUDFunction quickly it seems that the ClassLoader is only used when you use the UDFExecutorService to execute your UDFs. You can try to disable it using "enable_user_defined_functions_threads: false" and see if it works. Now that also means that you have to ensure that only trusted persons can create UDF or UDA as it removes all safety mechanisms. Le lun. 28 mars 2022 à 13:23, Sébastien Rebecchi a écrit : > Hi Benjamin, > > Thanks for the answer. > Is there a way to customize that default behaviour? If no, could you > indicate where to find this class loader in the github of Cassandra please? > > Le lun. 28 mars 2022 à 12:40, Benjamin Lerer a écrit : > >> Hi Sébastien, >> >> Cassandra uses a special classloader for UDFs that limit which classes >> can be used. >> You cannot rely on non-JDK classes for UDFs and some of the JDK packages >> like the IO package for example cannot be used. >> The goal is simply to ensure that UDFs cannot compromise the server >> security. >> >> Le lun. 28 mars 2022 à 11:31, Sébastien Rebecchi >> a écrit : >> >>> Hello, >>> >>> I am trying to create a UDF based on custom methods. >>> So I set enable_user_defined_functions to true and added a jar in >>> "/usr/share/cassandra/lib/" folder on every node, restarted the nodes and I >>> can see from the command line that the jar is indeed used (in the classpath >>> with -cp). >>> >>> But when i create the UDF I got that error: >>> >>> CREATE OR REPLACE FUNCTION blobToJson (input blob) RETURNS NULL ON NULL >>> INPUT RETURNS text LANGUAGE java AS 'return >>> com.kameleoon.visit.Visit.writeToJson(com.kameleoon.visit.Visit.readFromByteBuffer(input));'; >>> InvalidRequest: Error from server: code=2200 [Invalid query] >>> message="Java source compilation failed: >>> Line 1: com.kameleoon.visit.Visit cannot be resolved to a type >>> Line 1: com.kameleoon.visit.Visit cannot be resolved to a type >>> >>> Of course the class com.kameleoon.visit.Visit does exist in the jar and >>> the jar has read rights to every user (chmod 444). So I can not find the >>> reason. >>> >>> versions are: [cqlsh 6.0.0 | Cassandra 4.0.1 | CQL spec 3.4.5 | Native >>> protocol v5] >>> >>> Any help would be appreciated! >>> >>> Thanks! >>> >>> Sébastien. >>> >>
Re: UDF: adding custom jar to classpath
Hi Benjamin, Thanks for the answer. Is there a way to customize that default behaviour? If no, could you indicate where to find this class loader in the github of Cassandra please? Le lun. 28 mars 2022 à 12:40, Benjamin Lerer a écrit : > Hi Sébastien, > > Cassandra uses a special classloader for UDFs that limit which classes can > be used. > You cannot rely on non-JDK classes for UDFs and some of the JDK packages > like the IO package for example cannot be used. > The goal is simply to ensure that UDFs cannot compromise the server > security. > > Le lun. 28 mars 2022 à 11:31, Sébastien Rebecchi > a écrit : > >> Hello, >> >> I am trying to create a UDF based on custom methods. >> So I set enable_user_defined_functions to true and added a jar in >> "/usr/share/cassandra/lib/" folder on every node, restarted the nodes and I >> can see from the command line that the jar is indeed used (in the classpath >> with -cp). >> >> But when i create the UDF I got that error: >> >> CREATE OR REPLACE FUNCTION blobToJson (input blob) RETURNS NULL ON NULL >> INPUT RETURNS text LANGUAGE java AS 'return >> com.kameleoon.visit.Visit.writeToJson(com.kameleoon.visit.Visit.readFromByteBuffer(input));'; >> InvalidRequest: Error from server: code=2200 [Invalid query] >> message="Java source compilation failed: >> Line 1: com.kameleoon.visit.Visit cannot be resolved to a type >> Line 1: com.kameleoon.visit.Visit cannot be resolved to a type >> >> Of course the class com.kameleoon.visit.Visit does exist in the jar and >> the jar has read rights to every user (chmod 444). So I can not find the >> reason. >> >> versions are: [cqlsh 6.0.0 | Cassandra 4.0.1 | CQL spec 3.4.5 | Native >> protocol v5] >> >> Any help would be appreciated! >> >> Thanks! >> >> Sébastien. >> >
Re: UDF: adding custom jar to classpath
Hi Sébastien, Cassandra uses a special classloader for UDFs that limit which classes can be used. You cannot rely on non-JDK classes for UDFs and some of the JDK packages like the IO package for example cannot be used. The goal is simply to ensure that UDFs cannot compromise the server security. Le lun. 28 mars 2022 à 11:31, Sébastien Rebecchi a écrit : > Hello, > > I am trying to create a UDF based on custom methods. > So I set enable_user_defined_functions to true and added a jar in > "/usr/share/cassandra/lib/" folder on every node, restarted the nodes and I > can see from the command line that the jar is indeed used (in the classpath > with -cp). > > But when i create the UDF I got that error: > > CREATE OR REPLACE FUNCTION blobToJson (input blob) RETURNS NULL ON NULL > INPUT RETURNS text LANGUAGE java AS 'return > com.kameleoon.visit.Visit.writeToJson(com.kameleoon.visit.Visit.readFromByteBuffer(input));'; > InvalidRequest: Error from server: code=2200 [Invalid query] message="Java > source compilation failed: > Line 1: com.kameleoon.visit.Visit cannot be resolved to a type > Line 1: com.kameleoon.visit.Visit cannot be resolved to a type > > Of course the class com.kameleoon.visit.Visit does exist in the jar and > the jar has read rights to every user (chmod 444). So I can not find the > reason. > > versions are: [cqlsh 6.0.0 | Cassandra 4.0.1 | CQL spec 3.4.5 | Native > protocol v5] > > Any help would be appreciated! > > Thanks! > > Sébastien. >
UDF: adding custom jar to classpath
Hello, I am trying to create a UDF based on custom methods. So I set enable_user_defined_functions to true and added a jar in "/usr/share/cassandra/lib/" folder on every node, restarted the nodes and I can see from the command line that the jar is indeed used (in the classpath with -cp). But when i create the UDF I got that error: CREATE OR REPLACE FUNCTION blobToJson (input blob) RETURNS NULL ON NULL INPUT RETURNS text LANGUAGE java AS 'return com.kameleoon.visit.Visit.writeToJson(com.kameleoon.visit.Visit.readFromByteBuffer(input));'; InvalidRequest: Error from server: code=2200 [Invalid query] message="Java source compilation failed: Line 1: com.kameleoon.visit.Visit cannot be resolved to a type Line 1: com.kameleoon.visit.Visit cannot be resolved to a type Of course the class com.kameleoon.visit.Visit does exist in the jar and the jar has read rights to every user (chmod 444). So I can not find the reason. versions are: [cqlsh 6.0.0 | Cassandra 4.0.1 | CQL spec 3.4.5 | Native protocol v5] Any help would be appreciated! Thanks! Sébastien.
Re: UDF future
CASSANDRA-17280 and CASSANDRA-17281 were opened to cover the work. Thank you On Sun, 23 Jan 2022 at 11:46, Joshua McKenzie wrote: > Lazy consensus seems like a safe path here. > > https://community.apache.org/committers/lazyConsensus.html > > On Sat, Jan 22, 2022 at 4:24 PM Ekaterina Dimitrova > wrote: > >> >> Hi everyone, >> >> Thank you for responding to my email. >> >> Looking at the responses, I guess I should directly open tickets - one >> for deprecating in 4.0.2 (just warning and documenting it) and one for the >> removal/providing interface for trunk? >> >> Do we need more formal voting or anything else that I might be missing? >> Seems there is a consensus? >> >> Thank you, >> Ekaterina >> >> On Fri, 21 Jan 2022 at 12:44, Aleksei Zotov wrote: >> >>> +1 >>> >>> Best Regards, >>> >>> Aleksei Zotov. >>> >>> >>> On Thu, Jan 20, 2022 at 11:52 AM Marcus Eriksson >>> wrote: >>> >>>> +1 >>>> >>>> On Tue, Jan 18, 2022 at 11:30:01AM -0500, Ekaterina Dimitrova wrote: >>>> > Hi everyone, >>>> > >>>> > With the work to add Java 17 support for Cassandra, a new question >>>> around >>>> > the future of UDF was raised. The scripted UDF was using Nashorn >>>> which is >>>> > no longer packaged with the JDK. There are options to add new >>>> dependencies >>>> > to Graal JS for example but it seems people are not sure that it is >>>> worth >>>> > it. Please check the discussion on CASSANDRA-16895. >>>> > >>>> > The following suggestion was made by Marcus and supported by other PMC >>>> > members - "I think we should deprecate scripted UDFs now and drop >>>> them from >>>> > the next major, but possibly provide hooks for people to write their >>>> own >>>> > UDF "engines" and break out the current javascript implementation in >>>> to its >>>> > own repository (but not ship it with Cassandra)." >>>> > >>>> > As a result we decided to engage with our users and created a Twitter >>>> > survey. Results below: >>>> > >>>> > *We would love to understand how you use ApacheCassandra UDFs and >>>> UDAs.* >>>> > >>>> > *32 people responded as follows:* >>>> > >>>> >- *We do not use them - 75%* >>>> >- *We only use Java UDFs - 22%* >>>> >- *We only use JS UDFs - 0%* >>>> >- *We use Java and JS UDFs - 3%* >>>> > >>>> > We also received feedback on LinkedIN on the topic - >>>> > >>>> https://www.linkedin.com/feed/update/urn:li:activity:6886728406742970369?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6886793921020608512%29=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6887421509485248512%29 >>>> > >>>> > Thoughts? >>>> > >>>> > Best regards, >>>> > Ekaterina >>>> >>>
Re: UDF future
Lazy consensus seems like a safe path here. https://community.apache.org/committers/lazyConsensus.html On Sat, Jan 22, 2022 at 4:24 PM Ekaterina Dimitrova wrote: > > Hi everyone, > > Thank you for responding to my email. > > Looking at the responses, I guess I should directly open tickets - one for > deprecating in 4.0.2 (just warning and documenting it) and one for the > removal/providing interface for trunk? > > Do we need more formal voting or anything else that I might be missing? > Seems there is a consensus? > > Thank you, > Ekaterina > > On Fri, 21 Jan 2022 at 12:44, Aleksei Zotov wrote: > >> +1 >> >> Best Regards, >> >> Aleksei Zotov. >> >> >> On Thu, Jan 20, 2022 at 11:52 AM Marcus Eriksson >> wrote: >> >>> +1 >>> >>> On Tue, Jan 18, 2022 at 11:30:01AM -0500, Ekaterina Dimitrova wrote: >>> > Hi everyone, >>> > >>> > With the work to add Java 17 support for Cassandra, a new question >>> around >>> > the future of UDF was raised. The scripted UDF was using Nashorn which >>> is >>> > no longer packaged with the JDK. There are options to add new >>> dependencies >>> > to Graal JS for example but it seems people are not sure that it is >>> worth >>> > it. Please check the discussion on CASSANDRA-16895. >>> > >>> > The following suggestion was made by Marcus and supported by other PMC >>> > members - "I think we should deprecate scripted UDFs now and drop them >>> from >>> > the next major, but possibly provide hooks for people to write their >>> own >>> > UDF "engines" and break out the current javascript implementation in >>> to its >>> > own repository (but not ship it with Cassandra)." >>> > >>> > As a result we decided to engage with our users and created a Twitter >>> > survey. Results below: >>> > >>> > *We would love to understand how you use ApacheCassandra UDFs and >>> UDAs.* >>> > >>> > *32 people responded as follows:* >>> > >>> >- *We do not use them - 75%* >>> >- *We only use Java UDFs - 22%* >>> >- *We only use JS UDFs - 0%* >>> >- *We use Java and JS UDFs - 3%* >>> > >>> > We also received feedback on LinkedIN on the topic - >>> > >>> https://www.linkedin.com/feed/update/urn:li:activity:6886728406742970369?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6886793921020608512%29=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6887421509485248512%29 >>> > >>> > Thoughts? >>> > >>> > Best regards, >>> > Ekaterina >>> >>
Re: UDF future
Hi everyone, Thank you for responding to my email. Looking at the responses, I guess I should directly open tickets - one for deprecating in 4.0.2 (just warning and documenting it) and one for the removal/providing interface for trunk? Do we need more formal voting or anything else that I might be missing? Seems there is a consensus? Thank you, Ekaterina On Fri, 21 Jan 2022 at 12:44, Aleksei Zotov wrote: > +1 > > Best Regards, > > Aleksei Zotov. > > > On Thu, Jan 20, 2022 at 11:52 AM Marcus Eriksson > wrote: > >> +1 >> >> On Tue, Jan 18, 2022 at 11:30:01AM -0500, Ekaterina Dimitrova wrote: >> > Hi everyone, >> > >> > With the work to add Java 17 support for Cassandra, a new question >> around >> > the future of UDF was raised. The scripted UDF was using Nashorn which >> is >> > no longer packaged with the JDK. There are options to add new >> dependencies >> > to Graal JS for example but it seems people are not sure that it is >> worth >> > it. Please check the discussion on CASSANDRA-16895. >> > >> > The following suggestion was made by Marcus and supported by other PMC >> > members - "I think we should deprecate scripted UDFs now and drop them >> from >> > the next major, but possibly provide hooks for people to write their own >> > UDF "engines" and break out the current javascript implementation in to >> its >> > own repository (but not ship it with Cassandra)." >> > >> > As a result we decided to engage with our users and created a Twitter >> > survey. Results below: >> > >> > *We would love to understand how you use ApacheCassandra UDFs and UDAs.* >> > >> > *32 people responded as follows:* >> > >> >- *We do not use them - 75%* >> >- *We only use Java UDFs - 22%* >> >- *We only use JS UDFs - 0%* >> >- *We use Java and JS UDFs - 3%* >> > >> > We also received feedback on LinkedIN on the topic - >> > >> https://www.linkedin.com/feed/update/urn:li:activity:6886728406742970369?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6886793921020608512%29=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6887421509485248512%29 >> > >> > Thoughts? >> > >> > Best regards, >> > Ekaterina >> >
Re: UDF future
+1 Best Regards, Aleksei Zotov. On Thu, Jan 20, 2022 at 11:52 AM Marcus Eriksson wrote: > +1 > > On Tue, Jan 18, 2022 at 11:30:01AM -0500, Ekaterina Dimitrova wrote: > > Hi everyone, > > > > With the work to add Java 17 support for Cassandra, a new question around > > the future of UDF was raised. The scripted UDF was using Nashorn which is > > no longer packaged with the JDK. There are options to add new > dependencies > > to Graal JS for example but it seems people are not sure that it is worth > > it. Please check the discussion on CASSANDRA-16895. > > > > The following suggestion was made by Marcus and supported by other PMC > > members - "I think we should deprecate scripted UDFs now and drop them > from > > the next major, but possibly provide hooks for people to write their own > > UDF "engines" and break out the current javascript implementation in to > its > > own repository (but not ship it with Cassandra)." > > > > As a result we decided to engage with our users and created a Twitter > > survey. Results below: > > > > *We would love to understand how you use ApacheCassandra UDFs and UDAs.* > > > > *32 people responded as follows:* > > > >- *We do not use them - 75%* > >- *We only use Java UDFs - 22%* > >- *We only use JS UDFs - 0%* > >- *We use Java and JS UDFs - 3%* > > > > We also received feedback on LinkedIN on the topic - > > > https://www.linkedin.com/feed/update/urn:li:activity:6886728406742970369?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6886793921020608512%29=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6887421509485248512%29 > > > > Thoughts? > > > > Best regards, > > Ekaterina >
Re: UDF future
+1 On Tue, Jan 18, 2022 at 11:30:01AM -0500, Ekaterina Dimitrova wrote: > Hi everyone, > > With the work to add Java 17 support for Cassandra, a new question around > the future of UDF was raised. The scripted UDF was using Nashorn which is > no longer packaged with the JDK. There are options to add new dependencies > to Graal JS for example but it seems people are not sure that it is worth > it. Please check the discussion on CASSANDRA-16895. > > The following suggestion was made by Marcus and supported by other PMC > members - "I think we should deprecate scripted UDFs now and drop them from > the next major, but possibly provide hooks for people to write their own > UDF "engines" and break out the current javascript implementation in to its > own repository (but not ship it with Cassandra)." > > As a result we decided to engage with our users and created a Twitter > survey. Results below: > > *We would love to understand how you use ApacheCassandra UDFs and UDAs.* > > *32 people responded as follows:* > >- *We do not use them - 75%* >- *We only use Java UDFs - 22%* >- *We only use JS UDFs - 0%* >- *We use Java and JS UDFs - 3%* > > We also received feedback on LinkedIN on the topic - > https://www.linkedin.com/feed/update/urn:li:activity:6886728406742970369?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6886793921020608512%29=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6887421509485248512%29 > > Thoughts? > > Best regards, > Ekaterina
Re: UDF future
+1 to deprecate, drop, add hooks. On Wed, Jan 19, 2022 at 2:22 PM Brandon Williams wrote: > Yes, just javascript. > > On Wed, Jan 19, 2022 at 1:20 PM Yifan Cai wrote: > >> > >> I think we should deprecate scripted UDFs now and drop them from the > next major, but possibly provide hooks for people to write their own UDF > "engines" and break out the current javascript implementation in to its own > repository (but not ship it with Cassandra). > > > > > > +1 > > > > Just want to clarify, is the scripted UDF the one defined using > javascript? > > > > > > > > > > On Wed, Jan 19, 2022 at 9:41 AM Francisco Guerrero > wrote: > >> > >> +1 (nb) > >> > >> On 2022/01/19 15:10:20 Brandon Williams wrote: > >> > We can for completeness, but even with twice as much usage reported > as the > >> > other methods, I don't think it will affect the outcome of the vote. > >> > > >> > On Wed, Jan 19, 2022, 7:25 AM Paulo Motta > wrote: > >> > > >> > > This proposal looks good to me, +1. I was wondering if we should > not run > >> > > this proposal on the user@ list to check if there's any additional > >> > > feedback in addition to the informal Twitter and Linkedin channels? > >> > > > >> > > Em qua., 19 de jan. de 2022 às 10:18, Sylwester Lachiewicz < > >> > > slachiew...@gmail.com> escreveu: > >> > > > >> > >> +1 (Nb) > >> > >> > >> > >> śr., 19 sty 2022, 12:31 użytkownik Brandon Williams < > dri...@gmail.com> > >> > >> napisał: > >> > >> > >> > >>> +1 > >> > >>> > >> > >>> On Tue, Jan 18, 2022 at 10:30 AM Ekaterina Dimitrova > >> > >>> wrote: > >> > >>> > > >> > >>> > Hi everyone, > >> > >>> > > >> > >>> > With the work to add Java 17 support for Cassandra, a new > question > >> > >>> around the future of UDF was raised. The scripted UDF was using > Nashorn > >> > >>> which is no longer packaged with the JDK. There are options to > add new > >> > >>> dependencies to Graal JS for example but it seems people are not > sure that > >> > >>> it is worth it. Please check the discussion on CASSANDRA-16895. > >> > >>> > > >> > >>> > The following suggestion was made by Marcus and supported by > other PMC > >> > >>> members - "I think we should deprecate scripted UDFs now and drop > them from > >> > >>> the next major, but possibly provide hooks for people to write > their own > >> > >>> UDF "engines" and break out the current javascript implementation > in to its > >> > >>> own repository (but not ship it with Cassandra)." > >> > >>> > > >> > >>> > As a result we decided to engage with our users and created a > Twitter > >> > >>> survey. Results below: > >> > >>> > > >> > >>> > We would love to understand how you use ApacheCassandra UDFs > and UDAs. > >> > >>> > > >> > >>> > 32 people responded as follows: > >> > >>> > > >> > >>> > We do not use them - 75% > >> > >>> > We only use Java UDFs - 22% > >> > >>> > We only use JS UDFs - 0% > >> > >>> > We use Java and JS UDFs - 3% > >> > >>> > > >> > >>> > We also received feedback on LinkedIN on the topic - > >> > >>> > https://www.linkedin.com/feed/update/urn:li:activity:6886728406742970369?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6886793921020608512%29=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6887421509485248512%29 > >> > >>> > > >> > >>> > > >> > >>> > Thoughts? > >> > >>> > > >> > >>> > Best regards, > >> > >>> > Ekaterina > >> > >>> > >> > >> > >> > >
Re: UDF future
Yes, just javascript. On Wed, Jan 19, 2022 at 1:20 PM Yifan Cai wrote: >> >> I think we should deprecate scripted UDFs now and drop them from the next >> major, but possibly provide hooks for people to write their own UDF >> "engines" and break out the current javascript implementation in to its own >> repository (but not ship it with Cassandra). > > > +1 > > Just want to clarify, is the scripted UDF the one defined using javascript? > > > > > On Wed, Jan 19, 2022 at 9:41 AM Francisco Guerrero wrote: >> >> +1 (nb) >> >> On 2022/01/19 15:10:20 Brandon Williams wrote: >> > We can for completeness, but even with twice as much usage reported as the >> > other methods, I don't think it will affect the outcome of the vote. >> > >> > On Wed, Jan 19, 2022, 7:25 AM Paulo Motta wrote: >> > >> > > This proposal looks good to me, +1. I was wondering if we should not run >> > > this proposal on the user@ list to check if there's any additional >> > > feedback in addition to the informal Twitter and Linkedin channels? >> > > >> > > Em qua., 19 de jan. de 2022 às 10:18, Sylwester Lachiewicz < >> > > slachiew...@gmail.com> escreveu: >> > > >> > >> +1 (Nb) >> > >> >> > >> śr., 19 sty 2022, 12:31 użytkownik Brandon Williams >> > >> napisał: >> > >> >> > >>> +1 >> > >>> >> > >>> On Tue, Jan 18, 2022 at 10:30 AM Ekaterina Dimitrova >> > >>> wrote: >> > >>> > >> > >>> > Hi everyone, >> > >>> > >> > >>> > With the work to add Java 17 support for Cassandra, a new question >> > >>> around the future of UDF was raised. The scripted UDF was using Nashorn >> > >>> which is no longer packaged with the JDK. There are options to add new >> > >>> dependencies to Graal JS for example but it seems people are not sure >> > >>> that >> > >>> it is worth it. Please check the discussion on CASSANDRA-16895. >> > >>> > >> > >>> > The following suggestion was made by Marcus and supported by other >> > >>> > PMC >> > >>> members - "I think we should deprecate scripted UDFs now and drop them >> > >>> from >> > >>> the next major, but possibly provide hooks for people to write their >> > >>> own >> > >>> UDF "engines" and break out the current javascript implementation in >> > >>> to its >> > >>> own repository (but not ship it with Cassandra)." >> > >>> > >> > >>> > As a result we decided to engage with our users and created a Twitter >> > >>> survey. Results below: >> > >>> > >> > >>> > We would love to understand how you use ApacheCassandra UDFs and >> > >>> > UDAs. >> > >>> > >> > >>> > 32 people responded as follows: >> > >>> > >> > >>> > We do not use them - 75% >> > >>> > We only use Java UDFs - 22% >> > >>> > We only use JS UDFs - 0% >> > >>> > We use Java and JS UDFs - 3% >> > >>> > >> > >>> > We also received feedback on LinkedIN on the topic - >> > >>> https://www.linkedin.com/feed/update/urn:li:activity:6886728406742970369?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6886793921020608512%29=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6887421509485248512%29 >> > >>> > >> > >>> > >> > >>> > Thoughts? >> > >>> > >> > >>> > Best regards, >> > >>> > Ekaterina >> > >>> >> > >> >> >
Re: UDF future
> > I think we should deprecate scripted UDFs now and drop them from the next > major, but possibly provide hooks for people to write their own UDF > "engines" and break out the current javascript implementation in to its own > repository (but not ship it with Cassandra). +1 Just want to clarify, is the scripted UDF the one defined using javascript? On Wed, Jan 19, 2022 at 9:41 AM Francisco Guerrero wrote: > +1 (nb) > > On 2022/01/19 15:10:20 Brandon Williams wrote: > > We can for completeness, but even with twice as much usage reported as > the > > other methods, I don't think it will affect the outcome of the vote. > > > > On Wed, Jan 19, 2022, 7:25 AM Paulo Motta > wrote: > > > > > This proposal looks good to me, +1. I was wondering if we should not > run > > > this proposal on the user@ list to check if there's any additional > > > feedback in addition to the informal Twitter and Linkedin channels? > > > > > > Em qua., 19 de jan. de 2022 às 10:18, Sylwester Lachiewicz < > > > slachiew...@gmail.com> escreveu: > > > > > >> +1 (Nb) > > >> > > >> śr., 19 sty 2022, 12:31 użytkownik Brandon Williams > > > >> napisał: > > >> > > >>> +1 > > >>> > > >>> On Tue, Jan 18, 2022 at 10:30 AM Ekaterina Dimitrova > > >>> wrote: > > >>> > > > >>> > Hi everyone, > > >>> > > > >>> > With the work to add Java 17 support for Cassandra, a new question > > >>> around the future of UDF was raised. The scripted UDF was using > Nashorn > > >>> which is no longer packaged with the JDK. There are options to add > new > > >>> dependencies to Graal JS for example but it seems people are not > sure that > > >>> it is worth it. Please check the discussion on CASSANDRA-16895. > > >>> > > > >>> > The following suggestion was made by Marcus and supported by other > PMC > > >>> members - "I think we should deprecate scripted UDFs now and drop > them from > > >>> the next major, but possibly provide hooks for people to write their > own > > >>> UDF "engines" and break out the current javascript implementation in > to its > > >>> own repository (but not ship it with Cassandra)." > > >>> > > > >>> > As a result we decided to engage with our users and created a > Twitter > > >>> survey. Results below: > > >>> > > > >>> > We would love to understand how you use ApacheCassandra UDFs and > UDAs. > > >>> > > > >>> > 32 people responded as follows: > > >>> > > > >>> > We do not use them - 75% > > >>> > We only use Java UDFs - 22% > > >>> > We only use JS UDFs - 0% > > >>> > We use Java and JS UDFs - 3% > > >>> > > > >>> > We also received feedback on LinkedIN on the topic - > > >>> > https://www.linkedin.com/feed/update/urn:li:activity:6886728406742970369?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6886793921020608512%29=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6887421509485248512%29 > > >>> > > > >>> > > > >>> > Thoughts? > > >>> > > > >>> > Best regards, > > >>> > Ekaterina > > >>> > > >> > > >
Re: UDF future
+1 (nb) On 2022/01/19 15:10:20 Brandon Williams wrote: > We can for completeness, but even with twice as much usage reported as the > other methods, I don't think it will affect the outcome of the vote. > > On Wed, Jan 19, 2022, 7:25 AM Paulo Motta wrote: > > > This proposal looks good to me, +1. I was wondering if we should not run > > this proposal on the user@ list to check if there's any additional > > feedback in addition to the informal Twitter and Linkedin channels? > > > > Em qua., 19 de jan. de 2022 às 10:18, Sylwester Lachiewicz < > > slachiew...@gmail.com> escreveu: > > > >> +1 (Nb) > >> > >> śr., 19 sty 2022, 12:31 użytkownik Brandon Williams > >> napisał: > >> > >>> +1 > >>> > >>> On Tue, Jan 18, 2022 at 10:30 AM Ekaterina Dimitrova > >>> wrote: > >>> > > >>> > Hi everyone, > >>> > > >>> > With the work to add Java 17 support for Cassandra, a new question > >>> around the future of UDF was raised. The scripted UDF was using Nashorn > >>> which is no longer packaged with the JDK. There are options to add new > >>> dependencies to Graal JS for example but it seems people are not sure that > >>> it is worth it. Please check the discussion on CASSANDRA-16895. > >>> > > >>> > The following suggestion was made by Marcus and supported by other PMC > >>> members - "I think we should deprecate scripted UDFs now and drop them > >>> from > >>> the next major, but possibly provide hooks for people to write their own > >>> UDF "engines" and break out the current javascript implementation in to > >>> its > >>> own repository (but not ship it with Cassandra)." > >>> > > >>> > As a result we decided to engage with our users and created a Twitter > >>> survey. Results below: > >>> > > >>> > We would love to understand how you use ApacheCassandra UDFs and UDAs. > >>> > > >>> > 32 people responded as follows: > >>> > > >>> > We do not use them - 75% > >>> > We only use Java UDFs - 22% > >>> > We only use JS UDFs - 0% > >>> > We use Java and JS UDFs - 3% > >>> > > >>> > We also received feedback on LinkedIN on the topic - > >>> https://www.linkedin.com/feed/update/urn:li:activity:6886728406742970369?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6886793921020608512%29=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6887421509485248512%29 > >>> > > >>> > > >>> > Thoughts? > >>> > > >>> > Best regards, > >>> > Ekaterina > >>> > >> >
Re: UDF future
We can for completeness, but even with twice as much usage reported as the other methods, I don't think it will affect the outcome of the vote. On Wed, Jan 19, 2022, 7:25 AM Paulo Motta wrote: > This proposal looks good to me, +1. I was wondering if we should not run > this proposal on the user@ list to check if there's any additional > feedback in addition to the informal Twitter and Linkedin channels? > > Em qua., 19 de jan. de 2022 às 10:18, Sylwester Lachiewicz < > slachiew...@gmail.com> escreveu: > >> +1 (Nb) >> >> śr., 19 sty 2022, 12:31 użytkownik Brandon Williams >> napisał: >> >>> +1 >>> >>> On Tue, Jan 18, 2022 at 10:30 AM Ekaterina Dimitrova >>> wrote: >>> > >>> > Hi everyone, >>> > >>> > With the work to add Java 17 support for Cassandra, a new question >>> around the future of UDF was raised. The scripted UDF was using Nashorn >>> which is no longer packaged with the JDK. There are options to add new >>> dependencies to Graal JS for example but it seems people are not sure that >>> it is worth it. Please check the discussion on CASSANDRA-16895. >>> > >>> > The following suggestion was made by Marcus and supported by other PMC >>> members - "I think we should deprecate scripted UDFs now and drop them from >>> the next major, but possibly provide hooks for people to write their own >>> UDF "engines" and break out the current javascript implementation in to its >>> own repository (but not ship it with Cassandra)." >>> > >>> > As a result we decided to engage with our users and created a Twitter >>> survey. Results below: >>> > >>> > We would love to understand how you use ApacheCassandra UDFs and UDAs. >>> > >>> > 32 people responded as follows: >>> > >>> > We do not use them - 75% >>> > We only use Java UDFs - 22% >>> > We only use JS UDFs - 0% >>> > We use Java and JS UDFs - 3% >>> > >>> > We also received feedback on LinkedIN on the topic - >>> https://www.linkedin.com/feed/update/urn:li:activity:6886728406742970369?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6886793921020608512%29=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6887421509485248512%29 >>> > >>> > >>> > Thoughts? >>> > >>> > Best regards, >>> > Ekaterina >>> >>
Re: UDF future
This proposal looks good to me, +1. I was wondering if we should not run this proposal on the user@ list to check if there's any additional feedback in addition to the informal Twitter and Linkedin channels? Em qua., 19 de jan. de 2022 às 10:18, Sylwester Lachiewicz < slachiew...@gmail.com> escreveu: > +1 (Nb) > > śr., 19 sty 2022, 12:31 użytkownik Brandon Williams > napisał: > >> +1 >> >> On Tue, Jan 18, 2022 at 10:30 AM Ekaterina Dimitrova >> wrote: >> > >> > Hi everyone, >> > >> > With the work to add Java 17 support for Cassandra, a new question >> around the future of UDF was raised. The scripted UDF was using Nashorn >> which is no longer packaged with the JDK. There are options to add new >> dependencies to Graal JS for example but it seems people are not sure that >> it is worth it. Please check the discussion on CASSANDRA-16895. >> > >> > The following suggestion was made by Marcus and supported by other PMC >> members - "I think we should deprecate scripted UDFs now and drop them from >> the next major, but possibly provide hooks for people to write their own >> UDF "engines" and break out the current javascript implementation in to its >> own repository (but not ship it with Cassandra)." >> > >> > As a result we decided to engage with our users and created a Twitter >> survey. Results below: >> > >> > We would love to understand how you use ApacheCassandra UDFs and UDAs. >> > >> > 32 people responded as follows: >> > >> > We do not use them - 75% >> > We only use Java UDFs - 22% >> > We only use JS UDFs - 0% >> > We use Java and JS UDFs - 3% >> > >> > We also received feedback on LinkedIN on the topic - >> https://www.linkedin.com/feed/update/urn:li:activity:6886728406742970369?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6886793921020608512%29=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6887421509485248512%29 >> > >> > >> > Thoughts? >> > >> > Best regards, >> > Ekaterina >> >
Re: UDF future
+1 (Nb) śr., 19 sty 2022, 12:31 użytkownik Brandon Williams napisał: > +1 > > On Tue, Jan 18, 2022 at 10:30 AM Ekaterina Dimitrova > wrote: > > > > Hi everyone, > > > > With the work to add Java 17 support for Cassandra, a new question > around the future of UDF was raised. The scripted UDF was using Nashorn > which is no longer packaged with the JDK. There are options to add new > dependencies to Graal JS for example but it seems people are not sure that > it is worth it. Please check the discussion on CASSANDRA-16895. > > > > The following suggestion was made by Marcus and supported by other PMC > members - "I think we should deprecate scripted UDFs now and drop them from > the next major, but possibly provide hooks for people to write their own > UDF "engines" and break out the current javascript implementation in to its > own repository (but not ship it with Cassandra)." > > > > As a result we decided to engage with our users and created a Twitter > survey. Results below: > > > > We would love to understand how you use ApacheCassandra UDFs and UDAs. > > > > 32 people responded as follows: > > > > We do not use them - 75% > > We only use Java UDFs - 22% > > We only use JS UDFs - 0% > > We use Java and JS UDFs - 3% > > > > We also received feedback on LinkedIN on the topic - > https://www.linkedin.com/feed/update/urn:li:activity:6886728406742970369?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6886793921020608512%29=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6887421509485248512%29 > > > > > > Thoughts? > > > > Best regards, > > Ekaterina >
Re: UDF future
+1 On Tue, Jan 18, 2022 at 10:30 AM Ekaterina Dimitrova wrote: > > Hi everyone, > > With the work to add Java 17 support for Cassandra, a new question around the > future of UDF was raised. The scripted UDF was using Nashorn which is no > longer packaged with the JDK. There are options to add new dependencies to > Graal JS for example but it seems people are not sure that it is worth it. > Please check the discussion on CASSANDRA-16895. > > The following suggestion was made by Marcus and supported by other PMC > members - "I think we should deprecate scripted UDFs now and drop them from > the next major, but possibly provide hooks for people to write their own UDF > "engines" and break out the current javascript implementation in to its own > repository (but not ship it with Cassandra)." > > As a result we decided to engage with our users and created a Twitter survey. > Results below: > > We would love to understand how you use ApacheCassandra UDFs and UDAs. > > 32 people responded as follows: > > We do not use them - 75% > We only use Java UDFs - 22% > We only use JS UDFs - 0% > We use Java and JS UDFs - 3% > > We also received feedback on LinkedIN on the topic - > https://www.linkedin.com/feed/update/urn:li:activity:6886728406742970369?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6886793921020608512%29=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6887421509485248512%29 > > > Thoughts? > > Best regards, > Ekaterina
Re: UDF future
+1 > On Jan 18, 2022, at 8:38 AM, Jonathan Ellis wrote: > > > +1 > >> On Tue, Jan 18, 2022 at 10:34 AM C. Scott Andreas >> wrote: >> I also (+1nb) support a proposal to deprecate JavaScript UDFs; to offer an >> interface for those who would like to supply a UDF implementation; and to >> extract/remove our current implementation. >> >> JDK17 support seems like a much higher priority than in-tree JS UDFs. >> >> — Scott >> >>>> On Jan 18, 2022, at 8:30 AM, Ekaterina Dimitrova >>>> wrote: >>>> >>> Hi everyone, >>> >>> With the work to add Java 17 support for Cassandra, a new question around >>> the future of UDF was raised. The scripted UDF was using Nashorn which is >>> no longer packaged with the JDK. There are options to add new dependencies >>> to Graal JS for example but it seems people are not sure that it is worth >>> it. Please check the discussion on CASSANDRA-16895. >>> >>> The following suggestion was made by Marcus and supported by other PMC >>> members - "I think we should deprecate scripted UDFs now and drop them from >>> the next major, but possibly provide hooks for people to write their own >>> UDF "engines" and break out the current javascript implementation in to its >>> own repository (but not ship it with Cassandra)." >>> >>> As a result we decided to engage with our users and created a Twitter >>> survey. Results below: >>> We would love to understand how you use ApacheCassandra UDFs and UDAs. >>> 32 people responded as follows: >>> We do not use them - 75% >>> We only use Java UDFs - 22% >>> We only use JS UDFs - 0% >>> We use Java and JS UDFs - 3% >>> We also received feedback on LinkedIN on the topic - >>> https://www.linkedin.com/feed/update/urn:li:activity:6886728406742970369?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6886793921020608512%29=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6887421509485248512%29 >>> >>> Thoughts? >>> >>> Best regards, >>> Ekaterina > > > -- > Jonathan Ellis > co-founder, http://www.datastax.com > @spyced
Re: UDF future
+1 On Tue, Jan 18, 2022 at 10:34 AM C. Scott Andreas wrote: > I also (+1nb) support a proposal to deprecate JavaScript UDFs; to offer an > interface for those who would like to supply a UDF implementation; and to > extract/remove our current implementation. > > JDK17 support seems like a much higher priority than in-tree JS UDFs. > > — Scott > > On Jan 18, 2022, at 8:30 AM, Ekaterina Dimitrova > wrote: > > Hi everyone, > > With the work to add Java 17 support for Cassandra, a new question around > the future of UDF was raised. The scripted UDF was using Nashorn which is > no longer packaged with the JDK. There are options to add new dependencies > to Graal JS for example but it seems people are not sure that it is worth > it. Please check the discussion on CASSANDRA-16895. > > The following suggestion was made by Marcus and supported by other PMC > members - "I think we should deprecate scripted UDFs now and drop them > from the next major, but possibly provide hooks for people to write their > own UDF "engines" and break out the current javascript implementation in to > its own repository (but not ship it with Cassandra)." > > As a result we decided to engage with our users and created a Twitter > survey. Results below: > > *We would love to understand how you use ApacheCassandra UDFs and UDAs.* > > *32 people responded as follows:* > >- *We do not use them - 75%* >- *We only use Java UDFs - 22%* >- *We only use JS UDFs - 0%* >- *We use Java and JS UDFs - 3%* > > We also received feedback on LinkedIN on the topic - > https://www.linkedin.com/feed/update/urn:li:activity:6886728406742970369?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6886793921020608512%29=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6887421509485248512%29 > > Thoughts? > > Best regards, > Ekaterina > > -- Jonathan Ellis co-founder, http://www.datastax.com @spyced
Re: UDF future
I also (+1nb) support a proposal to deprecate JavaScript UDFs; to offer an interface for those who would like to supply a UDF implementation; and to extract/remove our current implementation. JDK17 support seems like a much higher priority than in-tree JS UDFs. — Scott > On Jan 18, 2022, at 8:30 AM, Ekaterina Dimitrova > wrote: > > Hi everyone, > > With the work to add Java 17 support for Cassandra, a new question around the > future of UDF was raised. The scripted UDF was using Nashorn which is no > longer packaged with the JDK. There are options to add new dependencies to > Graal JS for example but it seems people are not sure that it is worth it. > Please check the discussion on CASSANDRA-16895. > > The following suggestion was made by Marcus and supported by other PMC > members - "I think we should deprecate scripted UDFs now and drop them from > the next major, but possibly provide hooks for people to write their own UDF > "engines" and break out the current javascript implementation in to its own > repository (but not ship it with Cassandra)." > > As a result we decided to engage with our users and created a Twitter survey. > Results below: > We would love to understand how you use ApacheCassandra UDFs and UDAs. > 32 people responded as follows: > We do not use them - 75% > We only use Java UDFs - 22% > We only use JS UDFs - 0% > We use Java and JS UDFs - 3% > We also received feedback on LinkedIN on the topic - > https://www.linkedin.com/feed/update/urn:li:activity:6886728406742970369?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6886793921020608512%29=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6887421509485248512%29 > > Thoughts? > > Best regards, > Ekaterina
UDF future
Hi everyone, With the work to add Java 17 support for Cassandra, a new question around the future of UDF was raised. The scripted UDF was using Nashorn which is no longer packaged with the JDK. There are options to add new dependencies to Graal JS for example but it seems people are not sure that it is worth it. Please check the discussion on CASSANDRA-16895. The following suggestion was made by Marcus and supported by other PMC members - "I think we should deprecate scripted UDFs now and drop them from the next major, but possibly provide hooks for people to write their own UDF "engines" and break out the current javascript implementation in to its own repository (but not ship it with Cassandra)." As a result we decided to engage with our users and created a Twitter survey. Results below: *We would love to understand how you use ApacheCassandra UDFs and UDAs.* *32 people responded as follows:* - *We do not use them - 75%* - *We only use Java UDFs - 22%* - *We only use JS UDFs - 0%* - *We use Java and JS UDFs - 3%* We also received feedback on LinkedIN on the topic - https://www.linkedin.com/feed/update/urn:li:activity:6886728406742970369?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6886793921020608512%29=urn%3Ali%3Acomment%3A%28activity%3A6886728406742970369%2C6887421509485248512%29 Thoughts? Best regards, Ekaterina
Re: UDF
I'm +1 with this solution going in 4.0. That said, this make we realize that through this dependency we've ended up exposing (publicly) a bit too much to UDF. Namely, all we really need/want to expose for UDF is the "value" classes (UDTValue, TupleValue, Duration and LocalDate) and the types (DataType and his 2 subclasses UserType and TupleType). But we end up exposing quite a bit more: some abstract classes that are public (AbstractGettableData, AbstractSettableData, ...), but more problematic imo, the CodecRegistry class (whose only need to be public is, as far as I can tell, the UserType/TupeType.of() methods, which aren't useful to UDF at all) which 1) is overly complex for what we care about in UDF and 2) brings transitive dependencies (TypeCodec and all his sub-classes, TypeToken). So my suggestion is that we go with Robert's patch, but on top of that we add detection for usage of all those classes/methods that have no business being exposed. For brand new UDFs, we could just reject them if they use any of those in 4.0. For existing UDFs, we'd log a warning at startup that say it's deprecated and that the UDF should be replaced or things will break in the next major. Honestly, my expectation is that no-one uses those classes/methods (since again, they aren't exactly useful) so I don't anticipate doing so being a big deal. But unless we do it, we won't be able to clean up the code added by this. And there is quite a bit that would be nice to cleanup at some point here imo. Not that the driver code is bad in any way; just that 1) some of the complexity it exposes is overkill for UDF (registering TypeCodec typically), and that a lot of the code added duplicates things we already have server side). -- Sylvain On Wed, Sep 12, 2018 at 11:46 AM Aleksey Yeschenko wrote: > It’s my understanding that the duplicated code is being fully donated - so > it’ll have all the usual ASF license/copyright headers when it lands in > trunk. No special steps to take here. > > — > AY > > On 11 September 2018 at 19:43:58, Jeremiah D Jordan ( > jeremiah.jor...@gmail.com) wrote: > > Be careful when pulling in source files from the DataStax Java Driver (or > anywhere) to make sure and respect its Apache License, Version 2.0 and keep > all Copyright's etc with said files. > > -Jeremiah > > > On Sep 11, 2018, at 12:29 PM, Jeff Jirsa wrote: > > > > +1 as well. > > > > On Tue, Sep 11, 2018 at 10:27 AM Aleksey Yeschenko > > > wrote: > > > >> If this is about inclusion in 4.0, then I support it. > >> > >> Technically this is *mostly* just a move+donation of some code from > >> java-driver to Cassandra. Given how important this seemingly is to the > >> board and PMC for us to not have the dependency on the driver, the > sooner > >> it’s gone, the better. > >> > >> I’d be +1 for committing to trunk. > >> > >> — > >> AY > >> > >> On 11 September 2018 at 14:43:29, Robert Stupp (sn...@snazy.de) > wrote: > >> > >> The patch is technically complete - i.e. it works and does its thing. > >> > >> It's not strictly a bug fix but targets trunk. That's why I started > the > >> discussion. > >> > >> > >> On 09/11/2018 02:53 PM, Jason Brown wrote: > >>> Hi Robert, > >>> > >>> Thanks for taking on this work. Is this message a heads up that a > patch > >> is > >>> coming/complete, or to spawn a discussion about including this in > 4.0? > >>> > >>> Thanks, > >>> > >>> -Jason > >>> > >>> On Tue, Sep 11, 2018 at 2:32 AM, Robert Stupp > wrote: > >>> > >>>> In an effort to clean up our hygiene and limit the dependencies used > >> by > >>>> UDFs/UDAs, I think we should refactor the UDF code parts and remove > >> the > >>>> dependency to the Java Driver in that area without breaking existing > >>>> UDFs/UDAs. > >>>> > >>>> A working prototype is in this branch: > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_snazy_=DwIFaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=Gesm79MRSHznQEKqQabvh3Ie1L3xzqlPsfLfEfadHTM=6lUpmmETCKbmt_zcp_DCLIxCGPjVyf7zdX0UjBVOZX4= > > >>>> cassandra/tree/feature/remove-udf-driver-dep-trunk < > >>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_snazy_cassandra_tree_feature_remove-2D=DwIFaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=Gesm79MRSHznQEKqQabvh3Ie1L3xzqlPsfLfEfadHTM=fBx64l59d8Y9Q7m9j0nNH9VvcaHc3QfoCAx4st5UJDM= > > >>>> udf-d
Re: UDF
It’s my understanding that the duplicated code is being fully donated - so it’ll have all the usual ASF license/copyright headers when it lands in trunk. No special steps to take here. — AY On 11 September 2018 at 19:43:58, Jeremiah D Jordan (jeremiah.jor...@gmail.com) wrote: Be careful when pulling in source files from the DataStax Java Driver (or anywhere) to make sure and respect its Apache License, Version 2.0 and keep all Copyright's etc with said files. -Jeremiah > On Sep 11, 2018, at 12:29 PM, Jeff Jirsa wrote: > > +1 as well. > > On Tue, Sep 11, 2018 at 10:27 AM Aleksey Yeschenko > wrote: > >> If this is about inclusion in 4.0, then I support it. >> >> Technically this is *mostly* just a move+donation of some code from >> java-driver to Cassandra. Given how important this seemingly is to the >> board and PMC for us to not have the dependency on the driver, the sooner >> it’s gone, the better. >> >> I’d be +1 for committing to trunk. >> >> — >> AY >> >> On 11 September 2018 at 14:43:29, Robert Stupp (sn...@snazy.de) wrote: >> >> The patch is technically complete - i.e. it works and does its thing. >> >> It's not strictly a bug fix but targets trunk. That's why I started the >> discussion. >> >> >> On 09/11/2018 02:53 PM, Jason Brown wrote: >>> Hi Robert, >>> >>> Thanks for taking on this work. Is this message a heads up that a patch >> is >>> coming/complete, or to spawn a discussion about including this in 4.0? >>> >>> Thanks, >>> >>> -Jason >>> >>> On Tue, Sep 11, 2018 at 2:32 AM, Robert Stupp wrote: >>> >>>> In an effort to clean up our hygiene and limit the dependencies used >> by >>>> UDFs/UDAs, I think we should refactor the UDF code parts and remove >> the >>>> dependency to the Java Driver in that area without breaking existing >>>> UDFs/UDAs. >>>> >>>> A working prototype is in this branch: >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_snazy_=DwIFaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=Gesm79MRSHznQEKqQabvh3Ie1L3xzqlPsfLfEfadHTM=6lUpmmETCKbmt_zcp_DCLIxCGPjVyf7zdX0UjBVOZX4= >>>> >>>> cassandra/tree/feature/remove-udf-driver-dep-trunk < >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_snazy_cassandra_tree_feature_remove-2D=DwIFaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=Gesm79MRSHznQEKqQabvh3Ie1L3xzqlPsfLfEfadHTM=fBx64l59d8Y9Q7m9j0nNH9VvcaHc3QfoCAx4st5UJDM= >>>> >>>> udf-driver-dep-trunk> . The changes are rather trivial and provide >> 100% >>>> backwards compatibility for existing UDFs. >>>> >>>> The prototype copies the necessary parts from the Java Driver into the >> C* >>>> source tree to org.apache.cassandra.cql3.functions.types and adopts >> its >>>> usages - i.e. UDF/UDA code plus CQLSSTableWriter + >> StressCQLSSTableWriter. >>>> The latter two classes have a reference to UDF’s UDHelper and had to >> be >>>> changed as well. >>>> >>>> Some functionality, like type parsing & handling, is duplicated in the >>>> code base with this prototype - once in the “current” source tree and >> once >>>> for UDFs. However, unifying the code paths is not trivial, since the >> UDF >>>> sandbox prohibits the use of internal classes (direct and likely >> indirect >>>> dependencies). >>>> >>>> Robert >>>> >>>> — >>>> Robert Stupp >>>> @snazy >>>> >>>> >> >> -- >> Robert Stupp >> @snazy >> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: dev-h...@cassandra.apache.org >> >> - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: UDF
Be careful when pulling in source files from the DataStax Java Driver (or anywhere) to make sure and respect its Apache License, Version 2.0 and keep all Copyright's etc with said files. -Jeremiah > On Sep 11, 2018, at 12:29 PM, Jeff Jirsa wrote: > > +1 as well. > > On Tue, Sep 11, 2018 at 10:27 AM Aleksey Yeschenko > wrote: > >> If this is about inclusion in 4.0, then I support it. >> >> Technically this is *mostly* just a move+donation of some code from >> java-driver to Cassandra. Given how important this seemingly is to the >> board and PMC for us to not have the dependency on the driver, the sooner >> it’s gone, the better. >> >> I’d be +1 for committing to trunk. >> >> — >> AY >> >> On 11 September 2018 at 14:43:29, Robert Stupp (sn...@snazy.de) wrote: >> >> The patch is technically complete - i.e. it works and does its thing. >> >> It's not strictly a bug fix but targets trunk. That's why I started the >> discussion. >> >> >> On 09/11/2018 02:53 PM, Jason Brown wrote: >>> Hi Robert, >>> >>> Thanks for taking on this work. Is this message a heads up that a patch >> is >>> coming/complete, or to spawn a discussion about including this in 4.0? >>> >>> Thanks, >>> >>> -Jason >>> >>> On Tue, Sep 11, 2018 at 2:32 AM, Robert Stupp wrote: >>> >>>> In an effort to clean up our hygiene and limit the dependencies used >> by >>>> UDFs/UDAs, I think we should refactor the UDF code parts and remove >> the >>>> dependency to the Java Driver in that area without breaking existing >>>> UDFs/UDAs. >>>> >>>> A working prototype is in this branch: >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_snazy_=DwIFaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=Gesm79MRSHznQEKqQabvh3Ie1L3xzqlPsfLfEfadHTM=6lUpmmETCKbmt_zcp_DCLIxCGPjVyf7zdX0UjBVOZX4= >>>> cassandra/tree/feature/remove-udf-driver-dep-trunk < >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_snazy_cassandra_tree_feature_remove-2D=DwIFaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=Gesm79MRSHznQEKqQabvh3Ie1L3xzqlPsfLfEfadHTM=fBx64l59d8Y9Q7m9j0nNH9VvcaHc3QfoCAx4st5UJDM= >>>> udf-driver-dep-trunk> . The changes are rather trivial and provide >> 100% >>>> backwards compatibility for existing UDFs. >>>> >>>> The prototype copies the necessary parts from the Java Driver into the >> C* >>>> source tree to org.apache.cassandra.cql3.functions.types and adopts >> its >>>> usages - i.e. UDF/UDA code plus CQLSSTableWriter + >> StressCQLSSTableWriter. >>>> The latter two classes have a reference to UDF’s UDHelper and had to >> be >>>> changed as well. >>>> >>>> Some functionality, like type parsing & handling, is duplicated in the >>>> code base with this prototype - once in the “current” source tree and >> once >>>> for UDFs. However, unifying the code paths is not trivial, since the >> UDF >>>> sandbox prohibits the use of internal classes (direct and likely >> indirect >>>> dependencies). >>>> >>>> Robert >>>> >>>> — >>>> Robert Stupp >>>> @snazy >>>> >>>> >> >> -- >> Robert Stupp >> @snazy >> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: dev-h...@cassandra.apache.org >> >> - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: UDF
+1 as well. On Tue, Sep 11, 2018 at 10:27 AM Aleksey Yeschenko wrote: > If this is about inclusion in 4.0, then I support it. > > Technically this is *mostly* just a move+donation of some code from > java-driver to Cassandra. Given how important this seemingly is to the > board and PMC for us to not have the dependency on the driver, the sooner > it’s gone, the better. > > I’d be +1 for committing to trunk. > > — > AY > > On 11 September 2018 at 14:43:29, Robert Stupp (sn...@snazy.de) wrote: > > The patch is technically complete - i.e. it works and does its thing. > > It's not strictly a bug fix but targets trunk. That's why I started the > discussion. > > > On 09/11/2018 02:53 PM, Jason Brown wrote: > > Hi Robert, > > > > Thanks for taking on this work. Is this message a heads up that a patch > is > > coming/complete, or to spawn a discussion about including this in 4.0? > > > > Thanks, > > > > -Jason > > > > On Tue, Sep 11, 2018 at 2:32 AM, Robert Stupp wrote: > > > >> In an effort to clean up our hygiene and limit the dependencies used > by > >> UDFs/UDAs, I think we should refactor the UDF code parts and remove > the > >> dependency to the Java Driver in that area without breaking existing > >> UDFs/UDAs. > >> > >> A working prototype is in this branch: https://github.com/snazy/ > >> cassandra/tree/feature/remove-udf-driver-dep-trunk < > >> https://github.com/snazy/cassandra/tree/feature/remove- > >> udf-driver-dep-trunk> . The changes are rather trivial and provide > 100% > >> backwards compatibility for existing UDFs. > >> > >> The prototype copies the necessary parts from the Java Driver into the > C* > >> source tree to org.apache.cassandra.cql3.functions.types and adopts > its > >> usages - i.e. UDF/UDA code plus CQLSSTableWriter + > StressCQLSSTableWriter. > >> The latter two classes have a reference to UDF’s UDHelper and had to > be > >> changed as well. > >> > >> Some functionality, like type parsing & handling, is duplicated in the > >> code base with this prototype - once in the “current” source tree and > once > >> for UDFs. However, unifying the code paths is not trivial, since the > UDF > >> sandbox prohibits the use of internal classes (direct and likely > indirect > >> dependencies). > >> > >> Robert > >> > >> — > >> Robert Stupp > >> @snazy > >> > >> > > -- > Robert Stupp > @snazy > > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
Re: UDF
If this is about inclusion in 4.0, then I support it. Technically this is *mostly* just a move+donation of some code from java-driver to Cassandra. Given how important this seemingly is to the board and PMC for us to not have the dependency on the driver, the sooner it’s gone, the better. I’d be +1 for committing to trunk. — AY On 11 September 2018 at 14:43:29, Robert Stupp (sn...@snazy.de) wrote: The patch is technically complete - i.e. it works and does its thing. It's not strictly a bug fix but targets trunk. That's why I started the discussion. On 09/11/2018 02:53 PM, Jason Brown wrote: > Hi Robert, > > Thanks for taking on this work. Is this message a heads up that a patch is > coming/complete, or to spawn a discussion about including this in 4.0? > > Thanks, > > -Jason > > On Tue, Sep 11, 2018 at 2:32 AM, Robert Stupp wrote: > >> In an effort to clean up our hygiene and limit the dependencies used by >> UDFs/UDAs, I think we should refactor the UDF code parts and remove the >> dependency to the Java Driver in that area without breaking existing >> UDFs/UDAs. >> >> A working prototype is in this branch: https://github.com/snazy/ >> cassandra/tree/feature/remove-udf-driver-dep-trunk < >> https://github.com/snazy/cassandra/tree/feature/remove- >> udf-driver-dep-trunk> . The changes are rather trivial and provide 100% >> backwards compatibility for existing UDFs. >> >> The prototype copies the necessary parts from the Java Driver into the C* >> source tree to org.apache.cassandra.cql3.functions.types and adopts its >> usages - i.e. UDF/UDA code plus CQLSSTableWriter + StressCQLSSTableWriter. >> The latter two classes have a reference to UDF’s UDHelper and had to be >> changed as well. >> >> Some functionality, like type parsing & handling, is duplicated in the >> code base with this prototype - once in the “current” source tree and once >> for UDFs. However, unifying the code paths is not trivial, since the UDF >> sandbox prohibits the use of internal classes (direct and likely indirect >> dependencies). >> >> Robert >> >> — >> Robert Stupp >> @snazy >> >> -- Robert Stupp @snazy - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: UDF
The patch is technically complete - i.e. it works and does its thing. It's not strictly a bug fix but targets trunk. That's why I started the discussion. On 09/11/2018 02:53 PM, Jason Brown wrote: Hi Robert, Thanks for taking on this work. Is this message a heads up that a patch is coming/complete, or to spawn a discussion about including this in 4.0? Thanks, -Jason On Tue, Sep 11, 2018 at 2:32 AM, Robert Stupp wrote: In an effort to clean up our hygiene and limit the dependencies used by UDFs/UDAs, I think we should refactor the UDF code parts and remove the dependency to the Java Driver in that area without breaking existing UDFs/UDAs. A working prototype is in this branch: https://github.com/snazy/ cassandra/tree/feature/remove-udf-driver-dep-trunk < https://github.com/snazy/cassandra/tree/feature/remove- udf-driver-dep-trunk> . The changes are rather trivial and provide 100% backwards compatibility for existing UDFs. The prototype copies the necessary parts from the Java Driver into the C* source tree to org.apache.cassandra.cql3.functions.types and adopts its usages - i.e. UDF/UDA code plus CQLSSTableWriter + StressCQLSSTableWriter. The latter two classes have a reference to UDF’s UDHelper and had to be changed as well. Some functionality, like type parsing & handling, is duplicated in the code base with this prototype - once in the “current” source tree and once for UDFs. However, unifying the code paths is not trivial, since the UDF sandbox prohibits the use of internal classes (direct and likely indirect dependencies). Robert — Robert Stupp @snazy -- Robert Stupp @snazy - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: UDF
Hi Robert, Thanks for taking on this work. Is this message a heads up that a patch is coming/complete, or to spawn a discussion about including this in 4.0? Thanks, -Jason On Tue, Sep 11, 2018 at 2:32 AM, Robert Stupp wrote: > In an effort to clean up our hygiene and limit the dependencies used by > UDFs/UDAs, I think we should refactor the UDF code parts and remove the > dependency to the Java Driver in that area without breaking existing > UDFs/UDAs. > > A working prototype is in this branch: https://github.com/snazy/ > cassandra/tree/feature/remove-udf-driver-dep-trunk < > https://github.com/snazy/cassandra/tree/feature/remove- > udf-driver-dep-trunk> . The changes are rather trivial and provide 100% > backwards compatibility for existing UDFs. > > The prototype copies the necessary parts from the Java Driver into the C* > source tree to org.apache.cassandra.cql3.functions.types and adopts its > usages - i.e. UDF/UDA code plus CQLSSTableWriter + StressCQLSSTableWriter. > The latter two classes have a reference to UDF’s UDHelper and had to be > changed as well. > > Some functionality, like type parsing & handling, is duplicated in the > code base with this prototype - once in the “current” source tree and once > for UDFs. However, unifying the code paths is not trivial, since the UDF > sandbox prohibits the use of internal classes (direct and likely indirect > dependencies). > > Robert > > — > Robert Stupp > @snazy > >
UDF
In an effort to clean up our hygiene and limit the dependencies used by UDFs/UDAs, I think we should refactor the UDF code parts and remove the dependency to the Java Driver in that area without breaking existing UDFs/UDAs. A working prototype is in this branch: https://github.com/snazy/cassandra/tree/feature/remove-udf-driver-dep-trunk <https://github.com/snazy/cassandra/tree/feature/remove-udf-driver-dep-trunk> . The changes are rather trivial and provide 100% backwards compatibility for existing UDFs. The prototype copies the necessary parts from the Java Driver into the C* source tree to org.apache.cassandra.cql3.functions.types and adopts its usages - i.e. UDF/UDA code plus CQLSSTableWriter + StressCQLSSTableWriter. The latter two classes have a reference to UDF’s UDHelper and had to be changed as well. Some functionality, like type parsing & handling, is duplicated in the code base with this prototype - once in the “current” source tree and once for UDFs. However, unifying the code paths is not trivial, since the UDF sandbox prohibits the use of internal classes (direct and likely indirect dependencies). Robert — Robert Stupp @snazy
Performance improvement of Cassandra UDF/UDA
Hi All, I am new to the Cassandra community and thank you in advance for your kindly comments on an issue we met recently. We have found that running query with direct UDF execution is ten time more faster than the async UDF execution. The in-line comment: "Using async UDF execution is expensive (adds about 100us overhead per invocation on a Core-i7 MBPr)” https://insight.io/github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/cql3/functions/UDFunction.java?line=293 show that this is a known behavior. My questions are as below: 1. What are the main pros and cons of these two methods? Can I find any documents that discuss this? 2. Are there any plans to improve the performance of using async UDF? A simple way come to my mind is to use some sort of batch method, e.g., replace current row by row method with some rows by some rows. Are there any concerns on this? 3. How people solve this performance issue in general? It seems this performance issue is not an urgent or an important issue to solve because it is known and it is still there. Therefore people must have some sort of good solution solving this issue. I really appreciate your comments in advance. Best regards, Xin
