bschoening commented on code in PR #3812:
URL: https://github.com/apache/cassandra/pull/3812#discussion_r1940377828
##########
pylib/cqlshlib/cqlhandling.py:
##########
@@ -151,6 +151,39 @@ def cql_split_statements(self, text):
in_batch = True
return output, in_batch or in_pg_string
+ def group_tokens(self, items):
+ """
+ Split an iterable into sublists, using 'endtoken' to mark the end of
each sublist.
+ Each sublist accumulates elements until an 'endtoken' is encountered.
If the sublist
+ consists only of a single 'endtoken', it is excluded. An empty list is
added to the
+ result after the last 'endtoken' for cases like autocompletion.
+
+ Parameters:
+ - items (iterable): An iterable of tokens, including 'endtoken'
elements.
+
+ Returns:
+ - list: A list of sublists, with each sublist containing tokens split
by 'endtoken'.
+ """
+ thisresult = []
+ output = []
+
+ for i in items:
+ thisresult.append(i)
+
+ # When an 'endtoken' is encountered, start a new sublist
+ if i[0] == 'endtoken':
+ # Skip adding sublist if it contains just one "endtoken"
+ if len(thisresult) > 1:
+ output.append(thisresult) # Add valid sublist
+
+ # Start a new sublist after an 'endtoken'
+ thisresult = []
Review Comment:
@sedulam don't want to cause extra work, I thought this could be elegant.
```
result=[]
for key, group in itertools.groupby(items, lambda t: t[0] == endtoken):
if not key:
result.append(''.join(group))
```
the above will split a paragraph (items) with periods into sentences. But
if this doesn't work well, let's just ditch the groupby idea.
endtoken='.'
items="the rain. in Spain. stays mainly in. the plain"
result: ['the rain', ' in Spain', ' stays mainly in', ' the plain']
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
