On Thu, Dec 12, 2019 at 8:43 PM Kenneth Knowles <[email protected]> wrote:
> On Thu, Dec 12, 2019 at 3:30 PM Vincent Marquez <[email protected]> > wrote: > >> Hello, as I've mentioned in previous emails, I've found the CassandraIO >> connector lacking some essential features for efficient batch processing in >> real world scenarios. We've developed a more fully featured connector and >> had good results with it. >> > > Fantastic! > > >> Could I perhaps write up a JIRA proposal for some minor changes to the >> current connector that might improve things? >> > > Yes! > > >> The main pain point is the absense of a 'readAll' method as I documented >> here: >> >> https://gist.github.com/vmarquez/204b8f44b1279fdbae97b40f8681bc25 >> >> If I could write up a ticket, I don't mind submitting a small PR on GH as >> well addressing this lack of functionality. Thanks for your time. >> > > This would be excellent. Since it seems you already have implemented and > tested the functionality, a simple Jira with a title and description would > be enough, and then open a PR linked to the Jira with a title like > "[BEAM-1234567] Improve performance of CassandraIO" > I should clarify a bit. What has already been done and tested is a custom connector that has a 'readAll' cassandraIO functionality, I did not modify the existing beam connector. However, I spent some time the last couple days looking over the details of the current CassandraIO connector to verify it would be doable for me to do add something similar and still maintain all the current functionality. To share some code between both the 'read' and 'readAll' styles of CassandraIO, I'd want to modify the current 'Source' based 'connector' to be a 'ParDo' based one, so there is a minor (in my opinon, relative to the project) refactor involved. I'm happy to explain in more detail in the JIRA. Thank you for writing to dev@ to share your experience and intentions. We > are happy to help you with the Jira and PR, and find the best reviewers, if > you will open them to get started. > > Kenn > Thank you! > >> *-Vincent* >> >
