On Mon, Aug 26, 2024 at 1:59 PM Rejeb Ben Rejeb <benrejebre...@gmail.com> wrote:
> Hi, > > REMOVE DATASOURCE V1: > After removing V1 code, it is possible to configure V1 name as a V2 long > name. > I have to test it to be sure but this can be done by moving the > class PhoenixDataSource under package "org.apache.phoenix.spark". > In this way, it will have no impact on old applications which use the spark > API. > Would creating a compatibility child of the driver under the old package name work ? I don't like the idea of moving the up-to-date code to a new package. > When I wrote my first message I forget that there is also helper methods > like "phoenixTableAsDataFrame" or "saveToPhoenix", for those we have two > options: > > 1. Assume that these methods are no longer maintained, document to use > spark API instead and remove them. > 2. Keep methods and change method implementation to point to the V2 > datasource (all options of V1 are available with V2). > > Personally, I prefer option 1 as for old scala or java applications they > need code and dependencies update to use a newest version of connector > anyway. Python or R applications will not be impacted as they use Spark > API. > While I agree with you from a technical POV, the reality is that there are a lot of legacy spark jobs that I'd prefer not to break. Option 2 sounds better to me. > BUILD ARTIFACTS WITH DIFFERENT SCALA VERSIONS: > Yes, since the connector for spark 2 was compiled with scala 2.11 it can't > be run with spark 2 compiled with scala 2.12. Same applies for the spark 3 > connector with scala 2.12 vs 2.13. > I meant to have this for later releases, IMHO, actually this is a > limitation and it will be good to have the connector built with both scala > versions so usage will not be restricted to only one version of the spark > build. > I've done some quick research, it seems that there is a way to manage this > with the scala-maven-plugin throw multiple executions instead of using > maven profiles. > > That sounds fine, please open a ticket, and a PR with your preferred solution. > Rejeb > > > Le lun. 26 août 2024 à 08:49, Istvan Toth <st...@apache.org> a écrit : > > > Hi, > > > > Forgive my ignorance of Spark: > > > > REMOVE DATASOURCE V1: > > > > IIRC the V1 and V2 datasources have different names. > > Wouldn't this break applications using the old V1 name ? > > Is there a chance that this would break old applications ? > > > > BUILD ARTIFACTS WITH DIFFERENT SCALA VERSIONS: > > > > Is this required because scala 2.x runtimes are not backwards compatible > ? > > I don't see a problem with that. > > > > Its utility is limited until we start providing actual releases and > publish > > binary artifacts, but > > in theory I agree. > > > > The implementation would be a bit tricky, the solution that comes to my > > mind is generating the artifacts > > in multiple maven runs with different profiles, like we do for the > > different HBase profiles now. > > > > Istvan > > > > On Fri, Aug 23, 2024 at 7:36 PM Rejeb Ben Rejeb <benrejebre...@gmail.com > > > > wrote: > > > > > Hi, > > > > > > I would like to start a discussion about two changes to the > > phoenix5-spark > > > and phoenix5-spark3. > > > > > > REMOVE DATASOURCE V1 > > > It is not longer necessarie to keep Datasource V1 classes, since all > > > features are implemented in new connector version classes. T > > > When fixing the issue PHOENIX-6783, I checked for impacts and done some > > > modifications to make removing the classes safe and without impacts. > > > > > > BUILD ARTIFACTS WITH DIFFERENT SCALA VERSIONS > > > phoenix5-spark2 connector uses spark-2.4.8 wich is available with scala > > > 2.11 and scala 2.12. > > > Same for phoenix5-spark3 uses spark-3.2.4 wich is available with scala > > 2.12 > > > and scala 2.13. > > > > > > It would be nice to have connector supporting both scala version like > > other > > > connectors for exemple mongoDB or cassandra. > > > > > > Thanks, > > > Rejeb > > > > > > > > -- > Cordialement, > Rejeb Ben Rejeb > -- *István Tóth* | Sr. Staff Software Engineer *Email*: st...@cloudera.com cloudera.com <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> ------------------------------ ------------------------------
