+1 On Tue, Jul 2, 2024 at 7:19 AM yangjie01 <yangji...@baidu.com.invalid> wrote:
> I have manually attempted to only modify the `assembly/pom.xml` and > examined the results of executing `dev/make-distribution.sh --tgz`. The > `spark-connect_2.13-4.0.0-SNAPSHOT.jar` is indeed included in the jars > directory. However, if rearranging the directories would result in a > clearer project structure, I believe that would also be a viable approach. > > > > *发件人**: *Hyukjin Kwon <gurwls...@apache.org> > *日期**: *2024年7月2日 星期二 12:00 > *收件人**: *yangjie01 <yangji...@baidu.com.invalid> > *抄送**: *dev <dev@spark.apache.org> > *主题**: *Re: [外部邮件] [DISCUSS] Move Spark Connect server to builtin package > (Client API layer stays external) > > > > My concern is that the `connector` directory is really for > external/optional packages (and they aren't included in assembly IIRC).. so > I am hesitant to just change the assembly. > The actual changes are not quite large but it moves the files around. > > > > On Tue, 2 Jul 2024 at 12:23, yangjie01 <yangji...@baidu.com.invalid> > wrote: > > I'm supportive of this initiative. However, if the purpose is just to > avoid the additional `--packages` option, it seems that making some > adjustments to the `assembly/pom.xml` could potentially meet our goal. Is > it really necessary to restructure the code directory? > > > > Jie Yang > > > > *发件人**: *Hyukjin Kwon <gurwls...@apache.org> > *日期**: *2024年7月2日 星期二 08:19 > *收件人**: *dev <dev@spark.apache.org> > *主题**: *[外部邮件] [DISCUSS] Move Spark Connect server to builtin package > (Client API layer stays external) > > > > Hi all, > > I would like to discuss moving Spark Connect server to builtin package. > Right now, users have to specify —packages when they run Spark Connect > server script, for example: > > ./sbin/start-connect-server.sh --jars `ls > connector/connect/server/target/**/spark-connect*SNAPSHOT.jar` > > or > > ./sbin/start-connect-server.sh --packages > org.apache.spark:spark-connect_2.12:3.5.1 > > which is a little bit odd that sbin scripts should provide jars to start. > > Moving it to builtin package is pretty straightforward because most of > jars are shaded, and the impact would be minimal, I have a prototype here > apache/spark/#47157 > <https://mailshield.baidu.com/check?q=%2fcy8z7%2fYZNJHeBln9msQEVrqe7rk5HGMbpa7h9lbvgUO6rNqy1kPDW2sYiQgCGHS>. > This also simplifies Python local running logic a lot. > > User facing API layer, Spark Connect Client, stays external but I would > like the internal/admin server layer, Spark Connect Server, implementation > to be built in Spark. > > Please let me know if you have thoughts on this! > > > >
