RE: Re: Spark assembly in Maven repo?
Thanks for the info! Xiaoyong From: Sean Owen [mailto:so...@cloudera.com] Sent: Monday, December 14, 2015 12:20 AM To: Xiaoyong Zhu Cc: user Subject: Re: Re: Spark assembly in Maven repo? Yes, though I think the Maven Central repository is more canonical. http://repo1.maven.org/maven2/org/apache/spark/spark-core_2.10/1.5.2/<https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2frepo1.maven.org%2fmaven2%2forg%2fapache%2fspark%2fspark-core_2.10%2f1.5.2%2f&data=01%7c01%7cxiaoyzhu%40064d.mgd.microsoft.com%7cbc95bb73fe654d2c16df08d3045f61e3%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=UmuArU%2bj9WRhKE0VpQkm63TC4ewQaq150Ne5YbwmiL4%3d> On Mon, Dec 14, 2015, 06:35 Xiaoyong Zhu mailto:xiaoy...@microsoft.com>> wrote: Thanks! do you mean something here (for example for 1.5.1 using scala 2.10)? https://repository.apache.org/content/repositories/releases/org/apache/spark/spark-core_2.10/1.5.1/<https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2frepository.apache.org%2fcontent%2frepositories%2freleases%2forg%2fapache%2fspark%2fspark-core_2.10%2f1.5.1%2f&data=01%7c01%7cxiaoyzhu%40064d.mgd.microsoft.com%7cbc95bb73fe654d2c16df08d3045f61e3%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=XznNXDLoqiO7aE48Vg7BhTcnVico4ibnalzlmM67nec%3d> Xiaoyong From: Sean Owen [mailto:so...@cloudera.com<mailto:so...@cloudera.com>] Sent: Saturday, December 12, 2015 12:45 AM To: Xiaoyong Zhu mailto:xiaoy...@microsoft.com>> Cc: user mailto:user@spark.apache.org>> Subject: Re: Re: Spark assembly in Maven repo? That's exactly what the various artifacts in the Maven repo are for. The API classes for core are in the core artifact and so on. You don't need an assembly. On Sat, Dec 12, 2015 at 12:32 AM, Xiaoyong Zhu mailto:xiaoy...@microsoft.com>> wrote: Yes, so our scenario is to treat the spark assembly as an “SDK” so users can develop Spark applications easily without downloading them. In this case which way do you guys think might be good? Xiaoyong From: fightf...@163.com<mailto:fightf...@163.com> [mailto:fightf...@163.com<mailto:fightf...@163.com>] Sent: Friday, December 11, 2015 12:08 AM To: Mark Hamstra mailto:m...@clearstorydata.com>> Cc: Xiaoyong Zhu mailto:xiaoy...@microsoft.com>>; Jeff Zhang mailto:zjf...@gmail.com>>; user mailto:user@spark.apache.org>>; Zhaomin Xu mailto:z...@microsoft.com>>; Joe Zhang (SDE) mailto:gui...@microsoft.com>> Subject: Re: Re: Spark assembly in Maven repo? Agree with you that assembly jar is not good to publish. However, what he really need is to fetch an updatable maven jar file. fightf...@163.com<mailto:fightf...@163.com> From: Mark Hamstra<mailto:m...@clearstorydata.com> Date: 2015-12-11 15:34 To: fightf...@163.com<mailto:fightf...@163.com> CC: Xiaoyong Zhu<mailto:xiaoy...@microsoft.com>; Jeff Zhang<mailto:zjf...@gmail.com>; user<mailto:user@spark.apache.org>; Zhaomin Xu<mailto:z...@microsoft.com>; Joe Zhang (SDE)<mailto:gui...@microsoft.com> Subject: Re: RE: Spark assembly in Maven repo? No, publishing a spark assembly jar is not fine. See the doc attached to https://issues.apache.org/jira/browse/SPARK-11157<https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fissues.apache.org%2fjira%2fbrowse%2fSPARK-11157&data=01%7c01%7cxiaoyzhu%40064d.mgd.microsoft.com%7c999bc04245724187198808d3020238b3%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=cUBvArmmYasxoOl9wrefYO4L7JiTF5qqqZDyY%2b%2bLP3Y%3d> and be aware that a likely goal of Spark 2.0 will be the elimination of assemblies. On Thu, Dec 10, 2015 at 11:19 PM, fightf...@163.com<mailto:fightf...@163.com> mailto:fightf...@163.com>> wrote: Using maven to download the assembly jar is fine. I would recommend to deploy this assembly jar to your local maven repo, i.e. nexus repo, Or more likey a snapshot repository fightf...@163.com<mailto:fightf...@163.com> From: Xiaoyong Zhu<mailto:xiaoy...@microsoft.com> Date: 2015-12-11 15:10 To: Jeff Zhang<mailto:zjf...@gmail.com> CC: user@spark.apache.org<mailto:user@spark.apache.org>; Zhaomin Xu<mailto:z...@microsoft.com>; Joe Zhang (SDE)<mailto:gui...@microsoft.com> Subject: RE: Spark assembly in Maven repo? Sorry – I didn’t make it clear. It’s actually not a “dependency” – it’s actually that we are building a certain plugin for IntelliJ where we want to distribute this jar. But since the jar is updated frequently we don't want to distribute it together with our plugin but we would like to download it via Maven. In this case what’s the recommended way? Xiaoyong From: Jeff Zhang [mailto:zjf...@gmail.com<mailto:zjf...@gmail.com>] Sent: Thursday, December 10, 2015 11:03 PM To: Xiaoyong Zhu mailto:xiaoy...@microsoft.com>> Cc: user@spark.apache.org&l
Re: Re: Spark assembly in Maven repo?
Yes, though I think the Maven Central repository is more canonical. http://repo1.maven.org/maven2/org/apache/spark/spark-core_2.10/1.5.2/ On Mon, Dec 14, 2015, 06:35 Xiaoyong Zhu wrote: > Thanks! do you mean something here (for example for 1.5.1 using scala > 2.10)? > > > https://repository.apache.org/content/repositories/releases/org/apache/spark/spark-core_2.10/1.5.1/ > > > > Xiaoyong > > > > *From:* Sean Owen [mailto:so...@cloudera.com] > *Sent:* Saturday, December 12, 2015 12:45 AM > *To:* Xiaoyong Zhu > *Cc:* user > > > *Subject:* Re: Re: Spark assembly in Maven repo? > > > > That's exactly what the various artifacts in the Maven repo are for. The > API classes for core are in the core artifact and so on. You don't need an > assembly. > > > > On Sat, Dec 12, 2015 at 12:32 AM, Xiaoyong Zhu > wrote: > > Yes, so our scenario is to treat the spark assembly as an “SDK” so users > can develop Spark applications easily without downloading them. In this > case which way do you guys think might be good? > > > > Xiaoyong > > > > *From:* fightf...@163.com [mailto:fightf...@163.com] > *Sent:* Friday, December 11, 2015 12:08 AM > *To:* Mark Hamstra > *Cc:* Xiaoyong Zhu ; Jeff Zhang ; > user ; Zhaomin Xu ; Joe Zhang > (SDE) > *Subject:* Re: Re: Spark assembly in Maven repo? > > > > Agree with you that assembly jar is not good to publish. However, what he > really need is to fetch > > an updatable maven jar file. > > > -- > > fightf...@163.com > > > > *From:* Mark Hamstra > > *Date:* 2015-12-11 15:34 > > *To:* fightf...@163.com > > *CC:* Xiaoyong Zhu ; Jeff Zhang ; > user ; Zhaomin Xu ; Joe Zhang > (SDE) > > *Subject:* Re: RE: Spark assembly in Maven repo? > > No, publishing a spark assembly jar is not fine. See the doc attached to > https://issues.apache.org/jira/browse/SPARK-11157 > <https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fissues.apache.org%2fjira%2fbrowse%2fSPARK-11157&data=01%7c01%7cxiaoyzhu%40064d.mgd.microsoft.com%7c999bc04245724187198808d3020238b3%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=cUBvArmmYasxoOl9wrefYO4L7JiTF5qqqZDyY%2b%2bLP3Y%3d> > and be aware that a likely goal of Spark 2.0 will be the elimination of > assemblies. > > > > On Thu, Dec 10, 2015 at 11:19 PM, fightf...@163.com > wrote: > > Using maven to download the assembly jar is fine. I would recommend to > deploy this > > assembly jar to your local maven repo, i.e. nexus repo, Or more likey a > snapshot repository > > > -- > > fightf...@163.com > > > > *From:* Xiaoyong Zhu > > *Date:* 2015-12-11 15:10 > > *To:* Jeff Zhang > > *CC:* user@spark.apache.org; Zhaomin Xu ; Joe Zhang > (SDE) > > *Subject:* RE: Spark assembly in Maven repo? > > Sorry – I didn’t make it clear. It’s actually not a “dependency” – it’s > actually that we are building a certain plugin for IntelliJ where we want > to distribute this jar. But since the jar is updated frequently we don't > want to distribute it together with our plugin but we would like to > download it via Maven. > > > > In this case what’s the recommended way? > > > > Xiaoyong > > > > *From:* Jeff Zhang [mailto:zjf...@gmail.com] > *Sent:* Thursday, December 10, 2015 11:03 PM > *To:* Xiaoyong Zhu > *Cc:* user@spark.apache.org > *Subject:* Re: Spark assembly in Maven repo? > > > > I don't think make the assembly jar as dependency a good practice. You may > meet jar hell issue in that case. > > > > On Fri, Dec 11, 2015 at 2:46 PM, Xiaoyong Zhu > wrote: > > Hi Experts, > > > > We have a project which has a dependency for the following jar > > > > spark-assembly--hadoop.jar > > for example: > > spark-assembly-1.4.1.2.3.3.0-2983-hadoop2.7.1.2.3.3.0-2983.jar > > > > since this assembly might be updated in the future, I am not sure if there > is a Maven repo that has the above spark assembly jar? Or should we create > & upload it to Maven central? > > > > Thanks! > > > > Xiaoyong > > > > > > > > -- > > Best Regards > > Jeff Zhang > > > > >
RE: Re: Spark assembly in Maven repo?
Thanks! do you mean something here (for example for 1.5.1 using scala 2.10)? https://repository.apache.org/content/repositories/releases/org/apache/spark/spark-core_2.10/1.5.1/ Xiaoyong From: Sean Owen [mailto:so...@cloudera.com] Sent: Saturday, December 12, 2015 12:45 AM To: Xiaoyong Zhu Cc: user Subject: Re: Re: Spark assembly in Maven repo? That's exactly what the various artifacts in the Maven repo are for. The API classes for core are in the core artifact and so on. You don't need an assembly. On Sat, Dec 12, 2015 at 12:32 AM, Xiaoyong Zhu mailto:xiaoy...@microsoft.com>> wrote: Yes, so our scenario is to treat the spark assembly as an “SDK” so users can develop Spark applications easily without downloading them. In this case which way do you guys think might be good? Xiaoyong From: fightf...@163.com<mailto:fightf...@163.com> [mailto:fightf...@163.com<mailto:fightf...@163.com>] Sent: Friday, December 11, 2015 12:08 AM To: Mark Hamstra mailto:m...@clearstorydata.com>> Cc: Xiaoyong Zhu mailto:xiaoy...@microsoft.com>>; Jeff Zhang mailto:zjf...@gmail.com>>; user mailto:user@spark.apache.org>>; Zhaomin Xu mailto:z...@microsoft.com>>; Joe Zhang (SDE) mailto:gui...@microsoft.com>> Subject: Re: Re: Spark assembly in Maven repo? Agree with you that assembly jar is not good to publish. However, what he really need is to fetch an updatable maven jar file. fightf...@163.com<mailto:fightf...@163.com> From: Mark Hamstra<mailto:m...@clearstorydata.com> Date: 2015-12-11 15:34 To: fightf...@163.com<mailto:fightf...@163.com> CC: Xiaoyong Zhu<mailto:xiaoy...@microsoft.com>; Jeff Zhang<mailto:zjf...@gmail.com>; user<mailto:user@spark.apache.org>; Zhaomin Xu<mailto:z...@microsoft.com>; Joe Zhang (SDE)<mailto:gui...@microsoft.com> Subject: Re: RE: Spark assembly in Maven repo? No, publishing a spark assembly jar is not fine. See the doc attached to https://issues.apache.org/jira/browse/SPARK-11157<https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fissues.apache.org%2fjira%2fbrowse%2fSPARK-11157&data=01%7c01%7cxiaoyzhu%40064d.mgd.microsoft.com%7c999bc04245724187198808d3020238b3%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=cUBvArmmYasxoOl9wrefYO4L7JiTF5qqqZDyY%2b%2bLP3Y%3d> and be aware that a likely goal of Spark 2.0 will be the elimination of assemblies. On Thu, Dec 10, 2015 at 11:19 PM, fightf...@163.com<mailto:fightf...@163.com> mailto:fightf...@163.com>> wrote: Using maven to download the assembly jar is fine. I would recommend to deploy this assembly jar to your local maven repo, i.e. nexus repo, Or more likey a snapshot repository fightf...@163.com<mailto:fightf...@163.com> From: Xiaoyong Zhu<mailto:xiaoy...@microsoft.com> Date: 2015-12-11 15:10 To: Jeff Zhang<mailto:zjf...@gmail.com> CC: user@spark.apache.org<mailto:user@spark.apache.org>; Zhaomin Xu<mailto:z...@microsoft.com>; Joe Zhang (SDE)<mailto:gui...@microsoft.com> Subject: RE: Spark assembly in Maven repo? Sorry – I didn’t make it clear. It’s actually not a “dependency” – it’s actually that we are building a certain plugin for IntelliJ where we want to distribute this jar. But since the jar is updated frequently we don't want to distribute it together with our plugin but we would like to download it via Maven. In this case what’s the recommended way? Xiaoyong From: Jeff Zhang [mailto:zjf...@gmail.com<mailto:zjf...@gmail.com>] Sent: Thursday, December 10, 2015 11:03 PM To: Xiaoyong Zhu mailto:xiaoy...@microsoft.com>> Cc: user@spark.apache.org<mailto:user@spark.apache.org> Subject: Re: Spark assembly in Maven repo? I don't think make the assembly jar as dependency a good practice. You may meet jar hell issue in that case. On Fri, Dec 11, 2015 at 2:46 PM, Xiaoyong Zhu mailto:xiaoy...@microsoft.com>> wrote: Hi Experts, We have a project which has a dependency for the following jar spark-assembly--hadoop.jar for example: spark-assembly-1.4.1.2.3.3.0-2983-hadoop2.7.1.2.3.3.0-2983.jar since this assembly might be updated in the future, I am not sure if there is a Maven repo that has the above spark assembly jar? Or should we create & upload it to Maven central? Thanks! Xiaoyong -- Best Regards Jeff Zhang
Re: Re: Spark assembly in Maven repo?
That's exactly what the various artifacts in the Maven repo are for. The API classes for core are in the core artifact and so on. You don't need an assembly. On Sat, Dec 12, 2015 at 12:32 AM, Xiaoyong Zhu wrote: > Yes, so our scenario is to treat the spark assembly as an “SDK” so users > can develop Spark applications easily without downloading them. In this > case which way do you guys think might be good? > > > > Xiaoyong > > > > *From:* fightf...@163.com [mailto:fightf...@163.com] > *Sent:* Friday, December 11, 2015 12:08 AM > *To:* Mark Hamstra > *Cc:* Xiaoyong Zhu ; Jeff Zhang ; > user ; Zhaomin Xu ; Joe Zhang > (SDE) > *Subject:* Re: Re: Spark assembly in Maven repo? > > > > Agree with you that assembly jar is not good to publish. However, what he > really need is to fetch > > an updatable maven jar file. > > > -- > > fightf...@163.com > > > > *From:* Mark Hamstra > > *Date:* 2015-12-11 15:34 > > *To:* fightf...@163.com > > *CC:* Xiaoyong Zhu ; Jeff Zhang ; > user ; Zhaomin Xu ; Joe Zhang > (SDE) > > *Subject:* Re: RE: Spark assembly in Maven repo? > > No, publishing a spark assembly jar is not fine. See the doc attached to > https://issues.apache.org/jira/browse/SPARK-11157 > <https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fissues.apache.org%2fjira%2fbrowse%2fSPARK-11157&data=01%7c01%7cxiaoyzhu%40064d.mgd.microsoft.com%7c999bc04245724187198808d3020238b3%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=cUBvArmmYasxoOl9wrefYO4L7JiTF5qqqZDyY%2b%2bLP3Y%3d> > and be aware that a likely goal of Spark 2.0 will be the elimination of > assemblies. > > > > On Thu, Dec 10, 2015 at 11:19 PM, fightf...@163.com > wrote: > > Using maven to download the assembly jar is fine. I would recommend to > deploy this > > assembly jar to your local maven repo, i.e. nexus repo, Or more likey a > snapshot repository > > > ---------- > > fightf...@163.com > > > > *From:* Xiaoyong Zhu > > *Date:* 2015-12-11 15:10 > > *To:* Jeff Zhang > > *CC:* user@spark.apache.org; Zhaomin Xu ; Joe Zhang > (SDE) > > *Subject:* RE: Spark assembly in Maven repo? > > Sorry – I didn’t make it clear. It’s actually not a “dependency” – it’s > actually that we are building a certain plugin for IntelliJ where we want > to distribute this jar. But since the jar is updated frequently we don't > want to distribute it together with our plugin but we would like to > download it via Maven. > > > > In this case what’s the recommended way? > > > > Xiaoyong > > > > *From:* Jeff Zhang [mailto:zjf...@gmail.com] > *Sent:* Thursday, December 10, 2015 11:03 PM > *To:* Xiaoyong Zhu > *Cc:* user@spark.apache.org > *Subject:* Re: Spark assembly in Maven repo? > > > > I don't think make the assembly jar as dependency a good practice. You may > meet jar hell issue in that case. > > > > On Fri, Dec 11, 2015 at 2:46 PM, Xiaoyong Zhu > wrote: > > Hi Experts, > > > > We have a project which has a dependency for the following jar > > > > spark-assembly--hadoop.jar > > for example: > > spark-assembly-1.4.1.2.3.3.0-2983-hadoop2.7.1.2.3.3.0-2983.jar > > > > since this assembly might be updated in the future, I am not sure if there > is a Maven repo that has the above spark assembly jar? Or should we create > & upload it to Maven central? > > > > Thanks! > > > > Xiaoyong > > > > > > > > -- > > Best Regards > > Jeff Zhang > > > >
RE: Re: Spark assembly in Maven repo?
Yes, so our scenario is to treat the spark assembly as an “SDK” so users can develop Spark applications easily without downloading them. In this case which way do you guys think might be good? Xiaoyong From: fightf...@163.com [mailto:fightf...@163.com] Sent: Friday, December 11, 2015 12:08 AM To: Mark Hamstra Cc: Xiaoyong Zhu ; Jeff Zhang ; user ; Zhaomin Xu ; Joe Zhang (SDE) Subject: Re: Re: Spark assembly in Maven repo? Agree with you that assembly jar is not good to publish. However, what he really need is to fetch an updatable maven jar file. fightf...@163.com<mailto:fightf...@163.com> From: Mark Hamstra<mailto:m...@clearstorydata.com> Date: 2015-12-11 15:34 To: fightf...@163.com<mailto:fightf...@163.com> CC: Xiaoyong Zhu<mailto:xiaoy...@microsoft.com>; Jeff Zhang<mailto:zjf...@gmail.com>; user<mailto:user@spark.apache.org>; Zhaomin Xu<mailto:z...@microsoft.com>; Joe Zhang (SDE)<mailto:gui...@microsoft.com> Subject: Re: RE: Spark assembly in Maven repo? No, publishing a spark assembly jar is not fine. See the doc attached to https://issues.apache.org/jira/browse/SPARK-11157<https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fissues.apache.org%2fjira%2fbrowse%2fSPARK-11157&data=01%7c01%7cxiaoyzhu%40064d.mgd.microsoft.com%7c999bc04245724187198808d3020238b3%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=cUBvArmmYasxoOl9wrefYO4L7JiTF5qqqZDyY%2b%2bLP3Y%3d> and be aware that a likely goal of Spark 2.0 will be the elimination of assemblies. On Thu, Dec 10, 2015 at 11:19 PM, fightf...@163.com<mailto:fightf...@163.com> mailto:fightf...@163.com>> wrote: Using maven to download the assembly jar is fine. I would recommend to deploy this assembly jar to your local maven repo, i.e. nexus repo, Or more likey a snapshot repository fightf...@163.com<mailto:fightf...@163.com> From: Xiaoyong Zhu<mailto:xiaoy...@microsoft.com> Date: 2015-12-11 15:10 To: Jeff Zhang<mailto:zjf...@gmail.com> CC: user@spark.apache.org<mailto:user@spark.apache.org>; Zhaomin Xu<mailto:z...@microsoft.com>; Joe Zhang (SDE)<mailto:gui...@microsoft.com> Subject: RE: Spark assembly in Maven repo? Sorry – I didn’t make it clear. It’s actually not a “dependency” – it’s actually that we are building a certain plugin for IntelliJ where we want to distribute this jar. But since the jar is updated frequently we don't want to distribute it together with our plugin but we would like to download it via Maven. In this case what’s the recommended way? Xiaoyong From: Jeff Zhang [mailto:zjf...@gmail.com<mailto:zjf...@gmail.com>] Sent: Thursday, December 10, 2015 11:03 PM To: Xiaoyong Zhu mailto:xiaoy...@microsoft.com>> Cc: user@spark.apache.org<mailto:user@spark.apache.org> Subject: Re: Spark assembly in Maven repo? I don't think make the assembly jar as dependency a good practice. You may meet jar hell issue in that case. On Fri, Dec 11, 2015 at 2:46 PM, Xiaoyong Zhu mailto:xiaoy...@microsoft.com>> wrote: Hi Experts, We have a project which has a dependency for the following jar spark-assembly--hadoop.jar for example: spark-assembly-1.4.1.2.3.3.0-2983-hadoop2.7.1.2.3.3.0-2983.jar since this assembly might be updated in the future, I am not sure if there is a Maven repo that has the above spark assembly jar? Or should we create & upload it to Maven central? Thanks! Xiaoyong -- Best Regards Jeff Zhang
Re: Re: Spark assembly in Maven repo?
Agree with you that assembly jar is not good to publish. However, what he really need is to fetch an updatable maven jar file. fightf...@163.com From: Mark Hamstra Date: 2015-12-11 15:34 To: fightf...@163.com CC: Xiaoyong Zhu; Jeff Zhang; user; Zhaomin Xu; Joe Zhang (SDE) Subject: Re: RE: Spark assembly in Maven repo? No, publishing a spark assembly jar is not fine. See the doc attached to https://issues.apache.org/jira/browse/SPARK-11157 and be aware that a likely goal of Spark 2.0 will be the elimination of assemblies. On Thu, Dec 10, 2015 at 11:19 PM, fightf...@163.com wrote: Using maven to download the assembly jar is fine. I would recommend to deploy this assembly jar to your local maven repo, i.e. nexus repo, Or more likey a snapshot repository fightf...@163.com From: Xiaoyong Zhu Date: 2015-12-11 15:10 To: Jeff Zhang CC: user@spark.apache.org; Zhaomin Xu; Joe Zhang (SDE) Subject: RE: Spark assembly in Maven repo? Sorry – I didn’t make it clear. It’s actually not a “dependency” – it’s actually that we are building a certain plugin for IntelliJ where we want to distribute this jar. But since the jar is updated frequently we don't want to distribute it together with our plugin but we would like to download it via Maven. In this case what’s the recommended way? Xiaoyong From: Jeff Zhang [mailto:zjf...@gmail.com] Sent: Thursday, December 10, 2015 11:03 PM To: Xiaoyong Zhu Cc: user@spark.apache.org Subject: Re: Spark assembly in Maven repo? I don't think make the assembly jar as dependency a good practice. You may meet jar hell issue in that case. On Fri, Dec 11, 2015 at 2:46 PM, Xiaoyong Zhu wrote: Hi Experts, We have a project which has a dependency for the following jar spark-assembly--hadoop.jar for example: spark-assembly-1.4.1.2.3.3.0-2983-hadoop2.7.1.2.3.3.0-2983.jar since this assembly might be updated in the future, I am not sure if there is a Maven repo that has the above spark assembly jar? Or should we create & upload it to Maven central? Thanks! Xiaoyong -- Best Regards Jeff Zhang
Re: RE: Spark assembly in Maven repo?
No, publishing a spark assembly jar is not fine. See the doc attached to https://issues.apache.org/jira/browse/SPARK-11157 and be aware that a likely goal of Spark 2.0 will be the elimination of assemblies. On Thu, Dec 10, 2015 at 11:19 PM, fightf...@163.com wrote: > Using maven to download the assembly jar is fine. I would recommend to > deploy this > assembly jar to your local maven repo, i.e. nexus repo, Or more likey a > snapshot repository > > -- > fightf...@163.com > > > *From:* Xiaoyong Zhu > *Date:* 2015-12-11 15:10 > *To:* Jeff Zhang > *CC:* user@spark.apache.org; Zhaomin Xu ; Joe Zhang > (SDE) > *Subject:* RE: Spark assembly in Maven repo? > > Sorry – I didn’t make it clear. It’s actually not a “dependency” – it’s > actually that we are building a certain plugin for IntelliJ where we want > to distribute this jar. But since the jar is updated frequently we don't > want to distribute it together with our plugin but we would like to > download it via Maven. > > > > In this case what’s the recommended way? > > > > Xiaoyong > > > > *From:* Jeff Zhang [mailto:zjf...@gmail.com] > *Sent:* Thursday, December 10, 2015 11:03 PM > *To:* Xiaoyong Zhu > *Cc:* user@spark.apache.org > *Subject:* Re: Spark assembly in Maven repo? > > > > I don't think make the assembly jar as dependency a good practice. You may > meet jar hell issue in that case. > > > > On Fri, Dec 11, 2015 at 2:46 PM, Xiaoyong Zhu > wrote: > > Hi Experts, > > > > We have a project which has a dependency for the following jar > > > > spark-assembly--hadoop.jar > > for example: > > spark-assembly-1.4.1.2.3.3.0-2983-hadoop2.7.1.2.3.3.0-2983.jar > > > > since this assembly might be updated in the future, I am not sure if there > is a Maven repo that has the above spark assembly jar? Or should we create > & upload it to Maven central? > > > > Thanks! > > > > Xiaoyong > > > > > > > > -- > > Best Regards > > Jeff Zhang > >
Re: RE: Spark assembly in Maven repo?
Using maven to download the assembly jar is fine. I would recommend to deploy this assembly jar to your local maven repo, i.e. nexus repo, Or more likey a snapshot repository fightf...@163.com From: Xiaoyong Zhu Date: 2015-12-11 15:10 To: Jeff Zhang CC: user@spark.apache.org; Zhaomin Xu; Joe Zhang (SDE) Subject: RE: Spark assembly in Maven repo? Sorry – I didn’t make it clear. It’s actually not a “dependency” – it’s actually that we are building a certain plugin for IntelliJ where we want to distribute this jar. But since the jar is updated frequently we don't want to distribute it together with our plugin but we would like to download it via Maven. In this case what’s the recommended way? Xiaoyong From: Jeff Zhang [mailto:zjf...@gmail.com] Sent: Thursday, December 10, 2015 11:03 PM To: Xiaoyong Zhu Cc: user@spark.apache.org Subject: Re: Spark assembly in Maven repo? I don't think make the assembly jar as dependency a good practice. You may meet jar hell issue in that case. On Fri, Dec 11, 2015 at 2:46 PM, Xiaoyong Zhu wrote: Hi Experts, We have a project which has a dependency for the following jar spark-assembly--hadoop.jar for example: spark-assembly-1.4.1.2.3.3.0-2983-hadoop2.7.1.2.3.3.0-2983.jar since this assembly might be updated in the future, I am not sure if there is a Maven repo that has the above spark assembly jar? Or should we create & upload it to Maven central? Thanks! Xiaoyong -- Best Regards Jeff Zhang
RE: Spark assembly in Maven repo?
Sorry – I didn’t make it clear. It’s actually not a “dependency” – it’s actually that we are building a certain plugin for IntelliJ where we want to distribute this jar. But since the jar is updated frequently we don't want to distribute it together with our plugin but we would like to download it via Maven. In this case what’s the recommended way? Xiaoyong From: Jeff Zhang [mailto:zjf...@gmail.com] Sent: Thursday, December 10, 2015 11:03 PM To: Xiaoyong Zhu Cc: user@spark.apache.org Subject: Re: Spark assembly in Maven repo? I don't think make the assembly jar as dependency a good practice. You may meet jar hell issue in that case. On Fri, Dec 11, 2015 at 2:46 PM, Xiaoyong Zhu mailto:xiaoy...@microsoft.com>> wrote: Hi Experts, We have a project which has a dependency for the following jar spark-assembly--hadoop.jar for example: spark-assembly-1.4.1.2.3.3.0-2983-hadoop2.7.1.2.3.3.0-2983.jar since this assembly might be updated in the future, I am not sure if there is a Maven repo that has the above spark assembly jar? Or should we create & upload it to Maven central? Thanks! Xiaoyong -- Best Regards Jeff Zhang
Re: Spark assembly in Maven repo?
I don't think make the assembly jar as dependency a good practice. You may meet jar hell issue in that case. On Fri, Dec 11, 2015 at 2:46 PM, Xiaoyong Zhu wrote: > Hi Experts, > > > > We have a project which has a dependency for the following jar > > > > spark-assembly--hadoop.jar > > for example: > > spark-assembly-1.4.1.2.3.3.0-2983-hadoop2.7.1.2.3.3.0-2983.jar > > > > since this assembly might be updated in the future, I am not sure if there > is a Maven repo that has the above spark assembly jar? Or should we create > & upload it to Maven central? > > > > Thanks! > > > > Xiaoyong > > > -- Best Regards Jeff Zhang
Spark assembly in Maven repo?
Hi Experts, We have a project which has a dependency for the following jar spark-assembly--hadoop.jar for example: spark-assembly-1.4.1.2.3.3.0-2983-hadoop2.7.1.2.3.3.0-2983.jar since this assembly might be updated in the future, I am not sure if there is a Maven repo that has the above spark assembly jar? Or should we create & upload it to Maven central? Thanks! Xiaoyong
