[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[ https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730926#comment-14730926 ] ASF GitHub Bot commented on FLINK-2161: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/805 > Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML) > -- > > Key: FLINK-2161 > URL: https://issues.apache.org/jira/browse/FLINK-2161 > Project: Flink > Issue Type: Improvement >Reporter: Till Rohrmann >Assignee: Nikolaas Steenbergen > Fix For: 0.10 > > > Currently, there is no easy way to load and ship external libraries/jars with > Flink's Scala shell. Assume that you want to run some Gelly graph algorithms > from within the Scala shell, then you have to put the Gelly jar manually in > the lib directory and make sure that this jar is also available on your > cluster, because it is not shipped with the user code. > It would be good to have a simple mechanism how to specify additional jars > upon startup of the Scala shell. These jars should then also be shipped to > the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[ https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14727426#comment-14727426 ] ASF GitHub Bot commented on FLINK-2161: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/805#issuecomment-137104511 Sorry, it fell behind in the long list of Pull Requests. I'll check it and will merge it if Travis gives a green light. > Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML) > -- > > Key: FLINK-2161 > URL: https://issues.apache.org/jira/browse/FLINK-2161 > Project: Flink > Issue Type: Improvement >Reporter: Till Rohrmann >Assignee: Nikolaas Steenbergen > > Currently, there is no easy way to load and ship external libraries/jars with > Flink's Scala shell. Assume that you want to run some Gelly graph algorithms > from within the Scala shell, then you have to put the Gelly jar manually in > the lib directory and make sure that this jar is also available on your > cluster, because it is not shipped with the user code. > It would be good to have a simple mechanism how to specify additional jars > upon startup of the Scala shell. These jars should then also be shipped to > the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[ https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14725291#comment-14725291 ] ASF GitHub Bot commented on FLINK-2161: --- Github user nikste commented on the pull request: https://github.com/apache/flink/pull/805#issuecomment-136689479 Could someone please check this, it has been here for a while and I've got some other stuff building on top of this! > Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML) > -- > > Key: FLINK-2161 > URL: https://issues.apache.org/jira/browse/FLINK-2161 > Project: Flink > Issue Type: Improvement >Reporter: Till Rohrmann >Assignee: Nikolaas Steenbergen > > Currently, there is no easy way to load and ship external libraries/jars with > Flink's Scala shell. Assume that you want to run some Gelly graph algorithms > from within the Scala shell, then you have to put the Gelly jar manually in > the lib directory and make sure that this jar is also available on your > cluster, because it is not shipped with the user code. > It would be good to have a simple mechanism how to specify additional jars > upon startup of the Scala shell. These jars should then also be shipped to > the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[ https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14642620#comment-14642620 ] ASF GitHub Bot commented on FLINK-2161: --- Github user nikste commented on the pull request: https://github.com/apache/flink/pull/805#issuecomment-125179652 The IT case now looks for a jar inside the ml-folder which does not contain "test" or "original". Thus the IT case should work for different flink-versions as long as the ml module is made before the scala-shell. > Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML) > -- > > Key: FLINK-2161 > URL: https://issues.apache.org/jira/browse/FLINK-2161 > Project: Flink > Issue Type: Improvement >Reporter: Till Rohrmann >Assignee: Nikolaas Steenbergen > > Currently, there is no easy way to load and ship external libraries/jars with > Flink's Scala shell. Assume that you want to run some Gelly graph algorithms > from within the Scala shell, then you have to put the Gelly jar manually in > the lib directory and make sure that this jar is also available on your > cluster, because it is not shipped with the user code. > It would be good to have a simple mechanism how to specify additional jars > upon startup of the Scala shell. These jars should then also be shipped to > the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[ https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14600917#comment-14600917 ] ASF GitHub Bot commented on FLINK-2161: --- Github user nikste commented on the pull request: https://github.com/apache/flink/pull/805#issuecomment-115180597 This did not work unfortunately, the class was available in the test, but unfortunately not in the shell which is invoked in the test. However, if you add the classpath of the external class to ```settings.classpath.value``` of the scala shell before starting it, it seems to work. I added a test for instantiating and printing a DenseVector with flink-ml jar. This should check if the external jar is sent to the cluster. The only remaining problem is the name of the jar, which will change if the flink-version changes. > Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML) > -- > > Key: FLINK-2161 > URL: https://issues.apache.org/jira/browse/FLINK-2161 > Project: Flink > Issue Type: Improvement >Reporter: Till Rohrmann >Assignee: Nikolaas Steenbergen > > Currently, there is no easy way to load and ship external libraries/jars with > Flink's Scala shell. Assume that you want to run some Gelly graph algorithms > from within the Scala shell, then you have to put the Gelly jar manually in > the lib directory and make sure that this jar is also available on your > cluster, because it is not shipped with the user code. > It would be good to have a simple mechanism how to specify additional jars > upon startup of the Scala shell. These jars should then also be shipped to > the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[ https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14595550#comment-14595550 ] ASF GitHub Bot commented on FLINK-2161: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/805#issuecomment-114046659 To add more jars to the Classpath, you can write the test in the following way: The test itself is just a wrapper that instantiates a custom classloader with your extra class files. It will the load the test class with Class.forName(classname, true, createdClassLoader). That way, the test has the extra classes available... > Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML) > -- > > Key: FLINK-2161 > URL: https://issues.apache.org/jira/browse/FLINK-2161 > Project: Flink > Issue Type: Improvement >Reporter: Till Rohrmann >Assignee: Nikolaas Steenbergen > > Currently, there is no easy way to load and ship external libraries/jars with > Flink's Scala shell. Assume that you want to run some Gelly graph algorithms > from within the Scala shell, then you have to put the Gelly jar manually in > the lib directory and make sure that this jar is also available on your > cluster, because it is not shipped with the user code. > It would be good to have a simple mechanism how to specify additional jars > upon startup of the Scala shell. These jars should then also be shipped to > the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[ https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14588259#comment-14588259 ] ASF GitHub Bot commented on FLINK-2161: --- Github user nikste commented on the pull request: https://github.com/apache/flink/pull/805#issuecomment-112477088 Changed according to comments, @tillrohrmann I did not check it on a cluster, since I don't have one available at the moment. I checked with a local jobmanager (using "start-local.sh"), that seemed to work. Maybe you can check if it works with a remote cluster as well? Concerning automated tests, its somewhat tricky. The jar has to be sent to the remote execution and at the same time be available in the shell. You cant add jars from inside the shell (scala 2.10 bug:https://issues.scala-lang.org/browse/SI-6502), only at startup with an additional parameter (see bash script: start-scala-shell.sh). If I try to invoke the start-scala-shell.sh script from inside java/scala and redirect the std out I only get half of the output (everything till the actual scala shell starts). Is it possible to add an additional classpath in maven for the tests? Then at least I could write a test, that checks if the jars are shipped to the remote environment.. Or do you have another suggestion maybe? > Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML) > -- > > Key: FLINK-2161 > URL: https://issues.apache.org/jira/browse/FLINK-2161 > Project: Flink > Issue Type: Improvement >Reporter: Till Rohrmann >Assignee: Nikolaas Steenbergen > > Currently, there is no easy way to load and ship external libraries/jars with > Flink's Scala shell. Assume that you want to run some Gelly graph algorithms > from within the Scala shell, then you have to put the Gelly jar manually in > the lib directory and make sure that this jar is also available on your > cluster, because it is not shipped with the user code. > It would be good to have a simple mechanism how to specify additional jars > upon startup of the Scala shell. These jars should then also be shipped to > the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[
https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587810#comment-14587810
]
ASF GitHub Bot commented on FLINK-2161:
---
Github user nikste commented on a diff in the pull request:
https://github.com/apache/flink/pull/805#discussion_r32508286
--- Diff:
flink-staging/flink-scala-shell/src/main/scala/org.apache.flink/api/scala/FlinkShell.scala
---
@@ -44,6 +46,16 @@ object FlinkShell {
c.copy (host = x)
} text("host specifies host name of running JobManager")
+ opt[(String)] ('a',"addclasspath") action {
+case (x,c) =>
+ val xArray = x.split(":")
+ var extJarArray = new Array[String](xArray.length)
+ for(i <- 0 to xArray.length - 1){
+extJarArray(i) = xArray(i)
+ }
--- End diff --
good point, it's a remainder of an older version with a slightly different
approach.
> Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
> --
>
> Key: FLINK-2161
> URL: https://issues.apache.org/jira/browse/FLINK-2161
> Project: Flink
> Issue Type: Improvement
>Reporter: Till Rohrmann
>Assignee: Nikolaas Steenbergen
>
> Currently, there is no easy way to load and ship external libraries/jars with
> Flink's Scala shell. Assume that you want to run some Gelly graph algorithms
> from within the Scala shell, then you have to put the Gelly jar manually in
> the lib directory and make sure that this jar is also available on your
> cluster, because it is not shipped with the user code.
> It would be good to have a simple mechanism how to specify additional jars
> upon startup of the Scala shell. These jars should then also be shipped to
> the cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[
https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587808#comment-14587808
]
ASF GitHub Bot commented on FLINK-2161:
---
Github user nikste commented on a diff in the pull request:
https://github.com/apache/flink/pull/805#discussion_r32508271
--- Diff:
flink-staging/flink-scala-shell/src/main/scala/org.apache.flink/api/scala/FlinkILoop.scala
---
@@ -191,5 +203,14 @@ HINT: You can use print() on a DataSet to print the
contents to this shell.
)
}
+
+ def getExternalJars(): Array[String] ={
+var extJars : Array[String] = Array.empty[String]
+this.externalJars match{
+ case Some(ej) => extJars = ej
+ case None =>
+}
+extJars
--- End diff --
This is much nicer, thanks!
> Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
> --
>
> Key: FLINK-2161
> URL: https://issues.apache.org/jira/browse/FLINK-2161
> Project: Flink
> Issue Type: Improvement
>Reporter: Till Rohrmann
>Assignee: Nikolaas Steenbergen
>
> Currently, there is no easy way to load and ship external libraries/jars with
> Flink's Scala shell. Assume that you want to run some Gelly graph algorithms
> from within the Scala shell, then you have to put the Gelly jar manually in
> the lib directory and make sure that this jar is also available on your
> cluster, because it is not shipped with the user code.
> It would be good to have a simple mechanism how to specify additional jars
> upon startup of the Scala shell. These jars should then also be shipped to
> the cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[
https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587809#comment-14587809
]
ASF GitHub Bot commented on FLINK-2161:
---
Github user nikste commented on a diff in the pull request:
https://github.com/apache/flink/pull/805#discussion_r32508276
--- Diff:
flink-staging/flink-scala-shell/src/main/scala/org.apache.flink/api/scala/FlinkShell.scala
---
@@ -30,7 +31,8 @@ object FlinkShell {
// scopt, command line arguments
case class Config(port: Int = -1,
--- End diff --
This goes totally against my sense of aesthetics, but ok.
> Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
> --
>
> Key: FLINK-2161
> URL: https://issues.apache.org/jira/browse/FLINK-2161
> Project: Flink
> Issue Type: Improvement
>Reporter: Till Rohrmann
>Assignee: Nikolaas Steenbergen
>
> Currently, there is no easy way to load and ship external libraries/jars with
> Flink's Scala shell. Assume that you want to run some Gelly graph algorithms
> from within the Scala shell, then you have to put the Gelly jar manually in
> the lib directory and make sure that this jar is also available on your
> cluster, because it is not shipped with the user code.
> It would be good to have a simple mechanism how to specify additional jars
> upon startup of the Scala shell. These jars should then also be shipped to
> the cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[ https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14585869#comment-14585869 ] ASF GitHub Bot commented on FLINK-2161: --- Github user tillrohrmann commented on the pull request: https://github.com/apache/flink/pull/805#issuecomment-112053328 Looks good @nikste. I only had some minor comments. Did you test the code on a cluster? Can we maybe add a test case which tests that external jars are shipped? > Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML) > -- > > Key: FLINK-2161 > URL: https://issues.apache.org/jira/browse/FLINK-2161 > Project: Flink > Issue Type: Improvement >Reporter: Till Rohrmann >Assignee: Nikolaas Steenbergen > > Currently, there is no easy way to load and ship external libraries/jars with > Flink's Scala shell. Assume that you want to run some Gelly graph algorithms > from within the Scala shell, then you have to put the Gelly jar manually in > the lib directory and make sure that this jar is also available on your > cluster, because it is not shipped with the user code. > It would be good to have a simple mechanism how to specify additional jars > upon startup of the Scala shell. These jars should then also be shipped to > the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[
https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14585866#comment-14585866
]
ASF GitHub Bot commented on FLINK-2161:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/805#discussion_r32415111
--- Diff:
flink-staging/flink-scala-shell/src/main/scala/org.apache.flink/api/scala/FlinkShell.scala
---
@@ -51,14 +63,16 @@ object FlinkShell {
// parse arguments
parser.parse (args, Config () ) match {
case Some(config) =>
-startShell(config.host,config.port)
+startShell(config.host,config.port,config.externalJars)
case _ => println("Could not parse program arguments")
}
}
- def startShell(userHost : String, userPort : Int): Unit ={
+ def startShell(userHost : String,
--- End diff --
parameters start in a new line with 4 spaces indentation
> Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
> --
>
> Key: FLINK-2161
> URL: https://issues.apache.org/jira/browse/FLINK-2161
> Project: Flink
> Issue Type: Improvement
>Reporter: Till Rohrmann
>Assignee: Nikolaas Steenbergen
>
> Currently, there is no easy way to load and ship external libraries/jars with
> Flink's Scala shell. Assume that you want to run some Gelly graph algorithms
> from within the Scala shell, then you have to put the Gelly jar manually in
> the lib directory and make sure that this jar is also available on your
> cluster, because it is not shipped with the user code.
> It would be good to have a simple mechanism how to specify additional jars
> upon startup of the Scala shell. These jars should then also be shipped to
> the cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[
https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14585865#comment-14585865
]
ASF GitHub Bot commented on FLINK-2161:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/805#discussion_r32415084
--- Diff:
flink-staging/flink-scala-shell/src/main/scala/org.apache.flink/api/scala/FlinkShell.scala
---
@@ -44,6 +46,16 @@ object FlinkShell {
c.copy (host = x)
} text("host specifies host name of running JobManager")
+ opt[(String)] ('a',"addclasspath") action {
+case (x,c) =>
+ val xArray = x.split(":")
+ var extJarArray = new Array[String](xArray.length)
+ for(i <- 0 to xArray.length - 1){
+extJarArray(i) = xArray(i)
+ }
--- End diff --
Why do you copy `xArray` here? And why not simply calling `xArray.clone`?
> Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
> --
>
> Key: FLINK-2161
> URL: https://issues.apache.org/jira/browse/FLINK-2161
> Project: Flink
> Issue Type: Improvement
>Reporter: Till Rohrmann
>Assignee: Nikolaas Steenbergen
>
> Currently, there is no easy way to load and ship external libraries/jars with
> Flink's Scala shell. Assume that you want to run some Gelly graph algorithms
> from within the Scala shell, then you have to put the Gelly jar manually in
> the lib directory and make sure that this jar is also available on your
> cluster, because it is not shipped with the user code.
> It would be good to have a simple mechanism how to specify additional jars
> upon startup of the Scala shell. These jars should then also be shipped to
> the cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[
https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14585860#comment-14585860
]
ASF GitHub Bot commented on FLINK-2161:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/805#discussion_r32414932
--- Diff:
flink-staging/flink-scala-shell/src/main/scala/org.apache.flink/api/scala/FlinkShell.scala
---
@@ -30,7 +31,8 @@ object FlinkShell {
// scopt, command line arguments
case class Config(port: Int = -1,
--- End diff --
parameters should start in a new line with 4 spaces indentation
> Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
> --
>
> Key: FLINK-2161
> URL: https://issues.apache.org/jira/browse/FLINK-2161
> Project: Flink
> Issue Type: Improvement
>Reporter: Till Rohrmann
>Assignee: Nikolaas Steenbergen
>
> Currently, there is no easy way to load and ship external libraries/jars with
> Flink's Scala shell. Assume that you want to run some Gelly graph algorithms
> from within the Scala shell, then you have to put the Gelly jar manually in
> the lib directory and make sure that this jar is also available on your
> cluster, because it is not shipped with the user code.
> It would be good to have a simple mechanism how to specify additional jars
> upon startup of the Scala shell. These jars should then also be shipped to
> the cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[
https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14585859#comment-14585859
]
ASF GitHub Bot commented on FLINK-2161:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/805#discussion_r32414909
--- Diff:
flink-staging/flink-scala-shell/src/main/scala/org.apache.flink/api/scala/FlinkILoop.scala
---
@@ -191,5 +203,14 @@ HINT: You can use print() on a DataSet to print the
contents to this shell.
)
}
+
+ def getExternalJars(): Array[String] ={
+var extJars : Array[String] = Array.empty[String]
+this.externalJars match{
+ case Some(ej) => extJars = ej
+ case None =>
+}
+extJars
--- End diff --
Why not doing simply:
```
def getExternalJars(): Array[String] =
externalJars.getOrElse(Array.empty[String])
```
?
> Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
> --
>
> Key: FLINK-2161
> URL: https://issues.apache.org/jira/browse/FLINK-2161
> Project: Flink
> Issue Type: Improvement
>Reporter: Till Rohrmann
>Assignee: Nikolaas Steenbergen
>
> Currently, there is no easy way to load and ship external libraries/jars with
> Flink's Scala shell. Assume that you want to run some Gelly graph algorithms
> from within the Scala shell, then you have to put the Gelly jar manually in
> the lib directory and make sure that this jar is also available on your
> cluster, because it is not shipped with the user code.
> It would be good to have a simple mechanism how to specify additional jars
> upon startup of the Scala shell. These jars should then also be shipped to
> the cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[ https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14585842#comment-14585842 ] ASF GitHub Bot commented on FLINK-2161: --- Github user nikste commented on the pull request: https://github.com/apache/flink/pull/805#issuecomment-112041788 This should work now. It sends the jar alongside the compiled user-code to the cluster. > Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML) > -- > > Key: FLINK-2161 > URL: https://issues.apache.org/jira/browse/FLINK-2161 > Project: Flink > Issue Type: Improvement >Reporter: Till Rohrmann >Assignee: Nikolaas Steenbergen > > Currently, there is no easy way to load and ship external libraries/jars with > Flink's Scala shell. Assume that you want to run some Gelly graph algorithms > from within the Scala shell, then you have to put the Gelly jar manually in > the lib directory and make sure that this jar is also available on your > cluster, because it is not shipped with the user code. > It would be good to have a simple mechanism how to specify additional jars > upon startup of the Scala shell. These jars should then also be shipped to > the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[ https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577425#comment-14577425 ] ASF GitHub Bot commented on FLINK-2161: --- Github user nikste commented on the pull request: https://github.com/apache/flink/pull/805#issuecomment-110064581 You are right. I only tried it with the local execution environment. It does not work if you start a separate job manager on your machine. > Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML) > -- > > Key: FLINK-2161 > URL: https://issues.apache.org/jira/browse/FLINK-2161 > Project: Flink > Issue Type: Improvement >Reporter: Till Rohrmann >Assignee: Nikolaas Steenbergen > > Currently, there is no easy way to load and ship external libraries/jars with > Flink's Scala shell. Assume that you want to run some Gelly graph algorithms > from within the Scala shell, then you have to put the Gelly jar manually in > the lib directory and make sure that this jar is also available on your > cluster, because it is not shipped with the user code. > It would be good to have a simple mechanism how to specify additional jars > upon startup of the Scala shell. These jars should then also be shipped to > the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[ https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577207#comment-14577207 ] ASF GitHub Bot commented on FLINK-2161: --- Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/805#issuecomment-110001500 I think this is only half of what is needed. The other half would be sending the jar files along when submitting the job with the modified RemoteEnvironment. > Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML) > -- > > Key: FLINK-2161 > URL: https://issues.apache.org/jira/browse/FLINK-2161 > Project: Flink > Issue Type: Improvement >Reporter: Till Rohrmann >Assignee: Nikolaas Steenbergen > > Currently, there is no easy way to load and ship external libraries/jars with > Flink's Scala shell. Assume that you want to run some Gelly graph algorithms > from within the Scala shell, then you have to put the Gelly jar manually in > the lib directory and make sure that this jar is also available on your > cluster, because it is not shipped with the user code. > It would be good to have a simple mechanism how to specify additional jars > upon startup of the Scala shell. These jars should then also be shipped to > the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[ https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577091#comment-14577091 ] ASF GitHub Bot commented on FLINK-2161: --- GitHub user nikste opened a pull request: https://github.com/apache/flink/pull/805 [FLINK-2161] modified Scala shell start script to take additional argument (-cp or --classpath… …) for external libraries You can merge this pull request into a Git repository by running: $ git pull https://github.com/nikste/flink Scala_shell_external_jar Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/805.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #805 commit 80ad72c05bb27413be98271ff16c907a7bc3a0e8 Author: Nikolaas Steenbergen Date: 2015-06-08T12:01:39Z [FLINK-2161] modified start script to take additional argument (-cp or --classpath) for external libraries > Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML) > -- > > Key: FLINK-2161 > URL: https://issues.apache.org/jira/browse/FLINK-2161 > Project: Flink > Issue Type: Improvement >Reporter: Till Rohrmann >Assignee: Nikolaas Steenbergen > > Currently, there is no easy way to load and ship external libraries/jars with > Flink's Scala shell. Assume that you want to run some Gelly graph algorithms > from within the Scala shell, then you have to put the Gelly jar manually in > the lib directory and make sure that this jar is also available on your > cluster, because it is not shipped with the user code. > It would be good to have a simple mechanism how to specify additional jars > upon startup of the Scala shell. These jars should then also be shipped to > the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[ https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14576958#comment-14576958 ] Nikolaas Steenbergen commented on FLINK-2161: - Unfortunately runtime adding of jars (for 2.10) is broken see: https://issues.scala-lang.org/browse/SI-6502 I'm editing the bashscript to load it at startup > Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML) > -- > > Key: FLINK-2161 > URL: https://issues.apache.org/jira/browse/FLINK-2161 > Project: Flink > Issue Type: Improvement >Reporter: Till Rohrmann >Assignee: Nikolaas Steenbergen > > Currently, there is no easy way to load and ship external libraries/jars with > Flink's Scala shell. Assume that you want to run some Gelly graph algorithms > from within the Scala shell, then you have to put the Gelly jar manually in > the lib directory and make sure that this jar is also available on your > cluster, because it is not shipped with the user code. > It would be good to have a simple mechanism how to specify additional jars > upon startup of the Scala shell. These jars should then also be shipped to > the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[ https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14576867#comment-14576867 ] Till Rohrmann commented on FLINK-2161: -- It would also be cool to be able to specify them a start up via a command line option. On Mon, Jun 8, 2015 at 11:45 AM Aljoscha Krettek (JIRA) > Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML) > -- > > Key: FLINK-2161 > URL: https://issues.apache.org/jira/browse/FLINK-2161 > Project: Flink > Issue Type: Improvement >Reporter: Till Rohrmann >Assignee: Nikolaas Steenbergen > > Currently, there is no easy way to load and ship external libraries/jars with > Flink's Scala shell. Assume that you want to run some Gelly graph algorithms > from within the Scala shell, then you have to put the Gelly jar manually in > the lib directory and make sure that this jar is also available on your > cluster, because it is not shipped with the user code. > It would be good to have a simple mechanism how to specify additional jars > upon startup of the Scala shell. These jars should then also be shipped to > the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[ https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14576837#comment-14576837 ] Aljoscha Krettek commented on FLINK-2161: - The Scala REPL has support for adding jars at runtime. We would just need to add support for sending them along with the compiled classes. > Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML) > -- > > Key: FLINK-2161 > URL: https://issues.apache.org/jira/browse/FLINK-2161 > Project: Flink > Issue Type: Improvement >Reporter: Till Rohrmann >Assignee: Nikolaas Steenbergen > > Currently, there is no easy way to load and ship external libraries/jars with > Flink's Scala shell. Assume that you want to run some Gelly graph algorithms > from within the Scala shell, then you have to put the Gelly jar manually in > the lib directory and make sure that this jar is also available on your > cluster, because it is not shipped with the user code. > It would be good to have a simple mechanism how to specify additional jars > upon startup of the Scala shell. These jars should then also be shipped to > the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2161) Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
[
https://issues.apache.org/jira/browse/FLINK-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14576751#comment-14576751
]
Till Rohrmann commented on FLINK-2161:
--
A simple example for FlinkML would be
{code}
import org.apache.flink.ml.math._
val denseVectors = env.fromElements(DenseVector(1.0, 2.0, 3.0))
denseVectors.print()
{code}
> Flink Scala Shell does not support external jars (e.g. Gelly, FlinkML)
> --
>
> Key: FLINK-2161
> URL: https://issues.apache.org/jira/browse/FLINK-2161
> Project: Flink
> Issue Type: Improvement
>Reporter: Till Rohrmann
>Assignee: Nikolaas Steenbergen
>
> Currently, there is no easy way to load and ship external libraries/jars with
> Flink's Scala shell. Assume that you want to run some Gelly graph algorithms
> from within the Scala shell, then you have to put the Gelly jar manually in
> the lib directory and make sure that this jar is also available on your
> cluster, because it is not shipped with the user code.
> It would be good to have a simple mechanism how to specify additional jars
> upon startup of the Scala shell. These jars should then also be shipped to
> the cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
