[jira] [Commented] (CONNECTORS-1639) Upgrade Elastic Search Version
[ https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087254#comment-17087254 ] Michael Cizmar commented on CONNECTORS-1639: Was away from my desk, the setup script for something very vanilla for elastic would be: * Download elastic * unpack it * Run bin/elasticsearch We could modify the configuration files that we would need but you would get a one node cluster to test. I saw this as well just now and it's got a 'wait for elastic' to come up. [https://github.com/dadoonet/fscrawler/blob/2829f49074ccc2692fb257e36c0d3be6300b3c41/src/test/ant/integration-tests.xm|https://github.com/dadoonet/fscrawler/blob/2829f49074ccc2692fb257e36c0d3be6300b3c41/src/test/ant/integration-tests.xml] I will work on a script (sh) for you. > Upgrade Elastic Search Version > -- > > Key: CONNECTORS-1639 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1639 > Project: ManifoldCF > Issue Type: Improvement > Components: Elastic Search connector >Reporter: Cihad Guzel >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > Attachments: CONNECTORS-1639.diff, > elastic-search-1.0.1-java11-build-error.log > > > Current Elastic Search version is 1.0.1 . According to [this > matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not > supported by any ES version below 6.5. > Besides, ES 1.x is no longer supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (CONNECTORS-1639) Upgrade Elastic Search Version
[
https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087245#comment-17087245
]
Karl Wright edited comment on CONNECTORS-1639 at 4/19/20, 11:09 PM:
Since this is firing off a java task, we could use the Ant Java task, described
here:
https://ant.apache.org/manual/Tasks/java.html
However, this will not work for maven so I'd steer away from that.
We can do something similar to this instead, which would invoke the same Java
again, although we'd want ES's classpath obviously:
{code}
public final class JavaProcess {
private JavaProcess() {}
public static int exec(Class klass, List args) throws IOException,
InterruptedException {
String javaHome = System.getProperty("java.home");
String javaBin = javaHome +
File.separator + "bin" +
File.separator + "java";
String classpath = System.getProperty("java.class.path");
String className = klass.getName();
List command = new LinkedList();
command.add(javaBin);
command.add("-cp");
command.add(classpath);
command.add(className);
if (args != null) {
command.addAll(args);
}
ProcessBuilder builder = new ProcessBuilder(command);
Process process = builder.inheritIO().start();
process.waitFor();
return process.exitValue();
}
}
{code}
There are, however, the following problems to be addressed: (1) waiting for the
instance to start, and (2) stopping the instance.
was (Author: kwri...@metacarta.com):
Since this is firing off a java task, we could use the Ant Java task, described
here:
https://ant.apache.org/manual/Tasks/java.html
However, this will not work for maven so I'd steer away from that.
We can do something similar to this instead, which would invoke the same Java
again, although we'd want ES's classpath obviously:
{code}
> Upgrade Elastic Search Version
> --
>
> Key: CONNECTORS-1639
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1639
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Elastic Search connector
>Reporter: Cihad Guzel
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.16
>
> Attachments: CONNECTORS-1639.diff,
> elastic-search-1.0.1-java11-build-error.log
>
>
> Current Elastic Search version is 1.0.1 . According to [this
> matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not
> supported by any ES version below 6.5.
> Besides, ES 1.x is no longer supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1639) Upgrade Elastic Search Version
[
https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087245#comment-17087245
]
Karl Wright commented on CONNECTORS-1639:
-
Since this is firing off a java task, we could use the Ant Java task, described
here:
https://ant.apache.org/manual/Tasks/java.html
However, this will not work for maven so I'd steer away from that.
We can do something similar to this instead, which would invoke the same Java
again, although we'd want ES's classpath obviously:
{code}
> Upgrade Elastic Search Version
> --
>
> Key: CONNECTORS-1639
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1639
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Elastic Search connector
>Reporter: Cihad Guzel
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.16
>
> Attachments: CONNECTORS-1639.diff,
> elastic-search-1.0.1-java11-build-error.log
>
>
> Current Elastic Search version is 1.0.1 . According to [this
> matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not
> supported by any ES version below 6.5.
> Besides, ES 1.x is no longer supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1639) Upgrade Elastic Search Version
[
https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086994#comment-17086994
]
Karl Wright commented on CONNECTORS-1639:
-
It looks like we can reverse engineer the startup script here:
{code}
test-materials/elasticsearch-7.6.2/bin/elasticsearch"
{code}
The basic daemon runner is:
{code}
exec \
"$JAVA" \
$ES_JAVA_OPTS \
-Des.path.home="$ES_HOME" \
-Des.path.conf="$ES_PATH_CONF" \
-Des.distribution.flavor="$ES_DISTRIBUTION_FLAVOR" \
-Des.distribution.type="$ES_DISTRIBUTION_TYPE" \
-Des.bundled_jdk="$ES_BUNDLED_JDK" \
-cp "$ES_CLASSPATH" \
org.elasticsearch.bootstrap.Elasticsearch \
"$@"
{code}
This can be shelled out easily enough in Java, BUT we'd also need to make sure
it comes up before starting the test.
> Upgrade Elastic Search Version
> --
>
> Key: CONNECTORS-1639
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1639
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Elastic Search connector
>Reporter: Cihad Guzel
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.16
>
> Attachments: CONNECTORS-1639.diff,
> elastic-search-1.0.1-java11-build-error.log
>
>
> Current Elastic Search version is 1.0.1 . According to [this
> matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not
> supported by any ES version below 6.5.
> Besides, ES 1.x is no longer supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CONNECTORS-1639) Upgrade Elastic Search Version
[
https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086926#comment-17086926
]
Karl Wright edited comment on CONNECTORS-1639 at 4/19/20, 11:31 AM:
Unfortunately the dependencies for ElasticSearch Cluster Runner are huge. See:
https://mvnrepository.com/artifact/org.codelibs/elasticsearch-cluster-runner/7.6.2.0
This is a problem because we'd need to download all these dependencies and
their dependencies. But it may be that the cluster runner does not actually
use all these.
It may not even be the right thing to use. It doesn't seem like it would be
hard to write something that starts up a cluster based on a specific image.
[~michaelcizmar], if you were starting a cluster from a downloaded, unpacked
image, what steps would you take?
FWIW, if you want to try this, svn checkout
https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1639, and then
do: the following:
{code}
ant make-core-deps
ant build
cd connectors/elasticsearch
ant download-dependencies
{code}
The unpacked ES download, and the mapper attachments plugin, should live in
test-materials at that point.
was (Author: kwri...@metacarta.com):
Unfortunately the dependencies for ElasticSearch Cluster Runner are huge. See:
https://mvnrepository.com/artifact/org.codelibs/elasticsearch-cluster-runner/7.6.2.0
This is a problem because we'd need to download all these dependencies and
their dependencies. But it may be that the cluster runner does not actually
use all these.
It may not even be the right thing to use. It doesn't seem like it would be
hard to write something that starts up a cluster based on a specific image.
[~michaelcizmar], if you were starting a cluster from a downloaded, unpacked
image, what steps would you take?
(FWIW, if you want to try this, svn checkout
https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1639, and then
do: the following:
{code}
ant make-core-deps
ant build
cd connectors/elasticsearch
ant download-dependencies
{code}
The unpacked ES download, and the mapper attachments plugin, should live in
test-materials at that point.
> Upgrade Elastic Search Version
> --
>
> Key: CONNECTORS-1639
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1639
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Elastic Search connector
>Reporter: Cihad Guzel
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.16
>
> Attachments: CONNECTORS-1639.diff,
> elastic-search-1.0.1-java11-build-error.log
>
>
> Current Elastic Search version is 1.0.1 . According to [this
> matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not
> supported by any ES version below 6.5.
> Besides, ES 1.x is no longer supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (CONNECTORS-1639) Upgrade Elastic Search Version
[
https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086926#comment-17086926
]
Karl Wright edited comment on CONNECTORS-1639 at 4/19/20, 11:31 AM:
Unfortunately the dependencies for ElasticSearch Cluster Runner are huge. See:
https://mvnrepository.com/artifact/org.codelibs/elasticsearch-cluster-runner/7.6.2.0
This is a problem because we'd need to download all these dependencies and
their dependencies. But it may be that the cluster runner does not actually
use all these.
It may not even be the right thing to use. It doesn't seem like it would be
hard to write something that starts up a cluster based on a specific image.
[~michaelcizmar], if you were starting a cluster from a downloaded, unpacked
image, what steps would you take?
(FWIW, if you want to try this, svn checkout
https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1639, and then
do: the following:
{code}
ant make-core-deps
ant build
cd connectors/elasticsearch
ant download-dependencies
{code}
The unpacked ES download, and the mapper attachments plugin, should live in
test-materials at that point.
was (Author: kwri...@metacarta.com):
Unfortunately the dependencies for ElasticSearch Cluster Runner are huge. See:
https://mvnrepository.com/artifact/org.codelibs/elasticsearch-cluster-runner/7.6.2.0
> Upgrade Elastic Search Version
> --
>
> Key: CONNECTORS-1639
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1639
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Elastic Search connector
>Reporter: Cihad Guzel
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.16
>
> Attachments: CONNECTORS-1639.diff,
> elastic-search-1.0.1-java11-build-error.log
>
>
> Current Elastic Search version is 1.0.1 . According to [this
> matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not
> supported by any ES version below 6.5.
> Besides, ES 1.x is no longer supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1639) Upgrade Elastic Search Version
[ https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086926#comment-17086926 ] Karl Wright commented on CONNECTORS-1639: - Unfortunately the dependencies for ElasticSearch Cluster Runner are huge. See: https://mvnrepository.com/artifact/org.codelibs/elasticsearch-cluster-runner/7.6.2.0 > Upgrade Elastic Search Version > -- > > Key: CONNECTORS-1639 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1639 > Project: ManifoldCF > Issue Type: Improvement > Components: Elastic Search connector >Reporter: Cihad Guzel >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > Attachments: CONNECTORS-1639.diff, > elastic-search-1.0.1-java11-build-error.log > > > Current Elastic Search version is 1.0.1 . According to [this > matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not > supported by any ES version below 6.5. > Besides, ES 1.x is no longer supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1639) Upgrade Elastic Search Version
[ https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086902#comment-17086902 ] Karl Wright commented on CONNECTORS-1639: - Here are some examples of how to invoke these from code: https://www.programcreek.com/java-api-examples/?api=org.codelibs.elasticsearch.runner.ElasticsearchClusterRunner > Upgrade Elastic Search Version > -- > > Key: CONNECTORS-1639 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1639 > Project: ManifoldCF > Issue Type: Improvement > Components: Elastic Search connector >Reporter: Cihad Guzel >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > Attachments: CONNECTORS-1639.diff, > elastic-search-1.0.1-java11-build-error.log > > > Current Elastic Search version is 1.0.1 . According to [this > matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not > supported by any ES version below 6.5. > Besides, ES 1.x is no longer supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CONNECTORS-1639) Upgrade Elastic Search Version
[ https://issues.apache.org/jira/browse/CONNECTORS-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086901#comment-17086901 ] Karl Wright commented on CONNECTORS-1639: - I created a branch (https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1639) that updates the ant build to download ES 7.6.2 and mapper attachments version 3.1.2, and unpacks them. I hope these are compatible? But in any case, all that should be needed now is incorporating Elasticsearch Runner to start the instance programmatically, if that is how it works. Looking into that now. > Upgrade Elastic Search Version > -- > > Key: CONNECTORS-1639 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1639 > Project: ManifoldCF > Issue Type: Improvement > Components: Elastic Search connector >Reporter: Cihad Guzel >Assignee: Karl Wright >Priority: Major > Fix For: ManifoldCF 2.16 > > Attachments: CONNECTORS-1639.diff, > elastic-search-1.0.1-java11-build-error.log > > > Current Elastic Search version is 1.0.1 . According to [this > matrix|https://www.elastic.co/support/matrix#matrix_jvm], Java 11 is not > supported by any ES version below 6.5. > Besides, ES 1.x is no longer supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
