[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16507061#comment-16507061 ] Shawn Heisey commented on SOLR-6734: I know that logically, it makes a lot of sense to have the agent be the only piece that the system init handles. But then we have to answer the question of what we should do if a novice types 'service solr stop'. I think a novice would expect that this will only stop Solr, not everything. But if the agent is the only thing handled by the system, the pedantically correct thing to do would be to kill ZK, kill Solr, and kill the agent. If we were to name the service "solr8-agent" when the given service name during install is "solr8", that would eliminate the confusion, but I know I would not be happy to have an installation program use something different for the service name than EXACTLY what I told it to use. Another possibility, one that I like more as I imagine how it would work, is to configure multiple service names for the OS, but have them call the script/agent with different options. If the name given during install is solr8, then we create solr8, solr8-all, and solr8-zk, with only solr8-all configured for startup and shutdown. If the user types 'service solr8 stop' then ONLY Solr will be affected - the agent will stay running, and so will ZK if it is configured. Similarly, 'service solr8-zk stop' would only affect ZK, not the agent or Solr. For both solr8 and solr8-all, the start action will produce identical results -- any of the services that isn't running and is configured will be started. For solr8-zk, the start action will be slightly different. It will only take action if ZK is configured to run, and it will only concern itself with starting ZK and the agent, not Solr. A detail that may not be obvious from what I've said so far: I want to make it so that the user can use bin/solr directly for everything, including start/stop, even when Solr is installed as a service. Right now, if a user installs a service named 'solr7' with the installer script, I don't think they can use bin/solr directly for much of anything. If the service name is left at default, then bin/solr will probably work, because it is hard-coded to check for /etc/default/solr.in.sh. I have not verified that this is the case, but I believe it is. If there is only one service associated with the /opt/solr-X.Y.Z directory (such as the 'solr8' name mentioned before), then bin/solr will assume that service. So there needs to be a text file in the program directory that just lists names of services. If there are multiple services (where things like /opt/XXX and /opt/YYY are symlinked to that directory), and a service name wasn't provided, then the command would fail, and the error message would describe how to indicate which service to act on. It would also be really awesome if there was a service uninstall option. Not sure if it should automatically delete /opt/solr-X.Y.Z if the last service has been removed ... thinking maybe it shouldn't, but should output a message telling the user how to get rid of that last remnant if that's what they want to do. > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey >Priority: Major > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506261#comment-16506261 ] Shawn Heisey commented on SOLR-6734: bq. If you want the agent process (service) to be able to do e.g. a cluster-wide shutdown or rolling restart, would it not then need to listen to commands from the network? I did think about this, although admittedly it didn't occur to me early on. As I thought about your question just now, I was initially thinking about having that use the Solr API. But then I thought of something where it wouldn't work. Communicating cluster-wide operations through the Solr API would work for shutdown or restart. But if a full shutdown were executed, it would not be possible for one agent to start the whole cluster back up. Also, if a Solr node died, only the agent on that machine would be able to start it back up. So the answer to your question is yes, it must be able to listen to commands from the network. If we use a symmetrical cipher with the authentication password as a pre-shared key, that should take care of encryption and authentication in one step. without having to worry about keystores. And speaking of keystores ... when things progress to that point, I have a few thoughts about making life easier for users who want Solr to use https. > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey >Priority: Major > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506079#comment-16506079 ] Jan Høydahl commented on SOLR-6734: --- If you want the agent process (service) to be able to do e.g. a cluster-wide shutdown or rolling restart, would it not then need to listen to commands from the network? Or is your plan that such commands will be received by the main Solr APIs, that will in turn ask the local daemon to restart itself through a socket command on localhost? > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey >Priority: Major > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[
https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505714#comment-16505714
]
Shawn Heisey commented on SOLR-6734:
Putting a bit of code in a comment so that I don't lose track of it.
{code:java}
package foo;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;
public class Main {
private static final AtomicInteger threadCounter = new AtomicInteger(0);
public static void main(String[] args) throws InterruptedException {
Thread gce = new GCExerciser();
/*
* To see the difference made by the GC exerciser, run the program once
as-is,
* and then again with the following start() call commented.
*/
gce.start();
Thread pd = new PauseDetector();
pd.start();
}
/**
* Background thread to allocate a byte array every 10 milliseconds. This
serves
* to create garbage, so that there will be GC pauses.
*/
public static class GCExerciser extends Thread {
@Override
public void run() {
Thread.currentThread().setName("GCExerciser" +
threadCounter.incrementAndGet());
final long maxHeap = Runtime.getRuntime().maxMemory();
final long thirtyOneGig = 31L * 1024 * 1024 * 1024;
final int size;
if (maxHeap > thirtyOneGig) {
size = Integer.MAX_VALUE - 1;
} else {
size = (int) (maxHeap / 16);
}
byte[] b = null;
while (true) {
try {
Thread.sleep(10);
} catch (InterruptedException e) {
//
}
b = new byte[size];
@SuppressWarnings("unused")
int i = b.length;
}
}
}
public static class PauseDetector extends Thread {
private static final AtomicLong accumulatedNanos = new AtomicLong(0);
private static final int sleepMillis = 500;
private static final long sleepNanos =
TimeUnit.NANOSECONDS.convert(sleepMillis, TimeUnit.MILLISECONDS);
private static final long fiveMinuteNanos = TimeUnit.NANOSECONDS.convert(5,
TimeUnit.MINUTES);
@Override
public void run() {
Thread.currentThread().setName("PauseDetector" +
threadCounter.incrementAndGet());
Thread.currentThread().setPriority(Thread.MIN_PRIORITY);
long fiveMinuteStart = System.nanoTime();
while (true) {
long sleepStart = System.nanoTime();
try {
Thread.sleep(sleepMillis);
} catch (InterruptedException e) {
//
}
long sleepStop = System.nanoTime();
long sleepElapsed = sleepStop - sleepStart;
/*
* If elapsed time is more than the half second sleep, add the
difference to the
* accumulation counter.
*/
if (sleepElapsed > sleepNanos) {
accumulatedNanos.addAndGet(sleepElapsed - sleepNanos);
}
/*
* If total elapsed time is five minutes or longer, output the
accumulated time
* and reset everything.
*/
if ((sleepStop - fiveMinuteStart) >= fiveMinuteNanos) {
System.out.printf("Accumulated pause estimate for five minutes: %d
ms\n",
TimeUnit.MILLISECONDS.convert(accumulatedNanos.get(),
TimeUnit.NANOSECONDS));
fiveMinuteStart = System.nanoTime();
accumulatedNanos.set(0);
}
}
}
}
}
{code}
The PauseDetector thread in that code approximates what the jHiccup tool from
Azul does.
https://www.azul.com/products/open-source-tools/jhiccup-performance-tool/
With a version of this thread in Solr and any other services, it would be very
easy to detect and log severe application pauses, which typically are caused by
GC.
> Standalone solr as *two* applications -- Solr and a controlling agent
> -
>
> Key: SOLR-6734
> URL: https://issues.apache.org/jira/browse/SOLR-6734
> Project: Solr
> Issue Type: Sub-task
>Reporter: Shawn Heisey
>Priority: Major
>
> In a message to the dev list outlining reasons to switch from a webapp to a
> standalone app, Mark Miller included the idea of making Solr into two
> applications, rather than just one. There would be Solr itself, and an agent
> to control Solr.
> http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504722#comment-16504722 ] Shawn Heisey commented on SOLR-6734: bq. What do you mean by “service” here? Process or OS service/daemon? I'm referring to an OS service. Something in /etc/init.d, systemd, upstart, etc. Which I suppose also equates to a process, and each one would need to write a PID file. (Once the paradigm is nailed, come up with something similar for Windows) The only reason that I propose this for Solr and ZK, rather than having those be daemons under the sole control of the agent service, is to make it straightforward for anyone (the user OR the agent) to start/stop them. If the agent (node controller in your example) were the only service directly controlled by the OS and the others were controlled by the agent (probably by calling scripts, if Java's process control is braindead), I think that would work as well. If there's an open source framework for this that meets all the criteria, I'd be happy to look at it. The criteria I think it needs: 1) Has an acceptable license. 2) Is generic enough for our purposes. 3) Works on both POSIX platforms and Windows with minimal differences between the two. I'm developing a pretty clear mental picture of how all these pieces would work together, monitor each other, and report problems. It's a very 3-foot view, with a bunch of black boxes where the precise implementation hasn't been completely worked out, but the general mechanics seem doable. > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey >Priority: Major > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504404#comment-16504404 ] Jan Høydahl commented on SOLR-6734: --- What do you mean by “service” here? Process or OS service/daemon? I like the idea of evolving the control scripts+SolrCLI into a new binary (jar?) which can work one-off as today or optionally run in the background and then be able to monitor, restart etc other processes. Good old FAST ESP username to have exactly this, called the node controller (nctrl). You would simply say “nctrl start nctrl” and it would start itself as a daemon, looking at its config for what processes to start. Then you could use the same binary to control the controller “nctrl restart qrserver”. And of course it took commands from the GUI. The only process running as an OS service was the node controller. Perhaps such software exists in open source already, flexible enough to use? > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey >Priority: Major > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504164#comment-16504164 ] Shawn Heisey commented on SOLR-6734: One of the things that I would very much like to have is the ability in the admin UI to change things like heap size and restart Solr. One of the challenges around that is that the admin UI is part of Solr itself, so although it would be straightforward for Solr to stop itself, there must be a mechanism that starts Solr back up. Another challenge is related to how process forking works. On Linux, and probably other systems as well, when a process forks to spawn another process, most aspects of the original process are inherited by the new process. One of those aspects is memory allocation. So if we embed the daemon into Solr itself and a new process is required, and Solr is allocating 8GB of memory from the system, the new process will ALSO have 8GB of memory allocated. It would be free to shed that memory, but it's my understanding that it would all be allocated before the process would be able to drop it. On systems with limited resources, this could be a serious problem. As much as it might sound like a good idea, embedding the agent daemon into Solr isn't viable. So here's an idea that I'm starting to like, independent from any sort of cluster auto-discovery: Let's install (up to) three services. One for Solr, one for the agent, and one for ZK, for installs where an external ensemble is not provided. ZK would be handled through a small wrapper Main class that incorporates whatever IPC we settle on, whether it's a network socket on localhost or something else. Solr and the agent would also include the same IPC capability. Because they're all separate services, they can be started independently from each other. The agent would be capable of stopping or starting the other two services. The agent would have a very small heap, and might handle process control by running system commands or included scripts. Not 100% sure about monitoring, but some of it might be in the agent and some of it might be through external scripts. The IPC capability in all three services can also serve as a monitor point. The agent would absorb SolrCLI functionality and handle complex control script commands in a non-daemon mode, communicating with the services as required. > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey >Priority: Major > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16503760#comment-16503760 ] Shawn Heisey commented on SOLR-6734: bq. If the agent is capable of reconfiguring and restarting a node, it needs to be secured. That's easy enough. Only make it accessible via loopback/localhost. I had not imagined it as something that would be accessed remotely. If there's another viable method of interprocess communication available, we wouldn't even need a network socket. The only caveat is that I'd very much prefer a cross-platform implementation. If it *is* opened up beyond local access, then you're absolutely right -- authentication would be required, and probably should NOT have a default setting. Authentication would not be a bad plan even with local-only access. Jetty's stop mechanism requires it and is not remotely accessible. bq. How would the agent start the real Solr and monitor it? That's an excellent question. I have general thoughts, but they're mostly untested. If process monitoring/control is too braindead in Java, which would not surprise me at all, I have some other ideas. I will write a separate comment to discuss the various pros and cons of those ideas. bq. Again, I think we try to reinvent what should perhaps instead be an effort to improve official ansible/puppet modules for Solr and ease production-grade container-based installs. One of the principles driving me on this issue and its parent is to make things easier for someone to get a reasonable production installation going, using only the Solr download and nothing else. I'm not saying that we shouldn't be involved in designing or writing modules for deployment tools. But I don't think we should expect the casual novice to use those tools. Also, writing those modules will be easier if we have bulletproof installation, startup, and configuration. bq. Multicast is so 1990 and not supported at all in many environments, so you'd need to build separate discovery protocols to choose from (like ES) - and at the end of the day you'll probably end up with recommending stable hard-wired ZK_HOSTs for production. Another case where you might be completely right. And I would never want to take away anyone's ability to hard-wire their ZK hosts and/or use a completely external ensemble. The idea that had me think of multicast is automated setup in a LAN environment. Users install Solr on three nodes, type some *simple* commands, and the machines find each other. Solr detects details for each server like hostnames, IP addresses, ports (for both ZK and Solr), and possibly other things, all automatically. The user can override what's detected if they want to, and then one more simple command executed on any node finalizes the config to create a fully operational cluster. It is true that some environments will not allow multicast communication to work. Anything involving multiple IP subnets would be unable to use it without some significant network infrastructure configuration. But I think *most* environments where the Solr servers are all on the same subnet would work without issues, and I don't think the code to do it would be supremely difficult. One of the complaints I hear about most with mutlicast-based technologies is an abundance of traffic on the network, to the point where network links are so overloaded that other traffic gets dropped. The traffic levels here should be quite low, and with proper usage, will not bleed to unexpected places in the network. The cluster-building stuff would need a way to manually specify the information that the automatic detection would gather, in case the automatic detection doesn't work or the user doesn't want to use it. A mandatory key/password with no default would have to be part of starting the automatic detection on each node. > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey >Priority: Major > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[
https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16503362#comment-16503362
]
Jan Høydahl commented on SOLR-6734:
---
I'm not convinced we should go down that path, adding a whole new level of code
and network transport that needs its own discovery, protocols, failure
handling, security, monitoring and what not. Kind of like the simplicity of
Solr being a single process:)
Questions:
* How would the agent start the real Solr and monitor it? Native (child)
process control is quite challenging and lacking in Java. Add to that
service/daemon as a third process level.
* Multicast is so 1990 :) and not supported at all in many environments, so
you'd need to build separate discovery protocols to choose from (like ES) - and
at the end of the day you'll probably end up with recommending stable
hard-wired ZK_HOSTs for production.
* If the agent is capable of reconfiguring and restarting a node, it needs to
be secured. So all these new socket commands will need to be secured with TLS,
authentication and authorization from day one, then the agent is not so slim
anymore?
Again, I think we try to reinvent what should perhaps instead be an effort to
improve official ansible/puppet modules for Solr and ease production-grade
container-based installs.
We could ease part of the Zookeeper ensemble setup pain by shipping ZK with
Solr as we already do, but also offer a start script, e.g. {{bin/zk start -z
zoo1:2181,zoo2:2181,zoo3:2181}} to configure and start Zookeeper on a node.
With zk 3.5 we'll get more dynamic reconfig options too, which will perhaps
make it possible to go from 1 to 3 zk nodes by simply starting ZK on more
nodes...
> Standalone solr as *two* applications -- Solr and a controlling agent
> -
>
> Key: SOLR-6734
> URL: https://issues.apache.org/jira/browse/SOLR-6734
> Project: Solr
> Issue Type: Sub-task
>Reporter: Shawn Heisey
>Priority: Major
>
> In a message to the dev list outlining reasons to switch from a webapp to a
> standalone app, Mark Miller included the idea of making Solr into two
> applications, rather than just one. There would be Solr itself, and an agent
> to control Solr.
> http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501760#comment-16501760 ] Shawn Heisey commented on SOLR-6734: I've been thinking about how an agent might facilitate setting up a fault tolerant SolrCloud, complete with ZK servers. Users would start out by running a command like "bin/solr cloudbuild start" on one node that has been installed as a service. This would start or reconfigure the agent in a mode where it can begin communicating with other agents. Then another command or set of commands would be run on that first node to set it up with necessary services for the cluster. If multicast communication can be written in Java with relatively simple code, that would allow the agents to communicate with each other on LAN subnet and form a SolrCloud cluster without explicit address/hostname configuration. If somebody got really ambitious and configured multicast routing, they could even do it on different subnets or across datacenters. I envision using 239.89.83.0 as a default multicast address. UDP port 8983 seems fitting to use as a default. As each node is configured, with additional "bin/solr cloudbuild" commands, it will begin sending periodic status packets on the configured multicast group, which every node will listen to and build a whole picture of the cluster. Once there are at least three nodes configured, another command, perhaps "bin/solr cloudbuild build", can be called. If the overall cluster config passes validation, this will signal each node to write its configuration for ZK and Solr, start (or restart) the services, and turn off cloudbuild mode. If everything goes well, the user will have a functional fault-tolerant SolrCloud install, without any need to worry about how to install ZK as a separate service. I have most of this idea mapped out in my head at a high level. It will require fleshing out to realize as usable code. > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey >Priority: Major > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482225#comment-16482225 ] Shawn Heisey commented on SOLR-6734: [~markrmil...@gmail.com], I do not know if I'm having trouble parsing what you're saying because it's late and I'm exhausted, or if I'm experiencing a fundamental disconnect. Let me try to give you a 3-foot view of control script operation in the new world I'm envisioning. I'm going to describe a posix platform here, but for Windows, the general idea would be similar. * If running as a service, somehow get the service name. Lots of options for exactly how this is done. * The first primary job of the control script will be locating Java using whatever mechanisms make sense. Once it's found, run a little Java class whose entire job is looking at the vendor, version, and other characteristics of the JVM and the operating system. If the found JVM version has known issues, or if configurations are found that could be problematic, it would display a message alerting the user to those problems. The exit code of that program could be used to configure workarounds or abort script operation. * Use the service name if present to look somewhere central, such as /etc/solr/servicename. Otherwise look somewhere in the extracted directory structure, possibly $INSTALL_DIR/etc. Find the primary config there. Start the agent with the primary config and the service name. * The agent will be responsible for starting Solr using the information in the primary config. Solr will be responsible for loading the secondary config, either from ZK or the same location as the primary config. The secondary config would be a likely location for defining the solr home and similar settings. Then Solr would start the indexes. * In this brave new world, we no longer have solr.xml. Its functionality would be handled by the secondary config. The control script will hand off to the agent or SolrCLI for much of its functionality. It will be much less complex than before, so the differences between shell and batch languages won't be as much of a burden. When there are installed services, there should be something written in the install directory so that running bin/solr manually can know that services have been installed. If there's only one installed service, it should use that for its operation, with an optional CLI parameter to ignore services. If multiple services are installed and nothing was provided to indicate which service to use, the script should end early with a message describing how to tell it which service to talk to or how to ignore services. Having a Solr-specific plugin for systems like Kubernetes does sound like a good idea, but it should be reliant on our control infrastructure, not the other way around. I would like us to have a Windows service installer in addition to the shell script for other platforms. That will require research and experimentation. > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey >Priority: Major > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482170#comment-16482170 ] Mark Miller commented on SOLR-6734: --- I think the Solr start scripts we were able to add addressed a lot of what I was talking about. They also could easily drive anything we are still missing right? We don't yet have good start and stop cluster support, but I imagine we would add support for a simple version via the existing scripts. Same as if we added a monitoring process. So I think we have a lot of what I was talking about - I still believe we need to finish it as well though. Simple starting and stopping of the cluster should not require something like kubernetes. We want that very basic functionality out of the box I think. Now that we have the start scripts and are not constrained as a war, seems like we can do it relatively easily. Because the start scripts have so much outside of Java and require duplicating so much for Windows, not my favorite area to work though. > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey >Priority: Major > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482034#comment-16482034 ] Shawn Heisey commented on SOLR-6734: Here's an attempt to solidify some things: * I like the socket idea. Here's some things that I think the agent will need if we choose sockets for IPC: ** A port number. Solr port minus 500 as the default, maybe? ** Socket command to stop Solr. ** Socket command to start Solr. ** Socket command to get Solr status. ** Possibly similar operations for ZK. ** Socket command to stop itself (and any managed processes). * I think the agent should use SolrJ jars (and dependencies). Mostly for monitoring. * The control script will only start/stop Solr indirectly. It will be responsible for making sure the agent is running or stopped as required. * The other things the control script does (creating cores/collections, etc) probably do not need adjustment. After making sure the agent and its controlled processes are running, those operations would continue to be handled through SolrCLI. * If basic new-style configurations aren't found, then operations with the control script will fail fast. If old configurations are found, then the output should reference instructions for config conversion. Conversion should be easy and automated, but always initiated manually so the user knows what is being done. Whats below might belong on SOLR-6733, but dovetails with the above thoughts, so I'm putting it here: Probably the simplest way to handle agent configuration is a properties file, but if we load everything needed for SolrJ, we will have noggit. I'm really liking the notion of using one format for ALL Solr config files, and I think json has the most going for it in that goal. It can do almost as much as XML. The files are smaller than XML and easier for a newbie to grasp. I see that noggit supports comments, so that eliminates the one significant drawback that I see with the official json standard compared to other formats. Given the ubiquitous nature of properties files, I am not opposed to their use for things where the configuration is simple, even if we settle on another format like json as a standard. Particularly where a user wanting typical operation will never need to worry about them -- which describes core.properties. > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey >Priority: Major > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16478313#comment-16478313 ] Shawn Heisey commented on SOLR-6734: Further thought: The agent wouldn't need to be responsible for making config changes. It's probably better to have that handled in SolrCLI/Solr, for two reasons that come to mind immediately: 1) The user interface for these things would be there, not in the agent. 2) So the number of dependencies the agent requires can be as small as possible. I'd be willing to bet that the main Solr app already has all the dependencies we are likely to want for creating and manipulating config files. I do think it might be a good idea to have some limited monitoring capability in the agent. If the agent detects that the Solr process has died or isn't responsive on one of the /admin/system API calls that should complete quickly (after working correctly for a certain amount of time after startup), it would be fairly easy to have it send an alarm email, if the user configures that. The jar for that capability is only 213KB. https://mvnrepository.com/artifact/javax.mail/javax.mail-api/1.6.1 > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey >Priority: Major > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1643#comment-1643 ] Shawn Heisey commented on SOLR-6734: I am imagining a VERY slim process, with dependencies as minimal as possible. If it's Java, which I see as likely, then it should be able to run with a very small heap. Logging could be done with stdout, but we might want to have it use slf4 since those jars are already present. I see the primary job of the agent as configuring/starting/stopping the main Solr process, as you mentioned. We can think about other capabilities as the project evolves. With effective and cross-platform inter-process communication (IPC), the agent wouldn't need any kind of direct user interaction, which would keep its footprint small. All user interaction for the agent could be handled by SolrCLI and Solr itself (which are already set up to handle user input), sending messages to the agent. It looks like Java has a socket implementation that's not difficult to use. > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey >Priority: Major > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[
https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477522#comment-16477522
]
Jan Høydahl commented on SOLR-6734:
---
I think the original issue and email thread is very old, and mostly talking
about things that have already been solved or will be solved if we replace
start.jar with solr-start.jar and take control over more ourselves.
But are you envisioning the agent as a super slim process which is what is
really being started with {{bin/solr start}}, and then its only task is to
spawn the main Solr JVM with correct settings, and receive agent commands
(REST?) for restart, reconfiguring etc? Or do you think it should be a fatter
agent process which users need to relate to in terms of separate commands to
start, stop, configure etc? It is the latter situation I'd like to avoid by
keeping Solr as one process.
> Standalone solr as *two* applications -- Solr and a controlling agent
> -
>
> Key: SOLR-6734
> URL: https://issues.apache.org/jira/browse/SOLR-6734
> Project: Solr
> Issue Type: Sub-task
>Reporter: Shawn Heisey
>Priority: Major
>
> In a message to the dev list outlining reasons to switch from a webapp to a
> standalone app, Mark Miller included the idea of making Solr into two
> applications, rather than just one. There would be Solr itself, and an agent
> to control Solr.
> http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477345#comment-16477345 ] Shawn Heisey commented on SOLR-6734: I have no experience with that framework. The plugin for Kubernetes does sound like it would be useful, and something we can put some time into. I do still think it would be useful to have an agent app, though. It would allow the casual user (somebody who downloads Solr and runs it directly) to access an admin web page (probably just the Solr admin UI that already exists) where they can change startup options like the heap size and restart Solr. > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey >Priority: Major > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477256#comment-16477256 ] Jan Høydahl commented on SOLR-6734: --- I'm arguing back and forth with myself on this. While it gives easier cluster setups since we ship ZK, it also re-invents some of the orchestration tasks that frameworks like Kubernetes, Swarm & Mesos were built to handle. Like "Where do my zookeepers run?" and "Node A with ZK died, should we start it on node B?". I also realise that most people will not consider K8s for small clusters due to the (current) complexity of installing and overhead, but looking 3-5 years into the crystal ball I can see more and more users wanting to run their search cluster containerized and freely choose whether to deploy in a public cloud, a private cloud or on-premise. Also in that time frame K8s will probably be mature for small local DC installs, and your average hosting provider will offer container services. So my question is whether it makes sense to instead focus our efforts into lowering the bar for running Solr on Kubernetes, and aim to promote that as an officially supported environment and later perhaps the recommended one? This could translate into developing a high-quality and feature rich [Kubernetes Operator|https://coreos.com/operators/] for Solr. An Operator is a program that knows the application well and can support high-level cluster operations such as "install a 3x2 node cluster with 3 ZKs", or "do a rolling upgrade from SolrX to SolrY", or "Spin up more nodes / containers / pods if Solr' Metric X is above threshold Y for more than Z minutes". > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey >Priority: Major > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16476466#comment-16476466 ] Shawn Heisey commented on SOLR-6734: The dynamic reconfig capability in ZK 3.5 could perhaps be used by an agent process to help build ensembles in a way that's better automated than can happen currently. Have to look into it and ask dumb questions on the ZK mailing list! > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey >Priority: Major > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14320165#comment-14320165 ] Shawn Heisey commented on SOLR-6734: Having both zookeeper and Solr started and monitored by an agent process would be awesome. An ideal SolrCloud environment could be built and maintained by a novice with simple instructions and scripts that we provide -- one of the hardest things for a new SolrCloud user to grasp is setting up a zookeeper ensemble. IMHO, the agent should have the ability to do email notifications with smtp server, subject, and envelope from/to controlled by configuration. I have some experience with Oracle's JavaMail API. > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14320114#comment-14320114 ] Shawn Heisey commented on SOLR-6734: I don't know that it's NEEDED, but it sure would be nice. For cloud mode, storing config and state in zookeeper sounds like a clear win. Are there any serious plans to require zookeeper and always run in cloud mode? I know it's something that has been discussed, I just don't know where that landed. If non-cloud mode won't be going away, I think it should also benefit from this idea. > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319759#comment-14319759 ] Jan Høydahl commented on SOLR-6734: --- Is this still needed? If this was a "node" controller which monitors the JVM process for crashes/freeze/OOM and restart it, that would be valuable. Also, if the agent could configure itself from ZK and dynamically know what services to stop/start on that particular host, and with what startup parameters, then we could use ZK as a cluster topology truth, to decide on which nodes to automatically start ZK, Solr and any other app needed... > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211832#comment-14211832 ] Shawn Heisey commented on SOLR-6734: I thought of an alternative to the TCP port for agent communication -- a work queue in zookeeper like the SolrCloud overseer. A separate queue for each agent might be cleaner, but I don't know what the performance implications would be when scaling to a large number of servers. > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209177#comment-14209177 ] Shawn Heisey commented on SOLR-6734: The "chicken and egg" repercussions of that last bullet point might make your head spin. > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6734) Standalone solr as *two* applications -- Solr and a controlling agent
[ https://issues.apache.org/jira/browse/SOLR-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209174#comment-14209174 ] Shawn Heisey commented on SOLR-6734: Some additional detail I thought of for this idea: * The agent could listen on a TCP port for connections from agents running on other servers. ** This would be primarily for SolrCloud, but there might be non-cloud uses too. This would enable stopping and starting of Solr instances across an entire cluster. ** Not entirely sure about how to configure an entire cluster of agents ... perhaps like zookeeper, where all servers contain the entire list of host:port pairs. A centralized config in Zookeeper would not be a bad idea either, as long as we have some way of altering or resetting that config. ** A shared authentication key and a cluster name in the config would be a good idea. The info in an incoming request would need to validate against both, and perhaps even against the host:port list. This data might also be used for TLS. * The heap set by the script that starts the agent would be very small. Even with hundreds of servers in the config, I would imagine that the memory requirements would be minimal. * Most startup options for Solr would be configurable and used by the agent when starting Solr. Thinking about a cluster, we probably would want to have a common set of options as well as per-server options that can supplement or override common options. * We might even be able to control standalone zookeeper processes with this agent. > Standalone solr as *two* applications -- Solr and a controlling agent > - > > Key: SOLR-6734 > URL: https://issues.apache.org/jira/browse/SOLR-6734 > Project: Solr > Issue Type: Sub-task >Reporter: Shawn Heisey > > In a message to the dev list outlining reasons to switch from a webapp to a > standalone app, Mark Miller included the idea of making Solr into two > applications, rather than just one. There would be Solr itself, and an agent > to control Solr. > http://mail-archives.apache.org/mod_mbox/lucene-dev/201305.mbox/%3C807476C6-E4C3-4E7E-9F67-2BECB63990DE%40gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
