Thanks both of you. No, these changes would benefit me and this is my use case but we would not need to add anything parcel or cloudera specific. And I think that while I have a specific need they are useful in general as well.
It would probably be an overhaul of gateway.sh, knox-env.sh and knoxcli.sh. Maybe a few changes to Launcher. On Thu, Oct 11, 2018, 16:50 larry mccay <[email protected]> wrote: > This sounds like a worthwhile piece of work, Lars. > Would the "parcel" need to be added to the Knox project? > > +1 to Phil's response. > > > On Thu, Oct 11, 2018 at 9:33 AM Phil Zampino <[email protected]> wrote: > > > Cloudera specifics aside, some of these things have been on my personal > > "back burner" todo list; I just haven't as of yet been able to bring them > > to the front burner ;-) > > > > Please feel free to create issues for these, and provide patches as > you're > > able. > > > > Thanks for this useful feedback, > > Phil > > > > > > On Thu, Oct 11, 2018 at 8:43 AM Lars Francke <[email protected]> > > wrote: > > > > > Hi again, > > > > > > (long intro, jump to the line with the XXXXX if you're not interested > in > > > the background) > > > > > > I'm sorry for all the mails, they all come from my quest to get Knox > > > packaged up as a Cloudera Parcel plus a CSD (Custom Service Descriptor) > > to > > > run it from Cloudera Manager (CM) similar to what Ambari currently > > allows. > > > I did the same for NiFi and was facing similar issues there (e.g. < > > > https://issues.apache.org/jira/browse/NIFI-5350>, < > > > https://issues.apache.org/jira/browse/NIFI-5573> and others). > > > > > > For people unfamiliar with Cloudera Manager I'll explain how it works. > > That > > > should make it clearer why I have the issues I describe below. > > > > > > Cloudera Manager extracts the "things" it manages into a directory > (e.g. > > > /opt/cloudera/parcels/KNOX) and they are owned by root:root. This is > not > > to > > > be changed by any process (e.g. no configuration file changes, no > > changing > > > of symlinks, no storing of PIDs, logs etc.). > > > > > > Every time a process (e.g. Knox Gateway) is started CM creates a new > > > directory (/var/run/cloudera-scm-agent/process/XXX) where it > > copies/creates > > > the necessary config files + keytabs for _this run_ of the tool. > > > > > > It then starts the processes by pointing them at this directory, so > they > > > can pick up their config there and it also captures stdout & stderr in > > this > > > folder. > > > > > > This is different from Ambari. Ambari extracts Knox and creates > symlinks > > > from its conf, logs, pids and data directory to /etc/XXX. This is > > possible > > > here because those directories (/etc/...) don't change. > > > > > > > > > XXXXXXXXX > > > > > > > > > The problems I have with Knox so far (I'm sure I'll find more the > > further I > > > get) are: > > > > > > * gateway.sh has no way to take in options from the "outside". With > > Hadoop, > > > HBase, (now) NiFi you can pass in arbitrary Java options using > variables > > > like HADOOP_JAVA_OPTS and similar. > > > > > > In theory all the "setup" is already there for Knox as well using > > variables > > > like APP_CONF_DIR but unfortunately, they get set to hardcoded values > at > > > the beginning of the script. > > > > > > Proposal: Add at least a APP_JAVA_OPTS variable so I can pass in > > arbitrary > > > stuff to be added to the Java command line. But really, I'd love to > just > > > remove the defaults for APP_LOG_DIR etc. IFF they are already set > > > externally > > > > > > * gateway.sh checks whether various directories exist. These are > > hardcoded > > > (e.g. APP_HOME_DIR/conf). But those directories are configurable using > > > GATEWAY_HOME etc. so those checks should either be removed or fixed, so > > > they take those variables into account > > > > > > * knoxcli create-master takes a --master argument which I only found > out > > by > > > looking at Ambari. The source says it's for testing only. It seems as > if > > > that should be documented though. I think it's pretty useful to allow > the > > > master being created non-interactively > > > > > > * gateway.sh does allow one thing to be overridden externally and that > is > > > the pid dir using ENV_PID_DIR. Unfortunately, knox-env.sh (which is > being > > > sourced unconditionally) overrides this variable with an empty value. I > > > think this line should just be removed from knox-env.sh > > > > > > * Launcher looks for a file called gateway.cfg but it always and > > > unconditionally looks in its "own" directory (launcherDir). I need a > way > > to > > > point this to a different location. It allows me to define GATEWAY_HOME > > as > > > a system property. While I can also define that as an environment > > variable > > > the System property is checked first. And if it finds a > gateway-site.xml > > > there it uses that. I need it to use the one from the environment > > variable. > > > > > > * gateway.sh allows the process to run in the foreground but still > > captures > > > stdout & stderr to files. I would argue that it makes more sense to > leave > > > them as is and print them to the console instead. > > > > > > I'm happy to create issues for all of these and also provide patches > for > > > some/all of them depending on my available time. I just wanted to bring > > > this up before I started to see if anyone has any better ideas and/or > > > things that I might have missed. > > > > > > Thanks for reading! > > > > > > Cheers, > > > Lars > > > > > >
