Hi Rick - I have tried to wrap my head around what you are telling me here but must admit that I am falling short. Sorry for being thick headed on this one.
It seems that there may be multiple issues stemming from the fact that SOLR (like more ecosystem services than I like) doesn't support the trusted proxy pattern that is at the heart of the Knox dispatch. There are occasions where certain UIs service definitions are being committed to Knox when there is no trusted proxy support and these UIs require API service defs as well. In these circumstances, the assumption is that the API use is limited to that of the corresponding UI. I thought this was the case for SOLR. If it did support the trusted proxy pattern then the SOLR API would accept the asserted identity from such a proxy rather than require HTTP Basic credentials. Rather than having to try and adapt clients to fit these types of things - it may be better to implemented trusted proxy support along with kerberos on the server side. For this release, it seems that we don't really need to take any action here if the API is assumed to really only be used by the UI. If I am way off on this then I think I need to have the following statement made a little more clear: "At some point in the future I think we might want to have Java REST clients or detailed documentation for many of the well-known technologies if any changes are not accepted into their baselines or necessary to use earlier versions." I think that I just missing some additional context for that statement. thanks, --larry On Fri, Dec 1, 2017 at 8:48 PM, Rick Kellogg <rmkell...@comcast.net> wrote: > Larry, > > I added this ticket more as a form of documentation. The attachment shows > how to work around the issue. So not really a patch for the Knox project. > > At some point in the future I think we might want to have Java REST > clients or detailed documentation for many of the well-known technologies > if any changes are not accepted into their baselines or necessary to use > earlier versions. > > When adding SOLR support to the Knox User Guide I did document the issue > with POST operations. You might want to list the JIRA ticket along with > the caveat. > > Thanks, > Rick > > -----Original Message----- > From: larry mccay [mwanailto:lmc...@apache.org] > Sent: Friday, December 1, 2017 8:17 PM > To: dev@knox.apache.org > Subject: Re: Ambari Enhancements / Knox Clients > > I don't know why that isn't showing up in the 0.14.0 filter that I have > been using to close down the release. > Is there a reason that this JIRA doesn't have an actual patch with tests? > > If you need this in then we should try and get it complete in the next > couple days. > > On Fri, Dec 1, 2017 at 5:43 PM, Rick Kellogg <rmkell...@comcast.net> > wrote: > > > Kevin, > > > > The only ticket is one I filed against the Knox project ( > > https://issues.apache.org/jira/browse/KNOX-1066). Within the standard > > SOLRJ code it explicitly does not allow retry operations to prevent > > errors with duplicate data. This impact SOLR 6.x for certain. Have > > not tested with others. A patch is provided. > > > > Good luck! > > Rick > > > > -----Original Message----- > > From: Kevin Risden [mailto:kris...@apache.org] > > Sent: Friday, December 1, 2017 2:29 PM > > To: dev@knox.apache.org > > Subject: Re: Ambari Enhancements / Knox Clients > > > > > I discovered many months ago that just preemptive > > > basic-authentication > > and SOLRJ required me to write custom code. The standard SOLRJ client > > will not send the authentication header when performing a PUT operation. > > As such update operations via Knox would fail. > > > > Rick - Is there a Solr JIRA open for the above issue? Is this with a > > certain version of Solr? > > > > Kevin Risden > > > > On Fri, Dec 1, 2017 at 12:57 PM, Rick Kellogg <rmkell...@comcast.net> > > wrote: > > > > > Correct no negative comments. The changes sound very useful. I do > > > wish the Ambari support was stronger but documenting usage in Wiki > > > is a suitable compromise. I will try to coordinate some testing > > > with my Ops team time permitting. > > > > > > The third party clients I am referring to are the standard Java > > > clients (HBase HRemoteTable, SOLR SOLRJ/SolrClient) used by outside > > clients like > > > Tomcat. Had not even thought of the Knox Shell. Our job is not > > complete > > > if we abandon the Client API developers are accustomed to using. > > > > > > In reference to the Lite HBase REST Client I am creating, I fully > > > support the low level Google Protocol buffers. One item we as a > > > community have not addressed is the traditional consumer of a > > > service via Knox. I discovered many months ago that just preemptive > > > basic-authentication and SOLRJ required me to write custom code. > > > The standard SOLRJ client will not send the authentication header > > > when performing a PUT operation. As such update operations via Knox > > > would > > fail. > > > > > > When I started the HBase work, I used the HRemoteTable class. After > > > some discussion with their development team, they stated it was > > > really only there for testing purposes. Hence the creation of my > > > new project. My main goals were: minimize third party dependencies, > > > maintain yet simplify existing API compatibility plus Kerberos > > > support. At some point, we might do a complete redesign of the API. > > > > > > Hope that clarifies things. Take care, Rick > > > > > > -----Original Message----- > > > From: larry mccay [mailto:lmc...@apache.org] Sent: Friday, December > 1, > > > 2017 9:23 AM > > > To: dev@knox.apache.org > > > Subject: Re: Ambari Enhancements / Knox Clients > > > > > > Overall, it seems that you don't have any outstanding concerns - right? > > > > > > inline... > > > > > > On Fri, Dec 1, 2017 at 9:00 AM, Rick Kellogg <rmkell...@comcast.net> > > > wrote: > > > > > > > Greetings Larry, > > > > > > > > As always thanks for the community's good work and collaborative > > support. > > > > > > > > I have been following the traffic on Knox improvements for > > > > sometime but have not built and used the interim builds. All the > > > > deployment topology improvements sound very useful in production. > > > > To fully close the loop for our operations team, the integration > > > > with Ambari is going to prove critical. Many tools we use > > > > (Confluent Kafka REST, Livy, > > > > etc) require manual installation and then complex configuration. > > > > With these recent addition to Knox and future improvements to > > > > Ambari, it would be great to declaratively state which services to > > > > proxy and have automatic host detection with high > > > > availability/failover. Essentially a complete end-to-end solution > > > > with a low rate of failure opportunity for our operations teams. > > > > > > > > > > > I think you are saying that the changes look good. :) I agree that > > > the Ambari based service discovery is going to be a huge win from an > > > Ops perspective. > > > > > > > > > > So in the interim, I would ask that changes necessary to support > > > > automatic deployment/registration of Ambari managed services be > > > > sufficiently documented in the Knox. That way existing topologies > > > > could be manually setup or a configuration is set which uses > > > > Ambari as a registry. I just don't want all the good work done to > > > > be overshadowed by inadequate documentation. > > > > > > > > > > > I would like to get some wiki pages together for testing this > > > end-to-end for the upcoming RC - it would be amazing if someone from > > > your ops team could try it out. > > > Such folks are so important to our community. > > > > > > > > > > Personally, I have been focused on the Knox to client side of things. > > > > This includes creating a thin HBase REST client out of their > baseline. > > > > In the near future, we should have Kerberos support from the HBASE > > > > REST Client API as well. This work is being tracked in > > > > HBASE-19207 ( https://issues.apache.org/jira/browse/HBASE-19207). > > > > The project can be found on Github: > > > > https://github.com/rmkellogg/hbase-lite-rest-client > > > . > > > > After which I will be looking at tackling Kerberos support from > > > > the SOLRJ API as well. > > > > > > > > > > > Hmmm... what do you mean by "Knox to client side of things"? > > > That client looks nice - is it at all related to the KnoxShell hbase > > > stuff > > > - doesn't seem like it but it looks similar? > > > If not, perhaps we can look at some consolidation where our overall > > > framework can leverage your API? > > > Are you handing the base64 decoding bit for it - that is so awkward > > > to deal with for REST? > > > > > > Take care, > > > > Rick > > > > > > > > -----Original Message----- > > > > From: larry mccay [mailto:lmc...@apache.org] > > > > Sent: Thursday, November 30, 2017 4:52 PM > > > > To: dev@knox.apache.org > > > > Subject: Re: Ambari Enhancements > > > > > > > > Hi Rick - > > > > > > > > Good to hear from you! > > > > > > > > I don't think that there is any danger of getting out of sync with > > > > Ambari expectations here but I may be missing something. > > > > > > > > Let me try and summarize the work being done which is based on > > > > KIP-8 [1] and you can tell me what your concerns are... > > > > > > > > 1. We will continue to support the native topology file which is > > > > what Ambari is doing 2. We are introducing a simplified descriptor > > > > and shared provider config mechanism that results in generated > > > > topologies that Ambari will be unaware of in the near term 3. The > > > > topologies are generated by using the Ambari API to discover the > > > > service details which obviously is compliant with the current > > > > Ambari REST API but Ambari itself is unaware of it 4. The > > > > descriptors and provider config will be able to be monitored in > > > > Zookeeper as well as local file system Ambari will remain inline > > > > with local default.xml, knoxsso.xml and admin.xml topologies and > > > > unaware of ZK based configs for now 5. Ambari improvements are > > > > planned to be able to manage the descriptors and provider config > > > > and to push them to ZK when available otherwise to local disk 6. > > > > Editing topologies that are generated from descriptors would be > > > > problematic and lead to confusing when they get overwritten upon > > > > changes discovered in the descriptors or service details via > > > > Ambari REST API. So Sandeep is making the generated topologies > > > > readonly in the UI with an indication as to why. Ambari is > > > > completely > > > unaware of the Knox Admin-UI currently and will likely remain so. > > > > > > > > Does the above address your concerns and/or can you share what > > > > your concerns are? > > > > > > > > thanks! > > > > > > > > --larry > > > > > > > > 1. > > > > https://cwiki.apache.org/confluence/display/KNOX/KIP-8+ > > > > Service+Discovery+and+Topology+Generation > > > > > > > > > > > > On Thu, Nov 30, 2017 at 3:34 PM, Rick Kellogg > > > > <rmkell...@comcast.net> > > > > wrote: > > > > > > > > > Greetings, > > > > > > > > > > > > > > > > > > > > Has there been any enhancements made or even suggested to the > > > > > Ambari > > > > team? > > > > > > > > > > > > > > > > > > > > I see a lot of work being done on the Knox internally and to the > > > > > Knox Admin UI and have some concerns that Ambari v2.6.0+ might > > > > > not play well with all the new enhancements. > > > > > > > > > > > > > > > > > > > > Just my two cents, > > > > > > > > > > Rick > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
