Hi David, Kevin,
I have reached out to the spark users community to see if their are
alternatives or any work going on to fix this issue. One other thing I am
trying to test out is the limiting the request header sent from Knox to the
backend services, like spark history ui.
The main reason this issue appear on our end is the large number of
groups the user belongs to that is pushing the request header to go beyond
8192. Those groups are not needed in the backend services and I am looking
to see if there is a way for Knox to exclude them when forwarding the
request to the backend?
Best,
Theyaa.
On Wed, Oct 10, 2018 at 2:26 PM Kevin Risden <[email protected]> wrote:
> Moving this back to the user list since it was sent without the list:
>
> Question: Is this error coming from the spark history server or the knox
> server?
>
> Answer: It is coming from the spark history UI. Here is what I see in the
> log for spark history ui.
>
> 18/10/10 12:30:29 WARN HttpParser: Header is too large >8192
>
> 18/10/10 12:30:29 WARN HttpParser: badMessage: 413 for
> HttpChannelOverHttp@2756e900{r=1,c=false,a=IDLE,uri=/?doAs=<user>}
>
>
> With the above information there is probably nothing that can be done from
> Knox. The bad news is that there is no configuration in Spark for this yet.
> See the following:
> * https://issues.apache.org/jira/browse/SPARK-15090
> *
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala#L337
> * Should see something about:
> * connector.setRequestHeaderSize(requestHeaderSize);
> * connector.setResponseHeaderSize(responseHeaderSize);
>
> Without a configuration to change, the Spark history UI is going to
> default to the 8192 header size. A related change was done for most
> projects to make that header size configurable.
>
> Kevin Risden
>
>
> On Tue, Oct 9, 2018 at 8:45 PM Theyaa Matti <[email protected]> wrote:
>
>> Hi David,
>> Thank you for the quick response. I do have that property set
>> with the recommended value in Knox, but the issue still persists only with
>> the Spark History UI. The issue appears only when I enable ssl for knox. If
>> I turn ssl off in Knox the issue disappears, and if I enable it, it shows
>> up right away.
>>
>> Are there any equivalent properties for Spark History UI? since
>> I suspect the issue is with the Spark History UI and not with Knox since
>> all the other UIs work fine with Knox.
>>
>> Best,
>>
>> Theyaa.
>>
>>
>>
>> On Tue, Oct 9, 2018 at 7:11 PM David Villarreal <
>> [email protected]> wrote:
>>
>>> Hi Theyaa,
>>>
>>> Change the size of gateway.httpserver.requestHeaderBuffer property. I
>>> think the default is 8912 (8k) change to 16384. See if that helps.
>>>
>>> For the second problem Request is a replay (34))] this message is often
>>> seen when the timing of one of the servers is off. Make sure you use NTPD
>>> on all servers and they are all in sync. If everything is in sync you can
>>> work around this issue by turning off krb5 replay cache. With the following
>>> parameter
>>> -Dsun.security.krb5.rcache=none
>>>
>>> dav
>>>
>>>
>>> On 10/9/18, 9:01 AM, "Theyaa Matti" <[email protected]> wrote:
>>>
>>> Hi,
>>> I am getting this error message "WARN HttpParser: Header is too
>>> large
>>> >8192" when trying to access the spark history ui through knox. Any
>>> idea
>>> please?
>>>
>>> Also when trying to load the executors page, I get : GSS initiate
>>> failed
>>> [Caused by GSSException: Failure unspecified at GSS-API level
>>> (Mechanism level:
>>> Request is a replay (34))]
>>>
>>> when knox is requesting executorspage-template.html
>>>
>>> appreciate any help here.
>>>
>>>
>>>
