Dear Hari,
Thank you for your reply.
Replying to your questions:
Yes, I have all needed entries in etc/hosts and hosts can 'see' each other.
I followed your suggestion and added mentioned entries in
server.properties_krb5. Now when starting Kafka Broker I see:
listeners = PLAINTEXT://:9092,SASL_PLAINTEXT://:9093
advertised.listeners = PLAINTEXT://:9092,SASL_PLAINTEXT://:9093
sasl.kerberos.service.name = kafka
advertised.host.name = plx164h.xx.xxx.xx
Unfortunately it didn't help. Error in StreamSets is the same. I've tried
to use built-in kafka console consumer and also not succeded. Here is my
config:
On host A I have Kafka broker which is running with the config from
previous email. On host B, I have another Kafka from which I used console
consumer with following config:
kafka_client_jaas.conf:
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/etc/security/keytabs/kafka_client.service.keytab"
principal="client/10.xxx.xxx...@hdpcybersecacc.xx.xx";
};
consumer.properties:
security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=client
I'm starting console consumer with the command:
./bin/kafka-console-consumer.sh --bootstrap-server plx164h:9093 --topic
streamsets2 --new-consumer --consumer.config consumer.properties
When started, there is no error, console consumer seems to work fine, but
when producing to this topic, no messages are read.
>From kerberos side everything looks correct:
Mar 04 16:00:31 lxhnlxx.xxx.xx krb5kdc[16307](info): AS_REQ (4 etypes {18
17 16 23}) 10.xxx.xxx.72
<http://www.google.com/url?q=http%3A%2F%2F10.111.159.72&sa=D&sntz=1&usg=AFQjCNGfeOUI5N_QC-VKyS_d9ouhNTfYpQ>:
ISSUE: authtime 1457103631, etypes {rep=18 tkt=18 ses=18},
client/10.xxx.xxxx...@hdpcybersecacc.xx.xx for
krbtgt/hdpcybersecacc.xx...@hdpcybersecacc.xx.xx
Mar 04 16:00:31 lxhnlxx.xxx.xx krb5kdc[16307](info): TGS_REQ (4 etypes {18
17 16 23}) 10.xxx.xxx.72
<http://www.google.com/url?q=http%3A%2F%2F10.111.159.72&sa=D&sntz=1&usg=AFQjCNGfeOUI5N_QC-VKyS_d9ouhNTfYpQ>:
ISSUE: authtime 1457103631, etypes {rep=18 tkt=18 ses=18},
client/10.xxx.xxx...@hdpcybersecacc.xx.xx for
client/plx164h.xx...@hdpcybersecacc.xx.xx
Could you please a look at this? Maybe you see configuration error?
Kind regards,
Michal
W dniu czwartek, 3 marca 2016 17:49:03 UTC+1 użytkownik Harikiran Nayak
napisał:
> Hi Michal,
>
> Can you please add the *advertised.listeners* and *advertised.host.name
> <http://advertised.host.name>* properties in your kafka server config
> file 'server.properties_krb5'?
>
> For example, I have the following configuration in my working setup
>
> listeners=SASL_PLAINTEXT://:9092
> advertised.listeners=SASL_PLAINTEXT://:9092
> host.name=kafka
> advertised.host.name=kafka
>
> 'kafka' is the hostname on which the Kafka broker is running in my setup.
> There is an entry for this host in '/etc/hosts' on the node where
> StreamSets is running.
>
> Thanks
> Hari.
>
> On Thu, Mar 3, 2016 at 8:19 AM Harikiran Nayak <ha...@streamsets.com
> <javascript:>> wrote:
>
>> Hi Michal,
>>
>> Are you able to write and read from the kerberized Kafka setup using the
>> Kafka Console Producer and Consumer?
>>
>> I am taking a look at your configuration files.
>>
>> Thanks
>> Hari.
>>
>> On Thu, Mar 3, 2016 at 8:09 AM Jonathan Natkins <na...@streamsets.com
>> <javascript:>> wrote:
>>
>>> Hey Michal,
>>>
>>> I'm cc'ing the StreamSets user list, which might be able to get you
>>> some better StreamSets-specific answers.
>>>
>>> Thanks!
>>> Natty
>>>
>>> On Thursday, March 3, 2016, Michał Kabocik <michal....@gmail.com
>>> <javascript:>> wrote:
>>>
>>>> Dears,
>>>>
>>>> I’m Middleware Engineer and I’m trying to configure secure Kafka
>>>> Cluster with SSL and Kerberos authentication with StreamSets, which will
>>>> be
>>>> used for data injection to HDP.
>>>>
>>>> I have two Kafka Clusters; one with SSL enabled and there I
>>>> successfully connected StreamSets to Kafka with SSL authentication, and
>>>> second one with Kerberos authentication and here I’m facing with the
>>>> problem:
>>>>
>>>> Both Kafka (with Zookeeper) and StreamSets are configured to
>>>> authenticate via Kerberos. When starting all of them, I see in the logs,
>>>> that they are successfully authenticated (TGT granted etc.)
>>>>
>>>> I have two listeners defined in Kafka:
>>>> listeners=PLAINTEXT://:9092,SASL_PLAINTEXT://:9093. When starting Kafka, I
>>>> see Kafka listens on both, 9092 and 9093.
>>>>
>>>> When I connect StreamSets to Kafka on port 9092, everything works
>>>> smooth. But when I try to connect to port 9093, error occurs:
>>>>
>>>> KAFKA_41 - Could not get partition count for topic 'streamsets5' :
>>>> com.streamsets.pipeline.api.StageException: KAFKA_41 - Could not get
>>>> caseition count for topic 'streamsets5' :
>>>> org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
>>>>
>>>> I see no errors in Kafka, in the log of StreamSets, there is only above
>>>> error visible. I attached major config files of Kafka, Zookeeper and
>>>> StreamSets.
>>>>
>>>> Will greatly appreciate your help in solving this case!
>>>>
>>>> Kind regards,
>>>>
>>>
>>>
>>> --
>>> Jonathan "Natty" Natkins
>>> StreamSets | Field Engineer
>>> mobile: 609.577.1600 <#> | linkedin
>>> <http://www.linkedin.com/in/nattyice>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "sdc-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to sdc-user+u...@streamsets.com <javascript:>.
>>> Visit this group at
>>> https://groups.google.com/a/streamsets.com/group/sdc-user/.
>>>
>>
