Ottomata has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/404870 )
Change subject: Update Kafka to 1.0 with SSL support ...................................................................... Update Kafka to 1.0 with SSL support This will make testing Mediawiki integration with Kafka and SSL easier Bug: T126494 Change-Id: I93d7c7cb98664e3e41b5a383ba8f9976a0b09099 --- M puppet/modules/kafka/files/kafka.profile.sh M puppet/modules/kafka/files/kafka.sh D puppet/modules/kafka/files/server.properties A puppet/modules/kafka/files/ssl/kafka_broker/ca.crt.pem A puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.crt.pem A puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.csr.pem A puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.key.private.pem A puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.key.public.pem A puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.keystore.jks A puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.keystore.p12 A puppet/modules/kafka/files/ssl/kafka_broker/truststore.jks A puppet/modules/kafka/files/ssl/local_ca/ca.crt.pem A puppet/modules/kafka/files/ssl/local_ca/local_ca.crt.pem A puppet/modules/kafka/files/ssl/local_ca/local_ca.csr.pem A puppet/modules/kafka/files/ssl/local_ca/local_ca.key.private.pem A puppet/modules/kafka/files/ssl/local_ca/local_ca.key.public.pem A puppet/modules/kafka/files/ssl/local_ca/local_ca.keystore.jks A puppet/modules/kafka/files/ssl/local_ca/local_ca.keystore.p12 A puppet/modules/kafka/files/ssl/local_ca/truststore.jks A puppet/modules/kafka/files/ssl/test0/ca.crt.pem A puppet/modules/kafka/files/ssl/test0/test0.crt.pem A puppet/modules/kafka/files/ssl/test0/test0.csr.pem A puppet/modules/kafka/files/ssl/test0/test0.key.private.pem A puppet/modules/kafka/files/ssl/test0/test0.key.public.pem A puppet/modules/kafka/files/ssl/test0/test0.keystore.jks A puppet/modules/kafka/files/ssl/test0/test0.keystore.p12 A puppet/modules/kafka/files/ssl/test0/truststore.jks M puppet/modules/kafka/manifests/init.pp A puppet/modules/kafka/templates/server.properties.erb M puppet/modules/kafka/templates/systemd/kafka.erb M puppet/modules/role/settings/kafka.yaml 31 files changed, 421 insertions(+), 119 deletions(-) Approvals: Ottomata: Verified; Looks good to me, approved BryanDavis: Looks good to me, but someone else must approve diff --git a/puppet/modules/kafka/files/kafka.profile.sh b/puppet/modules/kafka/files/kafka.profile.sh index ab3ed80..f1f2a8a 100644 --- a/puppet/modules/kafka/files/kafka.profile.sh +++ b/puppet/modules/kafka/files/kafka.profile.sh @@ -3,5 +3,6 @@ # These environment variables are used by the kafka CLI # so that you don't have to provide them as arguments # every time you use it. -export ZOOKEEPER_URL=localhost:2181 -export BROKER_LIST=localhost:9092 +export KAFKA_ZOOKEEPER_URL=localhost:2181/kafka +export KAFKA_BOOTSTRAP_SERVERS=localhost:9092 +export KAFKA_JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 diff --git a/puppet/modules/kafka/files/kafka.sh b/puppet/modules/kafka/files/kafka.sh index e7db2bb..e2c1c8b 100755 --- a/puppet/modules/kafka/files/kafka.sh +++ b/puppet/modules/kafka/files/kafka.sh @@ -1,5 +1,7 @@ #!/bin/bash +# NOTE: This file is managed by Puppet. + SCRIPT_NAME=$(basename "$0") commands=$(ls /usr/bin/kafka-* | xargs -n 1 basename | sed 's@kafka-@ @g') @@ -8,9 +10,9 @@ $SCRIPT_NAME <command> [options] Handy wrapper around various kafka-* scripts. Set the environment variables -ZOOKEEPER_URL and BROKER_LIST so you don't have to keep typing ---zookeeper-connect or --broker-list each time you want to use a kafka-* -script. +KAFKA_ZOOKEEPER_URL, KAFKA_BOOTSTRAP_SERVERS so you don't have to keep typing +--zookeeper-connect, --broker-list or --bootstrap-server each time you want to +use a kafka-* script. Usage: @@ -20,11 +22,18 @@ $commands Environment Variables: - ZOOKEEPER_URL - If this is set, any commands that take a --zookeeper flag will be given this value. - BROKER_LIST - If this is set, any commands that take a --broker-list flag will be given this value. + KAFKA_JAVA_HOME - Value of JAVA_HOME to use for invoking Kafka commands. + KAFKA_ZOOKEEPER_URL - If this is set, any commands that take a --zookeeper + flag will be given this value. + KAFKA_BOOTSTRAP_SERVERS - If this is set, any commands that take a --broker-list or + --bootstrap-server flag will be given this value. + Also any command that take a --authorizer-properties + will get the correct zookeeper.connect value. + " -if [ -z "${1}" -o ${1:0:1} == '-' ]; then +# Print usage if no <command> given, or $1 starts with '-' +if [ -z "${1}" -o "${1:0:1}" == '-' ]; then echo "${USAGE}" exit 1 fi @@ -33,43 +42,77 @@ command="kafka-${1}" shift +# Export JAVA_HOME as KAFKA_JAVA_HOME if it is set. +# This makes kafka-run-class use the preferred JAVA_HOME for Kafka. +if [ -n "${KAFKA_JAVA_HOME}" ]; then + : ${JAVA_HOME="$KAFKA_JAVA_HOME"} + export JAVA_HOME +fi + # Set ZOOKEEPER_OPT if ZOOKEEPER_URL is set and --zookeeper has not # also been passed in as a CLI arg. This will be included # in command functions that take a --zookeeper argument. -if [ -n "${ZOOKEEPER_URL}" -a -z "$(echo $@ | grep -- --zookeeper)" ]; then - ZOOKEEPER_OPT="--zookeeper ${ZOOKEEPER_URL}" +if [ -n "${KAFKA_ZOOKEEPER_URL}" -a -z "$(echo $@ | grep -- --zookeeper)" ]; then + ZOOKEEPER_OPT="--zookeeper ${KAFKA_ZOOKEEPER_URL}" fi -# Set BROKER_LIST_OPT if BROKER_LIST is set and --broker-list has not +# Set BROKER_LIST_OPT if KAFKA_BOOTSTRAP_SERVERS is set and --broker-list has not # also been passed in as a CLI arg. This will be included # in command functions that take a --broker-list argument. -if [ -n "${BROKER_LIST}" -a -z "$(echo $@ | grep -- --broker-list)" ]; then - BROKER_LIST_OPT="--broker-list ${BROKER_LIST}" +if [ -n "${KAFKA_BOOTSTRAP_SERVERS}" -a -z "$(echo $@ | grep -- --broker-list)" ]; then + BROKER_LIST_OPT="--broker-list ${KAFKA_BOOTSTRAP_SERVERS}" fi -# Each of these lists signifies that either --broker-list or --zookeeper needs -# to be given to the $command. If $command matches one of these, then we -# will add the opt if it is not provided already in $@. +# Set BOOTSTRAP_SERVER_OPT if KAFKA_BOOTSTRAP_SERVERS is set and --bootstrap-server has not +# also been passed in as a CLI arg. This will be included +# in command functions that take a --bootstrap-server argument. +if [ -n "${KAFKA_BOOTSTRAP_SERVERS}" -a -z "$(echo $@ | grep -- --bootstrap-server)" ]; then + BOOTSTRAP_SERVER_OPT="--bootstrap-server ${KAFKA_BOOTSTRAP_SERVERS}" +fi + +# Set ZOOKEEPER_CONNECT_OPT if KAFKA_ZOOKEEPER_URL is set and '--authorizer-properties zookeeper.connect' +# has not also been passed in as a CLI arg. This will be included +# in command functions that take '--authorizer-properties zookeeper.connect' argument. +if [ -n "${KAFKA_ZOOKEEPER_URL}" -a -z "$(echo $@ | egrep -- '--authorizer-properties\ *zookeeper\.connect')" ]; then + ZOOKEEPER_CONNECT_OPT="--authorizer-properties zookeeper.connect=${KAFKA_ZOOKEEPER_URL}" +fi + +# Each of these lists signifies that either --broker-list, --bootstrap-server, +# or --zookeeper needs to be given to the $command. If $command matches one of these, +# then we will add the opt if it is not provided already in $@. +# Until https://issues.apache.org/jira/browse/KAFKA-4307 is available, there are +# inconsistencies in broker CLI parameters. Some use --bootstrap-server, others +# use --broker-list, so we have to support both for now. +# --broker-list should be removed in later versions in favor of --bootstrap-server broker_list_commands="kafka-console-producer "\ "kafka-consumer-perf-test "\ +"kafka-replay-log-producer "\ "kafka-replica-verification "\ "kafka-simple-consumer-shell "\ "kafka-verifiable-consumer "\ "kafka-verifiable-producer" +bootstrap_server_commands="kafka-console-consumer "\ +"kafka-broker-api-versions "\ +"kafka-consumer-groups " + zookeeper_commands="kafka-configs "\ -"kafka-console-consumer "\ -"kafka-consumer-groups "\ -"kafka-consumer-perf-test "\ +"kafka-consumer-offset-checker.sh "\ "kafka-preferred-replica-election "\ "kafka-reassign-partitions "\ "kafka-replay-log-producer "\ "kafka-topics" +zookeeper_connect_commands="kafka-acls" + EXTRA_OPTS="" echo "${broker_list_commands}" | /bin/grep -q "${command}" && EXTRA_OPTS="${BROKER_LIST_OPT} " +echo "${bootstrap_server_commands}" | /bin/grep -q "${command}" && EXTRA_OPTS="${EXTRA_OPTS}${BOOTSTRAP_SERVER_OPT} " echo "${zookeeper_commands}" | /bin/grep -q "${command}" && EXTRA_OPTS="${EXTRA_OPTS}${ZOOKEEPER_OPT} " +echo "${zookeeper_connect_commands}" | /bin/grep -q "${command}" && EXTRA_OPTS="${EXTRA_OPTS}${ZOOKEEPER_CONNECT_OPT} " # Print out the command we are about to exec, and then run it -echo "${command} ${EXTRA_OPTS}$@" -${command} ${EXTRA_OPTS}$@ +# set -f to not expand wildcards in command, e.g. --topic '*' +set -f +echo ${command} ${EXTRA_OPTS}"$@" +${command} ${EXTRA_OPTS}"$@" diff --git a/puppet/modules/kafka/files/server.properties b/puppet/modules/kafka/files/server.properties deleted file mode 100644 index a64c8cd..0000000 --- a/puppet/modules/kafka/files/server.properties +++ /dev/null @@ -1,86 +0,0 @@ -# NOTE: This file is managed by Puppet. - -############################# Server Basics ############################# - -# The id of the broker. This must be set to a unique integer for each broker. -broker.id=0 - -############################# Socket Server Settings ############################# - -listeners=PLAINTEXT://:9092 - -# The port the socket server listens on -#port=9092 - -# The number of threads handling network requests -num.network.threads=3 - -# The number of threads doing disk I/O -num.io.threads=8 - -# The send buffer (SO_SNDBUF) used by the socket server -socket.send.buffer.bytes=102400 - -# The receive buffer (SO_RCVBUF) used by the socket server -socket.receive.buffer.bytes=102400 - -# The maximum size of a request that the socket server will accept (protection against OOM) -socket.request.max.bytes=104857600 - - -############################# Log Basics ############################# - -# A comma seperated list of directories under which to store log files -log.dirs=/var/lib/kafka - -# The default number of log partitions per topic. More partitions allow greater -# parallelism for consumption, but this will also result in more files across -# the brokers. -num.partitions=1 - -# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown. -# This value is recommended to be increased for installations with data dirs located in RAID array. -num.recovery.threads.per.data.dir=1 - -############################# Log Retention Policy ############################# - -# The following configurations control the disposal of log segments. The policy can -# be set to delete segments after a period of time, or after a given size has accumulated. -# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens -# from the end of the log. - -# The minimum age of a log file to be eligible for deletion -log.retention.hours=168 - -# A size-based retention policy for logs. Segments are pruned from the log as long as the remaining -# segments don't drop below log.retention.bytes. -#log.retention.bytes=1073741824 - -# The maximum size of a log segment file. When this size is reached a new log segment will be created. -log.segment.bytes=1073741824 - -# The interval at which log segments are checked to see if they can be deleted according -# to the retention policies -log.retention.check.interval.ms=300000 - -# Allow topic deletion -delete.topic.enable=true - -############################# Zookeeper ############################# - -# Zookeeper connection string (see zookeeper docs for details). -# This is a comma separated host:port pairs, each corresponding to a zk -# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002". -# You can also append an optional chroot string to the urls to specify the -# root directory for all kafka znodes. -zookeeper.connect=localhost:2181 - -# Timeout in ms for connecting to zookeeper -zookeeper.connection.timeout.ms=6000 - -##################### Confluent Proactive Support ###################### - -# If set to true, then the feature to collect and report support metrics -# ("Metrics") is enabled. If set to false, the feature is disabled. -# -confluent.support.metrics.enable=false diff --git a/puppet/modules/kafka/files/ssl/kafka_broker/ca.crt.pem b/puppet/modules/kafka/files/ssl/kafka_broker/ca.crt.pem new file mode 100644 index 0000000..5b15145 --- /dev/null +++ b/puppet/modules/kafka/files/ssl/kafka_broker/ca.crt.pem @@ -0,0 +1,18 @@ +-----BEGIN CERTIFICATE----- +MIIC6DCCAdCgAwIBAgIUB4c5VpnNmpxVHmwDd433tjPQ/FEwDQYJKoZIhvcNAQEL +BQAwEzERMA8GA1UEAwwIbG9jYWxfY2EwIBcNMTgwMTE3MjEzNzU1WhgPMjExNzEy +MjQyMTM3NTVaMBMxETAPBgNVBAMMCGxvY2FsX2NhMIIBIjANBgkqhkiG9w0BAQEF +AAOCAQ8AMIIBCgKCAQEAsMsG+Y+N2BUmivLvKjxD6/YhS8q0lkn5XdGdDwI/LHVW +iWu3NKYYjtwBKOF6QORrZBatiUiFAo7WlcjHBPdg+bd7CD4+rV6By3O2d2wJuvpO +Oex2xT1iLOM4vuZFc/GbOtGWSe7+fJVef8IDwUcMq9U2BrV1nKV4Eb1coJBMifBT +b36zl0zgRDkajpxKaj7LdLTeCU/Vn+bE9BO0uAAQqGBu4Eyj6GqXX/IIWoxQ1nxR +GaIBhM2pNEzoBza5lX1MgYjitK0rC/LAa54bE154hd40fTN5sfnGFjSK+jsiL9w1 +g3myH+Vlt7Y+3yGls7Km3sWAhuN/vDeAKmUslV0oWQIDAQABozIwMDAPBgNVHRMB +Af8EBTADAQH/MB0GA1UdDgQWBBRwYbjt7osV+1HSg6q0ey/JcljdgDANBgkqhkiG +9w0BAQsFAAOCAQEAqQGzKDl0x0LU/Q+YMfFY5499COmJ2ATp5I1Ou/GngrxhkdFk +ZlliPouVMNpqm+Xpr4vSHm5wlis09xxm508z8JAA/PzKo847tGMQBnLwmdW67f4H +njUsJsFBOfakfc64f0LUfHRgHL4EXF6hbC4W4PhZH09cbt91v7CjY0KGMdHThe2Y +1y5/QQmVH7tPyDeOnDUrCiXmdwtWe2KOyu04e36NYTx9hQzC26WXmdz4rbI/MEQ1 +K9SJs96pIz/X3MyQ/JfF8ThOplhJ1ACUkUyzxmjmzXc5evUHF3Ho3fAIGpl8z1fE +REWPT9f0L3p4mBuXyJN3KPfyd9ylec3wuSdPQw== +-----END CERTIFICATE----- diff --git a/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.crt.pem b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.crt.pem new file mode 100644 index 0000000..c7df18b --- /dev/null +++ b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.crt.pem @@ -0,0 +1,15 @@ +-----BEGIN CERTIFICATE----- +MIICPTCCASWgAwIBAgIUdgK/B0WTHrDyaitpTAgd+MajusMwDQYJKoZIhvcNAQEL +BQAwEzERMA8GA1UEAwwIbG9jYWxfY2EwHhcNMTgwMTE3MjEzODQ0WhcNMTkwMTE3 +MjEzODQ0WjAXMRUwEwYDVQQDDAxrYWZrYV9icm9rZXIwWTATBgcqhkjOPQIBBggq +hkjOPQMBBwNCAASGYVywLuFgffqmD0TswEHeyALZecYSpbp1qe6kgH7fXN71t3+l +SQ2f0maLq+vNqbDVGJGnkq+gJylMt+9h/UW7o1AwTjAMBgNVHRMBAf8EAjAAMB0G +A1UdDgQWBBQnM27LGtQA4rV6ct2I6G6ke8IMQTAfBgNVHSMEGDAWgBRwYbjt7osV ++1HSg6q0ey/JcljdgDANBgkqhkiG9w0BAQsFAAOCAQEApjV43RBvCGny/l725hck +zv1AkZiX3o1BajiLWqa8LTTGOr5u01f7FmCPZ9c0sZCSfkweRzAbj3uwKg/4fJab +dh4BCMsfbEU/azZ1dosKT14hdEWHawYKyRbdInmSB7u1cNGbDXToQg+wv/tsB6M0 +jA56sERp2FfdXC1sR5/LO26VXN0S8oDwCSb/nLbz/FBZA31rnitOJL2HzKnMfh/5 +3KdBngsVC17DA9Q7mKd11K8G6mSpM5aD+a8+SF755Jr/rGTme1lbJ3yVNohMRM7k +rEbWW1ZUMN1kNd1kwqb8l54DtPxWXpYkC3bDZm1qCbOAuC1z72E1UbMok/ZHqeeB +Rw== +-----END CERTIFICATE----- diff --git a/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.csr.pem b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.csr.pem new file mode 100644 index 0000000..789e93b --- /dev/null +++ b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.csr.pem @@ -0,0 +1,9 @@ +-----BEGIN CERTIFICATE REQUEST----- +MIIBMTCB2AIBADAXMRUwEwYDVQQDDAxrYWZrYV9icm9rZXIwWTATBgcqhkjOPQIB +BggqhkjOPQMBBwNCAASGYVywLuFgffqmD0TswEHeyALZecYSpbp1qe6kgH7fXN71 +t3+lSQ2f0maLq+vNqbDVGJGnkq+gJylMt+9h/UW7oF8wXQYJKoZIhvcNAQkOMVAw +TjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBQnM27LGtQA4rV6ct2I6G6ke8IMQTAf +BgNVHSMEGDAWgBRwYbjt7osV+1HSg6q0ey/JcljdgDAKBggqhkjOPQQDAgNIADBF +AiEAnMLETBbG4OCajAiKQcOPxstu1c8aRv7N4lEs1STPTW4CICwkzCuhzsLQ7E+V +mDLyUNhNeDxJ7YIKeY0Atl8EherX +-----END CERTIFICATE REQUEST----- diff --git a/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.key.private.pem b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.key.private.pem new file mode 100644 index 0000000..e508c8a --- /dev/null +++ b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.key.private.pem @@ -0,0 +1,7 @@ +-----BEGIN ENCRYPTED PRIVATE KEY----- +MIHsMFcGCSqGSIb3DQEFDTBKMCkGCSqGSIb3DQEFDDAcBAjRb3lYmKm2BwICCAAw +DAYIKoZIhvcNAgkFADAdBglghkgBZQMEASoEEIOlTfYrDLSN+s7dW9dKBPwEgZCe +oOFlEcPasrxHqF+p8vVZrgVacxco0+4Si1UipaNNTocJsxngOU4CUzOq+yZuOydx +7YJ+nTbn/rNmGtIeCpxrJ2SaCx0/U5XafaWY5jRjCi5NEWwkT3au7aamcmsRcaZN +gBb/R0P995nCzPgSZ4oHPFj8BEppDde8BYHfviLjxJdOYrw9kBa5c6+q+tfEuB8= +-----END ENCRYPTED PRIVATE KEY----- diff --git a/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.key.public.pem b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.key.public.pem new file mode 100644 index 0000000..4a4921b --- /dev/null +++ b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.key.public.pem @@ -0,0 +1,4 @@ +-----BEGIN PUBLIC KEY----- +MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEhmFcsC7hYH36pg9E7MBB3sgC2XnG +EqW6danupIB+31ze9bd/pUkNn9Jmi6vrzamw1RiRp5KvoCcpTLfvYf1Fuw== +-----END PUBLIC KEY----- diff --git a/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.keystore.jks b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.keystore.jks new file mode 100644 index 0000000..8e46fb9 --- /dev/null +++ b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.keystore.jks Binary files differ diff --git a/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.keystore.p12 b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.keystore.p12 new file mode 100644 index 0000000..44c9b0e --- /dev/null +++ b/puppet/modules/kafka/files/ssl/kafka_broker/kafka_broker.keystore.p12 Binary files differ diff --git a/puppet/modules/kafka/files/ssl/kafka_broker/truststore.jks b/puppet/modules/kafka/files/ssl/kafka_broker/truststore.jks new file mode 100644 index 0000000..df7ec3f --- /dev/null +++ b/puppet/modules/kafka/files/ssl/kafka_broker/truststore.jks Binary files differ diff --git a/puppet/modules/kafka/files/ssl/local_ca/ca.crt.pem b/puppet/modules/kafka/files/ssl/local_ca/ca.crt.pem new file mode 100644 index 0000000..5b15145 --- /dev/null +++ b/puppet/modules/kafka/files/ssl/local_ca/ca.crt.pem @@ -0,0 +1,18 @@ +-----BEGIN CERTIFICATE----- +MIIC6DCCAdCgAwIBAgIUB4c5VpnNmpxVHmwDd433tjPQ/FEwDQYJKoZIhvcNAQEL +BQAwEzERMA8GA1UEAwwIbG9jYWxfY2EwIBcNMTgwMTE3MjEzNzU1WhgPMjExNzEy +MjQyMTM3NTVaMBMxETAPBgNVBAMMCGxvY2FsX2NhMIIBIjANBgkqhkiG9w0BAQEF +AAOCAQ8AMIIBCgKCAQEAsMsG+Y+N2BUmivLvKjxD6/YhS8q0lkn5XdGdDwI/LHVW +iWu3NKYYjtwBKOF6QORrZBatiUiFAo7WlcjHBPdg+bd7CD4+rV6By3O2d2wJuvpO +Oex2xT1iLOM4vuZFc/GbOtGWSe7+fJVef8IDwUcMq9U2BrV1nKV4Eb1coJBMifBT +b36zl0zgRDkajpxKaj7LdLTeCU/Vn+bE9BO0uAAQqGBu4Eyj6GqXX/IIWoxQ1nxR +GaIBhM2pNEzoBza5lX1MgYjitK0rC/LAa54bE154hd40fTN5sfnGFjSK+jsiL9w1 +g3myH+Vlt7Y+3yGls7Km3sWAhuN/vDeAKmUslV0oWQIDAQABozIwMDAPBgNVHRMB +Af8EBTADAQH/MB0GA1UdDgQWBBRwYbjt7osV+1HSg6q0ey/JcljdgDANBgkqhkiG +9w0BAQsFAAOCAQEAqQGzKDl0x0LU/Q+YMfFY5499COmJ2ATp5I1Ou/GngrxhkdFk +ZlliPouVMNpqm+Xpr4vSHm5wlis09xxm508z8JAA/PzKo847tGMQBnLwmdW67f4H +njUsJsFBOfakfc64f0LUfHRgHL4EXF6hbC4W4PhZH09cbt91v7CjY0KGMdHThe2Y +1y5/QQmVH7tPyDeOnDUrCiXmdwtWe2KOyu04e36NYTx9hQzC26WXmdz4rbI/MEQ1 +K9SJs96pIz/X3MyQ/JfF8ThOplhJ1ACUkUyzxmjmzXc5evUHF3Ho3fAIGpl8z1fE +REWPT9f0L3p4mBuXyJN3KPfyd9ylec3wuSdPQw== +-----END CERTIFICATE----- diff --git a/puppet/modules/kafka/files/ssl/local_ca/local_ca.crt.pem b/puppet/modules/kafka/files/ssl/local_ca/local_ca.crt.pem new file mode 100644 index 0000000..5b15145 --- /dev/null +++ b/puppet/modules/kafka/files/ssl/local_ca/local_ca.crt.pem @@ -0,0 +1,18 @@ +-----BEGIN CERTIFICATE----- +MIIC6DCCAdCgAwIBAgIUB4c5VpnNmpxVHmwDd433tjPQ/FEwDQYJKoZIhvcNAQEL +BQAwEzERMA8GA1UEAwwIbG9jYWxfY2EwIBcNMTgwMTE3MjEzNzU1WhgPMjExNzEy +MjQyMTM3NTVaMBMxETAPBgNVBAMMCGxvY2FsX2NhMIIBIjANBgkqhkiG9w0BAQEF +AAOCAQ8AMIIBCgKCAQEAsMsG+Y+N2BUmivLvKjxD6/YhS8q0lkn5XdGdDwI/LHVW +iWu3NKYYjtwBKOF6QORrZBatiUiFAo7WlcjHBPdg+bd7CD4+rV6By3O2d2wJuvpO +Oex2xT1iLOM4vuZFc/GbOtGWSe7+fJVef8IDwUcMq9U2BrV1nKV4Eb1coJBMifBT +b36zl0zgRDkajpxKaj7LdLTeCU/Vn+bE9BO0uAAQqGBu4Eyj6GqXX/IIWoxQ1nxR +GaIBhM2pNEzoBza5lX1MgYjitK0rC/LAa54bE154hd40fTN5sfnGFjSK+jsiL9w1 +g3myH+Vlt7Y+3yGls7Km3sWAhuN/vDeAKmUslV0oWQIDAQABozIwMDAPBgNVHRMB +Af8EBTADAQH/MB0GA1UdDgQWBBRwYbjt7osV+1HSg6q0ey/JcljdgDANBgkqhkiG +9w0BAQsFAAOCAQEAqQGzKDl0x0LU/Q+YMfFY5499COmJ2ATp5I1Ou/GngrxhkdFk +ZlliPouVMNpqm+Xpr4vSHm5wlis09xxm508z8JAA/PzKo847tGMQBnLwmdW67f4H +njUsJsFBOfakfc64f0LUfHRgHL4EXF6hbC4W4PhZH09cbt91v7CjY0KGMdHThe2Y +1y5/QQmVH7tPyDeOnDUrCiXmdwtWe2KOyu04e36NYTx9hQzC26WXmdz4rbI/MEQ1 +K9SJs96pIz/X3MyQ/JfF8ThOplhJ1ACUkUyzxmjmzXc5evUHF3Ho3fAIGpl8z1fE +REWPT9f0L3p4mBuXyJN3KPfyd9ylec3wuSdPQw== +-----END CERTIFICATE----- diff --git a/puppet/modules/kafka/files/ssl/local_ca/local_ca.csr.pem b/puppet/modules/kafka/files/ssl/local_ca/local_ca.csr.pem new file mode 100644 index 0000000..ffd4f66 --- /dev/null +++ b/puppet/modules/kafka/files/ssl/local_ca/local_ca.csr.pem @@ -0,0 +1,16 @@ +-----BEGIN CERTIFICATE REQUEST----- +MIICmTCCAYECAQAwEzERMA8GA1UEAwwIbG9jYWxfY2EwggEiMA0GCSqGSIb3DQEB +AQUAA4IBDwAwggEKAoIBAQCwywb5j43YFSaK8u8qPEPr9iFLyrSWSfld0Z0PAj8s +dVaJa7c0phiO3AEo4XpA5GtkFq2JSIUCjtaVyMcE92D5t3sIPj6tXoHLc7Z3bAm6 ++k457HbFPWIs4zi+5kVz8Zs60ZZJ7v58lV5/wgPBRwyr1TYGtXWcpXgRvVygkEyJ +8FNvfrOXTOBEORqOnEpqPst0tN4JT9Wf5sT0E7S4ABCoYG7gTKPoapdf8ghajFDW +fFEZogGEzak0TOgHNrmVfUyBiOK0rSsL8sBrnhsTXniF3jR9M3mx+cYWNIr6OyIv +3DWDebIf5WW3tj7fIaWzsqbexYCG43+8N4AqZSyVXShZAgMBAAGgQTA/BgkqhkiG +9w0BCQ4xMjAwMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFHBhuO3uixX7UdKD +qrR7L8lyWN2AMA0GCSqGSIb3DQEBCwUAA4IBAQAWmdBr9XyKaL1Oe0XMJr23yjq+ +ouj5pLf/Z2zmb540HPDgpDJQIAZNhtUOdFzbyrP3TcHFZK1birMYB9lsyRzaXBlw +lIqD1eCxvqki2u+0t9/Qqs4kTYU9LQEW9T4oj1sbA+mbz6/F3FCmCgVG/XW0kGOq +CuiIRONDnF/QUs2NBb3eWbhnwGF5pW+dWWC3m4U+B8hqOOHBM4PHbUq0nBjA99u6 +LG3/bxog5pEE2wrp31HEizu8Ou3/dt7zhBAjwqSLq9rn6GNF08UVfENzJrYM2kGf +ST+wZFL8NzLw/HEmUlRSOOt4sa+fzJ1AD3lMHECf+34Hg4T14EGgM6iEMw3S +-----END CERTIFICATE REQUEST----- diff --git a/puppet/modules/kafka/files/ssl/local_ca/local_ca.key.private.pem b/puppet/modules/kafka/files/ssl/local_ca/local_ca.key.private.pem new file mode 100644 index 0000000..5a2a9a3 --- /dev/null +++ b/puppet/modules/kafka/files/ssl/local_ca/local_ca.key.private.pem @@ -0,0 +1,30 @@ +-----BEGIN ENCRYPTED PRIVATE KEY----- +MIIFLTBXBgkqhkiG9w0BBQ0wSjApBgkqhkiG9w0BBQwwHAQIp5YiQej3JXQCAggA +MAwGCCqGSIb3DQIJBQAwHQYJYIZIAWUDBAEqBBB1TdS3A2ynG4wnIrrHGG6uBIIE +0AjVrEAkvTtEemcBqCbGYa/IiwR9A7CaTpyCLufPfvBxW4euhIN6J9ucrYD4dS3m +tgrz7mXqUxs0HAapQttHpx1kEX2eSTfRMAhkEW3VKzFuxixUcpokKTMEf/9NVBd5 +C2aw2kXfi94plJPo9FX49Gst/nWBLzHin1Wh+Ferb3cxU5Rw53uBOvGGU04AgJ9Y +3Hbe5VCeUPl6yGuNn3iaFbTjOJuZwPFpOU7DLEL/kgl0oGjJKfjD9pcB04DJADB2 +grns1xsYqIscC0rRK5Cuv82KHtYvpRCT+68UW5ZySQpMRLRTwBu10z2FF20X092p +5PLV9OMqvBjN1Fv4QGCkjGIhIlHyHTiNMrozsN3upjvocG7y1TnUgT51FVUS4hYf +mFJvAZQuAP0dParpuva7YDJ9xEoYaz/mXj1rrvC7kZ3iPG5y+A20mhRR2gDo2aHU +W3ZZe+XHo6NV3ULLcgr6dykrmXldnmNzP5gXF1tZTSqXlu6Q+x+CaJtQAESEHTqV +7nk25DSM2U1/+Lw+wT0mgonAaLf2gQaYFjCeLUfFAm4EfFLSHZsYLc11oVmSJRFp +CU5afpCj6Ygc/FT68hclakutDwGTRii4tNN635pMknYh178sn5QKqv1uLN2PWt09 +OL4Z37Ma6YXtT/t7ruM745JbY2pcFt0BPYiHRCnm/fzjAJydqhM6eEl23+H1xgQO +BtFsqVs1W8UfnG/O+SdoSPRmzxcQ8GxOXZhllGaaNHionlvfqzFKtIVlz/2zPjld +TvELkyKuxnz1qDZDJs5PFC5DL7DR1yi/Aezsn+cRg0bj03M1qAJxZ7cihZqZc0IZ +rV+U2mlfbEIfndt79GpdQ5Vzgbxxk9BzAHN+tyMY6//CyeZqYUdYyH+mFipdyF95 +/019rxXEYy+JnFIAg26vAVGDgORVVuHphM53LOpk/AqA8ravFGhTO4ABHafXV3Gv +VAj4hu69tIkoX91xZPm6VMnDAeP76IfuKk/uje4JrlpJ3Hoqv6Qi0r4TVaiGwx1N +v93vgJ9CYKOKHwS+UaIwHzqLrEbQ+Q1bIGabvQXAmcTQJP0tPe/Xv8oOKL46tzhA +i1gihluu8REGf5/09+ibqb6ktj0X2hGDU17/FAowmPkFQEee03fsg3V41bBG1OL/ +YwQMD4EhvcxHnWfli+PtoM55GoCA1jhD4ucfDGCAiMEoUYfcj2KDlqhHGA+24LL1 +Ge6Dm4sbGfV1LnNkHKysG3cRsNzxZ1VHFnslB/5UShYNd4Bft8UgZ+VRustnbwiq +F0ZZYiNA8ENwjA6bida92CvZmq7gXug87FhKiFlHREjYtuqvkEWl1C7UjxbZUcx3 +X+fBg/fJo3M4FZ3wlrAYFi23szdtyf43UNe/dSuE9mJO85DJjTapyFEESnkwFoNQ +w2JhX7Wzv3OdwwNqfnpVXLdALBh0svqu2t7nRL3oOENpFU3AkWqLafTNV4vfFQsU ++U7Io1hHmuM3Tjcy+lGUTeuexSBN7EGpdoAeTlf1l+CruCOWYvbQ9NzyRUskDeMD +mdCmh41EhwjuwT6YcjQLWOD9uXPdUnHyUy7c3nmRrUBAgQd2A9of0x3fvzrwRWd6 +MDoQdYYasRJ8Q9Lub0jvW7Y8EPocA7ewZAv+SijxCB/w +-----END ENCRYPTED PRIVATE KEY----- diff --git a/puppet/modules/kafka/files/ssl/local_ca/local_ca.key.public.pem b/puppet/modules/kafka/files/ssl/local_ca/local_ca.key.public.pem new file mode 100644 index 0000000..8e9daab --- /dev/null +++ b/puppet/modules/kafka/files/ssl/local_ca/local_ca.key.public.pem @@ -0,0 +1,9 @@ +-----BEGIN PUBLIC KEY----- +MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsMsG+Y+N2BUmivLvKjxD +6/YhS8q0lkn5XdGdDwI/LHVWiWu3NKYYjtwBKOF6QORrZBatiUiFAo7WlcjHBPdg ++bd7CD4+rV6By3O2d2wJuvpOOex2xT1iLOM4vuZFc/GbOtGWSe7+fJVef8IDwUcM +q9U2BrV1nKV4Eb1coJBMifBTb36zl0zgRDkajpxKaj7LdLTeCU/Vn+bE9BO0uAAQ +qGBu4Eyj6GqXX/IIWoxQ1nxRGaIBhM2pNEzoBza5lX1MgYjitK0rC/LAa54bE154 +hd40fTN5sfnGFjSK+jsiL9w1g3myH+Vlt7Y+3yGls7Km3sWAhuN/vDeAKmUslV0o +WQIDAQAB +-----END PUBLIC KEY----- diff --git a/puppet/modules/kafka/files/ssl/local_ca/local_ca.keystore.jks b/puppet/modules/kafka/files/ssl/local_ca/local_ca.keystore.jks new file mode 100644 index 0000000..c63da93 --- /dev/null +++ b/puppet/modules/kafka/files/ssl/local_ca/local_ca.keystore.jks Binary files differ diff --git a/puppet/modules/kafka/files/ssl/local_ca/local_ca.keystore.p12 b/puppet/modules/kafka/files/ssl/local_ca/local_ca.keystore.p12 new file mode 100644 index 0000000..e88c4ef --- /dev/null +++ b/puppet/modules/kafka/files/ssl/local_ca/local_ca.keystore.p12 Binary files differ diff --git a/puppet/modules/kafka/files/ssl/local_ca/truststore.jks b/puppet/modules/kafka/files/ssl/local_ca/truststore.jks new file mode 100644 index 0000000..7dea6a4 --- /dev/null +++ b/puppet/modules/kafka/files/ssl/local_ca/truststore.jks Binary files differ diff --git a/puppet/modules/kafka/files/ssl/test0/ca.crt.pem b/puppet/modules/kafka/files/ssl/test0/ca.crt.pem new file mode 100644 index 0000000..5b15145 --- /dev/null +++ b/puppet/modules/kafka/files/ssl/test0/ca.crt.pem @@ -0,0 +1,18 @@ +-----BEGIN CERTIFICATE----- +MIIC6DCCAdCgAwIBAgIUB4c5VpnNmpxVHmwDd433tjPQ/FEwDQYJKoZIhvcNAQEL +BQAwEzERMA8GA1UEAwwIbG9jYWxfY2EwIBcNMTgwMTE3MjEzNzU1WhgPMjExNzEy +MjQyMTM3NTVaMBMxETAPBgNVBAMMCGxvY2FsX2NhMIIBIjANBgkqhkiG9w0BAQEF +AAOCAQ8AMIIBCgKCAQEAsMsG+Y+N2BUmivLvKjxD6/YhS8q0lkn5XdGdDwI/LHVW +iWu3NKYYjtwBKOF6QORrZBatiUiFAo7WlcjHBPdg+bd7CD4+rV6By3O2d2wJuvpO +Oex2xT1iLOM4vuZFc/GbOtGWSe7+fJVef8IDwUcMq9U2BrV1nKV4Eb1coJBMifBT +b36zl0zgRDkajpxKaj7LdLTeCU/Vn+bE9BO0uAAQqGBu4Eyj6GqXX/IIWoxQ1nxR +GaIBhM2pNEzoBza5lX1MgYjitK0rC/LAa54bE154hd40fTN5sfnGFjSK+jsiL9w1 +g3myH+Vlt7Y+3yGls7Km3sWAhuN/vDeAKmUslV0oWQIDAQABozIwMDAPBgNVHRMB +Af8EBTADAQH/MB0GA1UdDgQWBBRwYbjt7osV+1HSg6q0ey/JcljdgDANBgkqhkiG +9w0BAQsFAAOCAQEAqQGzKDl0x0LU/Q+YMfFY5499COmJ2ATp5I1Ou/GngrxhkdFk +ZlliPouVMNpqm+Xpr4vSHm5wlis09xxm508z8JAA/PzKo847tGMQBnLwmdW67f4H +njUsJsFBOfakfc64f0LUfHRgHL4EXF6hbC4W4PhZH09cbt91v7CjY0KGMdHThe2Y +1y5/QQmVH7tPyDeOnDUrCiXmdwtWe2KOyu04e36NYTx9hQzC26WXmdz4rbI/MEQ1 +K9SJs96pIz/X3MyQ/JfF8ThOplhJ1ACUkUyzxmjmzXc5evUHF3Ho3fAIGpl8z1fE +REWPT9f0L3p4mBuXyJN3KPfyd9ylec3wuSdPQw== +-----END CERTIFICATE----- diff --git a/puppet/modules/kafka/files/ssl/test0/test0.crt.pem b/puppet/modules/kafka/files/ssl/test0/test0.crt.pem new file mode 100644 index 0000000..af99729 --- /dev/null +++ b/puppet/modules/kafka/files/ssl/test0/test0.crt.pem @@ -0,0 +1,14 @@ +-----BEGIN CERTIFICATE----- +MIICNjCCAR6gAwIBAgIUVyTo+2/NNL7zi60tsX/mGhTtABkwDQYJKoZIhvcNAQEL +BQAwEzERMA8GA1UEAwwIbG9jYWxfY2EwHhcNMTgwMTE3MjEzOTA4WhcNMTkwMTE3 +MjEzOTA4WjAQMQ4wDAYDVQQDDAV0ZXN0MDBZMBMGByqGSM49AgEGCCqGSM49AwEH +A0IABDeSxfTQlf3w8Bizm3tXQJO/T+4ekZKr7BDEMaO9vaf4/aqJQTZ9UkMIlIKi +6wswg+JPmZcoZhAQgbt0drPrPw2jUDBOMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYE +FL4eoGQL6YYQqNYbC3fStuDI13sgMB8GA1UdIwQYMBaAFHBhuO3uixX7UdKDqrR7 +L8lyWN2AMA0GCSqGSIb3DQEBCwUAA4IBAQCZOzbMhYRjgzAeGWfrYuAhFhKYL2G6 +sQue4XsNSJoiSeqxZ82dMhmmtUigQOMoxzFgQWZ0imPCwf7rNhA1B4Ucy6QCFXIs +91O+DTjw7AqUBIEkhDNLbO6lwusJg+UfUbmW6djg8ruqVd6BULVX4KsJIXz/j6fH +2lnH9PnjDny39sBFU8jk/f/iH4ieW3nkpd+b0hgme1HL7oNgPO+OHtq9UtAsG5s3 +/7leFfpVhhXn+VIBgO8jyq10gat804hiXzm+m1R7pBzlwYoHk2bwa83VM4L9BB7s +dQKiTBGE+y4uxw/VK/pa6VzlSigDFsQU80JqRnXCUQ52hbXDB3PDt2Hw +-----END CERTIFICATE----- diff --git a/puppet/modules/kafka/files/ssl/test0/test0.csr.pem b/puppet/modules/kafka/files/ssl/test0/test0.csr.pem new file mode 100644 index 0000000..217739b --- /dev/null +++ b/puppet/modules/kafka/files/ssl/test0/test0.csr.pem @@ -0,0 +1,9 @@ +-----BEGIN CERTIFICATE REQUEST----- +MIIBKjCB0QIBADAQMQ4wDAYDVQQDDAV0ZXN0MDBZMBMGByqGSM49AgEGCCqGSM49 +AwEHA0IABDeSxfTQlf3w8Bizm3tXQJO/T+4ekZKr7BDEMaO9vaf4/aqJQTZ9UkMI +lIKi6wswg+JPmZcoZhAQgbt0drPrPw2gXzBdBgkqhkiG9w0BCQ4xUDBOMAwGA1Ud +EwEB/wQCMAAwHQYDVR0OBBYEFL4eoGQL6YYQqNYbC3fStuDI13sgMB8GA1UdIwQY +MBaAFHBhuO3uixX7UdKDqrR7L8lyWN2AMAoGCCqGSM49BAMCA0gAMEUCIQCOipDe +9zhSGYuqF6XIVRE8KIBsaIsFshzuc1JJGaFIzgIgM/CqEwEMirOgri3pw6BcItFO +rj0Ij4yBa1Phy5NU+bo= +-----END CERTIFICATE REQUEST----- diff --git a/puppet/modules/kafka/files/ssl/test0/test0.key.private.pem b/puppet/modules/kafka/files/ssl/test0/test0.key.private.pem new file mode 100644 index 0000000..f35b4dc --- /dev/null +++ b/puppet/modules/kafka/files/ssl/test0/test0.key.private.pem @@ -0,0 +1,7 @@ +-----BEGIN ENCRYPTED PRIVATE KEY----- +MIHsMFcGCSqGSIb3DQEFDTBKMCkGCSqGSIb3DQEFDDAcBAiWP25ydPgoiwICCAAw +DAYIKoZIhvcNAgkFADAdBglghkgBZQMEASoEECBJbSrJH/pwbuc1sSUiM34EgZB9 ++MLs/LExw2621Yk6PQjOXvbKUPdZnyXvmGzTe4OmsSuboVY9SRIbQcrsYgoAbrpC +ya030PPOvGGjQBl2mvei7Maz8EUQZKdROPaQyNbpJfUrzAx6V8A9q6ZwJS2CttRu +3siVNO8/xN89oyqTT0At+rC3aa4kyXar3nWwyUkCK8SrD7x07xpFivCfZivVZ7Q= +-----END ENCRYPTED PRIVATE KEY----- diff --git a/puppet/modules/kafka/files/ssl/test0/test0.key.public.pem b/puppet/modules/kafka/files/ssl/test0/test0.key.public.pem new file mode 100644 index 0000000..b62fef4 --- /dev/null +++ b/puppet/modules/kafka/files/ssl/test0/test0.key.public.pem @@ -0,0 +1,4 @@ +-----BEGIN PUBLIC KEY----- +MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEN5LF9NCV/fDwGLObe1dAk79P7h6R +kqvsEMQxo729p/j9qolBNn1SQwiUgqLrCzCD4k+ZlyhmEBCBu3R2s+s/DQ== +-----END PUBLIC KEY----- diff --git a/puppet/modules/kafka/files/ssl/test0/test0.keystore.jks b/puppet/modules/kafka/files/ssl/test0/test0.keystore.jks new file mode 100644 index 0000000..65e238d --- /dev/null +++ b/puppet/modules/kafka/files/ssl/test0/test0.keystore.jks Binary files differ diff --git a/puppet/modules/kafka/files/ssl/test0/test0.keystore.p12 b/puppet/modules/kafka/files/ssl/test0/test0.keystore.p12 new file mode 100644 index 0000000..d64712e --- /dev/null +++ b/puppet/modules/kafka/files/ssl/test0/test0.keystore.p12 Binary files differ diff --git a/puppet/modules/kafka/files/ssl/test0/truststore.jks b/puppet/modules/kafka/files/ssl/test0/truststore.jks new file mode 100644 index 0000000..453ff2c --- /dev/null +++ b/puppet/modules/kafka/files/ssl/test0/truststore.jks Binary files differ diff --git a/puppet/modules/kafka/manifests/init.pp b/puppet/modules/kafka/manifests/init.pp index c7e48d7..4068dce 100644 --- a/puppet/modules/kafka/manifests/init.pp +++ b/puppet/modules/kafka/manifests/init.pp @@ -1,14 +1,15 @@ # == Class: Kafka # -class kafka { +class kafka( + $ssl_enabled = true, +) { require ::service require ::mediawiki::ready_service - require ::kafka::repository + require kafka::repository - $kafka_package = 'confluent-kafka-2.11' - require_package('openjdk-8-jdk') + require_package('openjdk-8-jre') require_package('zookeeperd') - require_package($kafka_package) + require_package('confluent-kafka-2.11') require_package('kafkacat') $logdir = '/var/log/kafka' @@ -16,7 +17,7 @@ group { 'kafka': ensure => 'present', system => true, - require => Package[$kafka_package], + require => Package['confluent-kafka-2.11'], } # Kafka system user user { 'kafka': @@ -42,18 +43,29 @@ source => 'puppet:///modules/kafka/kafka.profile.sh', } + if $ssl_enabled { + file { '/etc/kafka/ssl': + ensure => 'directory', + source => 'puppet:///modules/kafka/ssl', + recurse => true, + owner => 'root', + group => 'root', + mode => '0755', + } + } + file { '/etc/kafka/server.properties': ensure => 'present', - source => 'puppet:///modules/kafka/server.properties', + content => template('kafka/server.properties.erb'), mode => '0444', - require => Package[$kafka_package], + require => Package['confluent-kafka-2.11'], } file { '/etc/kafka/log4j.properties': ensure => 'present', content => template('kafka/log4j.properties.erb'), mode => '0444', - require => Package[$kafka_package], + require => Package['confluent-kafka-2.11'], } file { [$logdir, '/var/lib/kafka']: @@ -61,7 +73,7 @@ owner => 'kafka', group => 'kafka', mode => '0755', - require => Package[$kafka_package], + require => Package['confluent-kafka-2.11'], } service { 'zookeeper': @@ -76,7 +88,7 @@ require => [ User['kafka'], Service['zookeeper'], - Package[$kafka_package], + Package['confluent-kafka-2.11'], ], subscribe => [ File['/etc/kafka/server.properties'], diff --git a/puppet/modules/kafka/templates/server.properties.erb b/puppet/modules/kafka/templates/server.properties.erb new file mode 100644 index 0000000..f2c9fd4 --- /dev/null +++ b/puppet/modules/kafka/templates/server.properties.erb @@ -0,0 +1,133 @@ +# NOTE: This file is managed by Puppet. + +############################# Server Basics ############################# + +# The id of the broker. This must be set to a unique integer for each broker. +broker.id=0 + +# Always require a static broker id. +broker.id.generation.enable=false + + +<% if @ssl_enabled -%> +listeners=PLAINTEXT://:9092,SSL://:9093 +<% else -%> +listeners=PLAINTEXT://:9092 +<% end -%> + +# Define whether the timestamp in the message is message create time or log append time. +# The value should be either `CreateTime` or `LogAppendTime` +log.message.timestamp.type=LogAppendTime + +######################### Socket Server Settings ######################## +<% if @ssl_enabled -%> +security.inter.broker.protocol=SSL + +ssl.keystore.location=/etc/kafka/ssl/kafka_broker/kafka_broker.keystore.jks +ssl.keystore.password=qwerty +ssl.key.password=qwerty +ssl.truststore.location=/etc/kafka/ssl/kafka_broker/truststore.jks +ssl.truststore.password=qwerty +ssl.enabled.protocols=TLSv1.2 +ssl.cipher.suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 + +ssl.client.auth=requested + +<% end -%> + +# The number of threads doing disk I/O +num.io.threads=1 + +# The send buffer (SO_SNDBUF) used by the socket server +socket.send.buffer.bytes=1048576 + +# The receive buffer (SO_RCVBUF) used by the socket server +socket.receive.buffer.bytes=1048576 + +############################# Log Basics ############################# + +# A comma seperated list of directories under which to store log files +log.dirs=/var/lib/kafka + +# The default number of log partitions per topic. More partitions allow greater +# parallelism for consumption, but this will also result in more files across +# the brokers. +num.partitions=1 + +# The default replication factor for automatically created topics. +# Default to the number of brokers in this cluster. +default.replication.factor=1 + +# Enables topic deletion +delete.topic.enable=true + +# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state" +# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3. +offsets.topic.replication.factor=1 + +# Enable auto creation of topic on the server. If this is set to true +# then attempts to produce, consume, or fetch metadata for a non-existent +# topic will automatically create it with the default replication factor +# and number of partitions. +auto.create.topics.enable=true + +# If this is enabled the controller will automatically try to balance +# leadership for partitions among the brokers by periodically returning +# leadership to the "preferred" replica for each partition if it is available. +auto.leader.rebalance.enable=true + +# Number of threads used to replicate messages from leaders. Increasing this +# value can increase the degree of I/O parallelism in the follower broker. +# This is useful to temporarily increase if you have a broker that needs +# to catch up on messages to get back into the ISR. +num.replica.fetchers=1 + +############################# Log Retention Policy ############################# + +# The following configurations control the disposal of log segments. The policy +# can be set to delete segments after a period of time, or after a given size +# has accumulated. A segment will be deleted whenever *either* of these +# criteria are met. Deletion always happens from the end of the log. + +# The minimum age of a log file to be eligible for deletion due to age +log.retention.hours=168 + +# A size-based retention policy for logs. Segments are pruned from the log as long as the remaining +# segments don't drop below log.retention.bytes. Functions independently of log.retention.hours. +log.retention.bytes=268435456 + +# Log retention window in minutes for offsets topic. If an offset +# commit for a consumer group has not been recieved in this amount of +# time, Kafka will drop the offset commit and consumers in the group +# will have to start a new. This can be overridden in an offset commit +# request. +offsets.retention.minutes=10080 + +############################# Zookeeper ############################# + +# Zookeeper connection string (see zookeeper docs for details). +# This is a comma separated host:port pairs, each corresponding to a zk +# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002". +# You can also append an optional chroot string to the urls to specify the +# root directory for all kafka znodes. +zookeeper.connect=localhost:2181/kafka + + +##################### Confluent Proactive Support ###################### +# If set to true, and confluent-support-metrics package is installed +# then the feature to collect and report support metrics +confluent.support.metrics.enable=false + +# The customer ID under which support metrics will be collected and +# reported. +# +# When the customer ID is set to "anonymous" (the default), then only a +# reduced set of metrics is being collected and reported. +# +# Confluent customers +# ------------------- +# If you are a Confluent customer, then you should replace the default +# value with your actual Confluent customer ID. Doing so will ensure +# that additional support metrics will be collected and reported. +# +confluent.support.customer.id=anonymous diff --git a/puppet/modules/kafka/templates/systemd/kafka.erb b/puppet/modules/kafka/templates/systemd/kafka.erb index d533515..ffcfb19 100644 --- a/puppet/modules/kafka/templates/systemd/kafka.erb +++ b/puppet/modules/kafka/templates/systemd/kafka.erb @@ -6,7 +6,7 @@ [Service] User=kafka Group=kafka -Environment="KAFKA_HEAP_OPTS=-Xmx164m -Xmx164m" +Environment="JAVA_OPTS=-Djava.awt.headless=true KAFKA_HEAP_OPTS=-Xmx164m -Xmx164m" ExecStart=/usr/bin/kafka-server-start /etc/kafka/server.properties Restart=always diff --git a/puppet/modules/role/settings/kafka.yaml b/puppet/modules/role/settings/kafka.yaml index fab9f85..19f8ea9 100644 --- a/puppet/modules/role/settings/kafka.yaml +++ b/puppet/modules/role/settings/kafka.yaml @@ -1 +1,4 @@ vagrant_ram: 128 +forward_ports: + 9092: 9092 + 9093: 9093 -- To view, visit https://gerrit.wikimedia.org/r/404870 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I93d7c7cb98664e3e41b5a383ba8f9976a0b09099 Gerrit-PatchSet: 3 Gerrit-Project: mediawiki/vagrant Gerrit-Branch: master Gerrit-Owner: Ottomata <ao...@wikimedia.org> Gerrit-Reviewer: BryanDavis <bda...@wikimedia.org> Gerrit-Reviewer: Dduvall <dduv...@wikimedia.org> Gerrit-Reviewer: Ottomata <ao...@wikimedia.org> Gerrit-Reviewer: jenkins-bot <> _______________________________________________ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits