In the previous post, we looked at how to build a three cluster Zookeeper ensemble. However, the ensemble was not secured in any way. This would allow unauthorised clients to query Zookeeper and to push data to znodes. It also allows unauthorised Zookeeper instances to join the ensemble and potentially even instruct the cluster to shut down.
Even in secured networks, it’s a good idea to use some of the security features available in Zookeeper. In this post we’ll look at two security mechanisms: mutual TLS (mTLS) and SASL authentication. We’ll set up these security features on the server-server communication (leader election protocols) and client-server communication (Kafta to Zookeeper).
mTLS for quorum protocol
Mutual TLS (mTLS) is a mechanism to allow communication only where both parties trust each other. If this is enabled for the leader election protocol (Zookeeper server to Zookeeper server) then only trusted servers can join the ensemble. To enable mTLS, every Zookeeper server must have a signed certificate and a trust store containing the certificates of other servers in the ensemble.
Let’s start by creating the certificates, keystores and the trust store. Assuming we have Zookeeper running on three servers called:
- zookeeper-1.europe-north1-a.c.zookeeper-12345.internal
- zookeeper-2.europe-north1-b.c.zookeeper-12345.internal
- zookeeper-3.europe-north1-c.c.zookeeper-12345.internal
Use the Java keytool to create a self-signed keystore each server and a trust store containing all three certificates:
HOSTNAME=zookeeper-1.europe-north1-a.c.zookeeper-12345.internal
PASSWORD=keypa55
keytool -genkeypair -alias $HOSTNAME -keyalg RSA -keysize 2048 -dname "cn=$HOSTNAME" -keypass $PASSWORD -keystore $HOSTNAME-zk-keystore.jks -storepass $PASSWORD -validity 3650
keytool -exportcert -alias $HOSTNAME -keystore $HOSTNAME-zk-keystore.jks -file $HOSTNAME.cer -rfc
keytool -importcert -alias $HOSTNAME -file $HOSTNAME.cer -keystore zk-truststore.jks -storepass $PASSWORD
Run the above for each HOSTNAME to produce four files:
- zookeeper-1.europe-north1-a.c.zookeeper-12345.internal-zk-keystore.jks
- zookeeper-2.europe-north1-b.c.zookeeper-12345.internal-zk-keystore.jks
- zookeeper-3.europe-north1-c.c.zookeeper-12345.internal-zk-keystore.jks
- zk-truststore.jks
Copy the files to the three Zookeeper servers and then add the following to each zoo.cfg:
sslQuorum=true
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
ssl.quorum.keyStore.location=/usr/local/zookeeper/conf/zookeeper-1.europe-north1-a.c.zookeeper-12345.internal-zk-keystore.jks
ssl.quorum.keyStore.password=keypa55
ssl.quorum.trustStore.location=/usr/local/zookeeper/conf/zk-truststore.jks
ssl.quorum.trustStore.password=keypa55
(adjust the ssl.quorum.keyStore.location property to pick the keystore for this server)
Restart the three Zookeeper instances to enable (and enforce) mTLS for server-server communication.
SASL Authentication for quorum protocol
Zookeeper supports server-server mutual authentication using Simple Authentication and Security Layer (SASL). It supports Kerberos and Digest-MD5 schemes. Kerberos is the stronger authentication scheme but requires additional infrastructure. This example uses the simpler Digest-MD5 scheme which just requires some usernames and passwords to be set up.
First, add some config to zoo.cfg to enable (and require) SASL:
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=true
quorum.auth.serverRequireSasl=true
quorum.auth.kerberos.servicePrincipal=servicename/_HOST
quorum.cnxn.threads.size=20
Then create a jaas.conf file containing the our username and password:
QuorumServer {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_zkquorumlearner="zkqlpa55";
};
QuorumLearner {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="zkquorumlearner"
password="zkqlpa55";
};
In this case our our Quorum Learner has username zkquorumlearner and password zkqlpa55. The Quorum Server is configured to accept that credential.
Finally, enable the JAAS config by adding it to Zookeeper’s SERVER_JVMFLAGS. The easiest way is to add this to java.env in /usr/local/zookeeper/conf/ (create the file if it doesn’t exist):
SERVER_JVMFLAGS="-Djava.security.auth.login.config=/usr/local/zookeeper/conf/jaas.conf"
Again, restart the three Zookeeper instances for this to take effect.
mTLS for client connections
Zookeeper supports the same security mechanisms for client-server communications. In this example we want to secure communication with a Kafka client.
To enforce mTLS for client communication, add the following to the zoo.cfg on all three servers:
secureClientPort=2182
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
authProvider.x509=org.apache.zookeeper.server.auth.X509AuthenticationProvider
ssl.keyStore.location=/usr/local/zookeeper/conf/zookeeper-1.europe-north1-a.c.zookeeper-12345.internal-zk-keystore.jks
ssl.keyStore.password=keypa55
ssl.trustStore.location=/usr/local/zookeeper/conf/zk-truststore.jks
ssl.trustStore.password=keypa55
Note that we’re using the same keystore / truststore files that we created earlier. Again, make sure that the ssl.keyStore.location is the keystore for this server.
Add the corresponding configuration to the Kafka server.properties
# Required to use TLS-to-ZooKeeper (default is false)
zookeeper.ssl.client.enable=true
# Required to use TLS-to-ZooKeeper
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
# Define key/trust stores to use TLS-to-ZooKeeper; ignored unless zookeeper.ssl.client.enable=true
zookeeper.ssl.keystore.location=/usr/local/zookeeper/conf/zookeeper-1.europe-north1-a.c.zookeeper-12345.internal-zk-keystore.jks
zookeeper.ssl.keystore.password=keypa55
zookeeper.ssl.truststore.location=/usr/local/zookeeper/conf/zk-truststore.jks
zookeeper.ssl.truststore.password=keypa55
# Tells broker to create ACLs on znodes
zookeeper.set.acl=true
SASL authentication for client connections
Finally, we’ve already enabled SASL authentication in Zookeeper so using it for client connections is as simple as adding a new entry to the existing jaas.conf file:
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_kafka="kafkapa55";
};
Create a corresponding kafka_jaas.conf file for Kafka, this one contains the credentials that the client will authenticate with:
Client {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="kafka"
password="kafkapa55";
};
Use the KAFKA_OPTS environment variable to enable this in Kafka:
export KAFKA_OPTS="-Djava.security.auth.login.config=/usr/local/kafka/config/kafka_jaas.conf"
Now restart all three Zookeeper instances and Kafka. The Kafka logs should show that it’s using Digest-MD5 authentication:
INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
INFO Client successfully logged in. (org.apache.zookeeper.Login)
INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
INFO Opening socket connection to server zookeeper-2.europe-north1-b.c.zookeeper-12345.internal/10.166.0.3:2182. (org.apache.zookeeper.ClientCnxn)
INFO SASL config status: Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
INFO SSL handler added for channel: [id: 0x661af545] (org.apache.zookeeper.ClientCnxnSocketNetty)
INFO Socket connection established, initiating session, client: /10.166.0.2:42300, server: zookeeper-2.europe-north1-b.c.zookeeper-12345.internal/10.166.0.3:2182 (org.apache.zookeeper.Clien$
INFO channel is connected: [id: 0x661af545, L:/10.166.0.2:42300 - R:zookeeper-2.europe-north1-b.c.zookeeper-12345.internal/10.166.0.3:2182] (org.apache.zookeeper.ClientCnxnSocketNetty)
INFO Session establishment complete on server zookeeper-2.europe-north1-b.c.zookeeper-12345.internal/10.166.0.3:2182, session id = 0x2000004d3be0001, negotiated timeout = 18000 (org.apache.$
INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[…] « Microsoft Bot Framework Part 3: Connect to channels Securing a Zookeeper ensemble » […]
Hi! First of all, thanks for this post, helped my a lot and sorry to bother this way.
Question, in your test/example/scenario, brokers comes up directly? what i mean with directly, zookeepers all comes up, no problem.
Brokers tries to come up but i receive ‘NoAuth’ errors (for acl/group, acl/topic) and the broker shuts down.
I’ve been searching and reading and it’s supposed there’s no need to manually set any ACL in this process (it’s later when clients tries to connect to the broker).
Have you set ACL entries to get this comming up?
Hi Alexis
In my example, all the brokers came up directly. I did set ACL entries to get the brokers up. That’s the QuorumServer and QuorumLearner sections in jaas.conf.