Zookeeper hardly needs an introduction — it is a distributed coordination service. A few questions I was curious about:

  • How does ZooKeeper implement asynchronous watcher callbacks? (code-level details)
  • How does ZooKeeper implement distributed locks?
  • How do Queue, barrier, and similar recipes work?

I read the Python client kazoo and got the general picture.

A simple client example

#!/usr/bin/python
import logging
from time import sleep
from kazoo.client import KazooClient

# print log to console
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)

zk = KazooClient('127.0.0.1:2181')
zk.start()

def children_callback(children):
    print '****' , children

children = zk.get_children('/zookeeper', children_callback)

zk.create('/zookeeper/goodboy')
#zk.delete('/zookeeper/goodboy')

while True: sleep(1)

How Kazoo handles asynchrony

First, a premise: every request from a client carries an xid. Each new request increments xid by 1, and the ZooKeeper server processes a single client’s requests strictly in increasing xid order before responding. Under that guarantee, the client can enqueue (request, async_object, xid) before sending (request holds the RPC details; async_object holds the callback). When any response arrives, dequeuing the head of the pending queue is enough to run the matching callback.

A more general pattern stores (request, async_object, xid) in a map keyed by xid, then looks up the entry when a response returns. Because ZooKeeper preserves order, Kazoo can get away with a single pending queue.

A few design constraints:

  • Every API can be called synchronously or asynchronously; sync can be built on top of async.
  • Watchers on each znode must fire asynchronously.
  • The main thread must not block, because it runs application code.

How Kazoo works (using the example above)

image

A few notes:

  • In step 2, the KazooClient main thread uses an OS pipe for inter-thread signaling. The main thread writes one byte to the write pipe to wake thread_1.

  • In steps 3 and 4, thread_1 uses select([socket, readpipe], [], []) to watch for read events on the socket and read pipe.

    • A readable socket means the ZooKeeper server has a response; read from the socket.
    • A readable read pipe means the main thread enqueued another request (it pushes to the queue, then writes a byte to the write pipe).

  • In step 5, thread_1 increments the client xid and sends the request to ZooKeeper, then returns to its while True loop watching the socket and write pipe — the two cases above.

  • In steps 7 and 8, when the socket is readable, read the response and compare its xid with the request at the head of the pending queue; they must match to pair request and response.

  • In steps 9 and 10, route the response to either a watcher handler (thread-2) or a normal API callback (thread-3), completing the async call.

Kazoo recipes and tricks

Most ZooKeeper patterns rely on one idea: when an ephemeral znode disappears, the server sends a ChildrenChangeEvent to every client that registered a watcher on its parent. Each live client can then run a shared algorithm to decide who acts — for example, on ChildrenChangeEvent, the client with the smallest sequence number does the work. Everyone else keeps watching whether that leader is still alive; if not, they negotiate again.

Lock

The classic use case is high availability. A stateful service should not run in multiple active instances at once. A ZooKeeper lock ensures only one machine serves traffic at a time; the rest block. When the active instance dies, the lock is released and another instance grabs it and takes over — giving HA within one ZooKeeper session timeout.

The lock recipe follows the pattern above: clients elect a leader; the leader holds the lock while others watch the leader’s parent znode with exists and block.

Semaphore

Allow at most N clients to hold the lock at once. Kazoo builds this on the lock recipe plus ephemeral nodes. You could also try registering ephemeral sequential nodes — would that work?

Counter

A distributed counter: N clients can increment concurrently. Kazoo does not use ephemeral nodes; it relies on znode versions. Clients read the same version, increment locally, and only one write succeeds (ZooKeeper’s versioning enforces this). Failed clients retry.

Barrier

A barrier blocks all N clients until some condition is met. Kazoo watches a znode with exists, then blocks on an event until that znode is deleted, which unblocks everyone.

DoubleBarrier

A double barrier: no client proceeds until all N have arrived; no client leaves until all N have left. Early arrivals or early departures stay blocked.

Party

A party is just a set of ephemeral nodes. Creating an ephemeral node joins the party; deleting it leaves. Nothing more to it.

Queue

Blocking and non-blocking variants:

  • Non-blocking queue: create child znodes under a parent and hand them to clients in lexicographic order (POP).
  • Blocking queue: concurrent PUT is allowed, but only one client may POP at a time.