HBase Off-Heap Today
HBase is a widely used distributed NoSQL database. Many workloads—feeds, ads, and similar—demand high throughput and low latency. HBase 2.0 off-heaped the core read and write paths: allocations go to JVM off-heap memory, which is not GC-managed and must be freed explicitly. On the write path, request buffers are allocated off-heap until data is written to the WAL and memstore. The memstore’s ConcurrentSkipListMap holds references to cells, not cell bodies; actual data lives in MSLAB chunks for easier off-heap management. On the read path, BucketCache is off-heap; on miss, blocks are read from HFile, encoded as cells, and returned—with little heap allocation along the way.
Recent benchmarks at Xiaomi showed that a 100% Get workload still suffered noticeably from Young GC. In HBASE-21879, Get p999 latency tracks G1 Young GC pause time—both around 100ms. After HBASE-11425, allocation was supposed to be off-heap with almost no heap traffic. Code review showed that reading blocks from HFile still copied into the heap until BucketCache’s WriterThread flushed them off-heap and released the heap DataBlock. With disk-heavy tests and ~70% cache hit rate, many blocks miss cache, heap young-gen objects pile up, and Young GC pressure rises.
The direct fix is to read DataBlocks from HFile straight into off-heap. The original gap was partly because HDFS lacked ByteBuffer pread (HDFS-3246 followed that). Another issue: DataBlocks read on the RPC path land in BucketCache’s RamCache map first; once in the map other RPCs can hit them, so the current RPC cannot free the buffer until RamCache drops it too. You need to track whether a buffer is still referenced by any RPC path or RamCache before freeing. Reference counting on ByteBuffer is the natural approach—and Netty already implements much of this, which led us to study Netty’s memory management.
Netty Memory Management Overview
Netty minimizes GC impact by using off-heap heavily. Off-heap memory must be allocated and freed by the application; leaks come from forgetting to free or freeing too early, so a good allocator matters. A “good” allocator should:
- Be high-concurrency and thread-safe—a process usually shares one global allocator.
- Allocate and free efficiently.
- Track lifetimes and help debug leaks.
- Use memory efficiently—some allocators fragment until they cannot satisfy moderately large requests; finer management improves utilization.
- Keep physically contiguous backing where possible. If 70MB contiguous is unavailable, some allocators stitch fragments; better algorithms preserve contiguity and improve access speed.
To reduce contention, PooledByteBufAllocator defaults to one memory pool per CPU core; threads hash to a pool so cores mostly use independent pools under load.
Netty’s design is granular. Memory is split into 16MB chunks; each chunk has 2048 pages of 8KB. Each request is power-of-two aligned—150B becomes 256B. Before entering cache, a page serves only one allocation: after 256B is taken from an 8KB page, further requests use other empty pages. That sounds wasteful, but after pages enter cache they can be subdivided further (e.g. 31 more 256B buffers from the same page).
Chunks form ChunkLists by utilization (used memory / 16MB × 100%). The figure shows six lists; e.g. q050 holds chunks at 50–100% utilization. As usage changes, chunks move between lists—favoring allocations from relatively idle chunks.
Example: object A allocates 150B, aligned to 256B. When A is freed, the page goes to cache—here TinySubPagesCaches, a set of queues by subdivision size (e.g. 32B). Requests for that size take a partially filled page from the matching queue. The same page can serve many equal-size allocations, so utilization stays high.
Caches split by element size inside a page (elemSizeOfPage = 8KB / elemSizeOfPage): TinySubPagesCaches for <512B, SmallSubPagesDirectCaches for <8KB, NormalDirectCaches for <16MB. On miss, search the six ChunkLists; if all are full, allocate a new chunk from off-heap.
Contiguity Inside a Chunk (Cache Coherence)
Above the chunk level, Netty balances allocation/free efficiency and utilization. Inside a chunk: to allocate 32KB efficiently while keeping access fast?
A simple approach: 2048 × 8KB pages in a queue—allocate dequeue, free enqueue—O(1), but a 32KB object spans four non-contiguous pages and hurts access.
Netty’s chunk allocator optimizes both efficiency and access. Keeping user allocations on contiguous physical memory is cache coherence in their terminology.
A 16MB chunk’s 2048 pages form a complete binary tree (heap-like), stored in int[] map: map[1] is root, map[2] left child, map[3] right child, … map[2048]–map[4095] are leaves mapping to pages.
Any subtree’s pages are contiguous in physical memory. Finding four free pages for 32KB becomes finding a subtree with four free leaves—cache coherence for the user.
Efficiency: each node id stores map[id] = depth of the first fully free subtree found in level order under id. In step 3 of the figure, for id=2 the first fully free subtree is rooted at id=5 with depth 2, so map[2]=2.
The figure walks allocate 8KB, 32KB, 16KB on a 64KB chunk (8 pages). Allocate and free are O(log N) with N=2048—effectively constant.
Netty thus keeps multi-page allocation efficient and user memory contiguous.
Reference Counting and Leak Detection
HBase’s ByteBuf path uses reference counting: refCount++ on retain, refCount– on release, recycle at zero—with thread safety considered.
Even with ref counts, missing a decrement is easy. Netty’s ResourceLeakDetector (when enabled) tracks outstanding ByteBufs and flags possible leaks when the tracked set grows too large. Production usually disables it; enable when investigating leaks.
Summary
Netty’s memory management is a useful reference for HBase off-heap work. HBase today has at least three allocators:
- RPC off-heap allocator—simple fixed 64KB pages; falls back to heap when the pool is exhausted.
- Memstore MSLAB—similar idea; could merge with (1).
- BucketAllocator for BucketCache.
For (1) and (2), adopting Netty’s PooledByteBufAllocator looks reasonable given concurrency, utilization, and cache-coherence optimizations. BucketCache can sit on memory, SSD, or HDD; BucketAllocator abstracts (offset, len) pairs that Netty’s API does not cover, so that layer likely stays as is.
HBase 2.x performance should keep improving, especially GC impact on p999.
References