HBase Region Balance in Practice

HBase is a distributed key-value database that supports automatic load balancing. With the balance switch (balance_switch) enabled, the HMaster process automatically selects regions according to a specified policy and assigns them to RegionServers with lower load. The official distribution currently supports two region-selection policies: DefaultLoadBalancer and StochasticLoadBalancer, both described in detail below. Because all HBase data (including HLog, meta, HStoreFile, and so on) is written to HDFS, region moves are very lightweight. During a region move, the HDFS files for the region stay in place; only the region’s metadata is reassigned to the target RegionServer. The move depends on RegionClose and RegionOpen latency, which is generally short.

This article covers how HBase balancing works.

Balance Flow

• First, the LoadBalancer identifies all region plans that need to move. Each plan includes three attributes: region, source RegionServer, and destination RegionServer.
• Unassign the region—remove it from the original RegionServer.
• Assign the region—bind it to the target RegionServer.

The unassign flow works as follows:

• Create a ZK closing node under /unassigned, containing znode state, region name, original RegionServer name, and payload.
• HMaster calls an RPC to close the region on the RegionServer. Region close roughly: acquire the region write lock, flush the memstore, concurrently close all store files under the region (note that one region has multiple stores, and each store has multiple store files, so store files can be closed in parallel), then release the region write lock.
• Set the ZK closing node’s znode state to closed.

The assign flow works as follows:

• Obtain the corresponding Region Plan.
• HMaster calls an RPC on the target RegionServer to open the region. It first updates the /unassigned node to opening, concurrently loads HStores, updates zk/ROOT/META so clients get correct routing on the next request, and finally sets the region state to OPEN.

DefaultLoadBalancer Policy

This policy keeps the number of regions on each RegionServer roughly equal. More precisely, if there are n RegionServers and the i-th RegionServer has Ai regions, let average = sigma(Ai)/n. Then every RegionServer’s region count falls within [floor(average), ceil(average)]. The implementation is simple and widely used.

However, this policy considers only one factor and ignores per-RegionServer read/write QPS and load pressure. You can end up with roughly equal region counts while 90% of requests hit one RegionServer because all of its regions are hot spots—so load balancing still fails. I think the primary goal of balancing is data distribution; if load remains concentrated despite balanced data, the row key design may need review. Personally, I still recommend DefaultLoadBalancer.

StochasticLoadBalancer Policy

StochasticLoadBalancer is quite complex. In short, it balances six weighted factors:

• Read request count per RegionServer (ReadRequestCostFunction)
• Write request count per RegionServer (WriteRequestCostFunction)
• Region count per RegionServer (RegionCountSkewCostFunction)
• Move cost (MoveCostFunction)
• Data locality (TableSkewCostFunction)
• Per-table region count cap per RegionServer (LocalityCostFunction)

For each cluster region layout, a weighted sum of these six factors produces a cost value that measures how balanced the layout is—lower cost means more balanced. The balancer then runs hundreds of thousands of random iterations to find a sequence of region moves that strictly decreases cost. That sequence is what HMaster executes.

Pseudocode for the iteration:

currentCost = MAX ; 
plans  = []
for(step = 0 ; step < 1000000; step ++ ){
	action = cluster.generateMove() 
	doAction( action );
	newCost  = computeCost(action) ;
	if (newCost < currentCost){
		currentCost = newCost;
		plans.add( action );
	}else{
		undoAction(action);
	}
}

generateMove() randomly picks one of three strategies each time:

1. Randomly choose two RegionServers, pick one region from each, and generate an action that is either a RegionMove (from the RegionServer with more regions to the one with fewer) or a RegionSwap (exchange regions between the two), each with 50% probability.
2. Choose the RegionServer with the most regions and the one with the fewest, then generate an action that is either RegionMove or RegionSwap with 50% probability each.
3. Pick a random RegionServer, find the region with the worst locality on that server, find the RegionServer where most of that region’s data lives, and generate an action to move the region there to improve locality.

The JavaDoc claims good results, but I think that needs test data to back it up—the project does not publish much here. If the weights for the six factors are wrong, the cluster can stay badly unbalanced. In one production incident with the balance config below, each balance cycle moved only about 60 plans while the cluster had 145 RegionServers—one with 700+ regions and others with as few as 2, with counts spread between 2 and 700. Far more balancing should have been needed, but HMaster generated only ~60 plans per period, so balancing was too slow and load stayed uneven for a long time.

hbase.master.loadbalancer.class=org.apache.hadoop.hbase.master.StochasticLoadBalancer
hbase.master.balancer.stochastic.regionCountCost=10
hbase.master.balancer.stochastic.tableSkewCost=5
hbase.master.balancer.stochastic.readRequestCost=5
hbase.master.balancer.stochastic.writeRequestCost=5
hbase.master.balancer.stochastic.localityCost=10
hbase.master.balancer.stochastic.moveCost=4
hbase.master.balancer.stochastic.maxMovePercent=1

Compared with the default weights, regionCountCost was probably too low. Unless you have offline test results showing that a given weight configuration behaves as expected, it is hard to tune weights confidently in production. For such a complex policy, proceed carefully and prefer decisions backed by historical test data on balance effectiveness.