HBase

How good is HBase 2.x write performance? Let’s benchmark it. Test environment: five-node cluster; each node has twelve 800GB SSDs, 24 CPU cores, 128GB RAM. HBase and HDFS are co-located—RegionServer and DataNode on the same host for better write locality (at least one replica local). Software: HBase 2.1.2, HDFS 2.6.0, OpenJDK 1.8.0_202. Per RegionServer we use 50GB heap and 50GB off-heap (100GB total)—heap mainly for memstore (~36GB), off-heap mainly for BucketCache (~36GB)....

HBase 2.0.0 shipped on April 30, 2018—about 15 months ago as of this writing. HBase 2.0.x is EOL with no further releases. HBase 2.1 is at 2.1.6 and likely will not be maintained much longer. Stable 2.x lines will be 2.2.x and 2.3.x, especially 2.2.x as the version battle-tested at large shops. Work I think deserves focus on HBase 2.x: Procedure V2 and Assignment V2 give distributed workflows atomicity through a framework—a pain point in 1....

After working in the Apache HBase community for a while, I noticed a few relentlessly productive leaders: Michael Stack, contributing day in and day out for a decade, and Duo Zhang, recently promoted to HBase project chair. Stack—a senior engineer I could call “uncle” by age—works across my entire day: morning mail shows a JIRA reply, afternoon another patch, late night comments on someone else’s patch. Duo Zhang—“Duo God” in Lei Jun’s blog—sits to my right....

The third Apache HBaseConAsia summit takes place in Beijing on July 20. As the Apache HBase community’s premier user conference in Asia, HBaseCon has run since 2012. More than 20 experts and community leaders from leading internet and big-data companies will share the latest on HBase and the surrounding ecosystem. Apache HBase is a highly available, high-performance, multi-version distributed NoSQL database on Apache Hadoop—Google Bigtable’s open-source counterpart. It delivers fast random read/write at scale on commodity servers....

Motivation Databases need flexible query patterns. A KV store with only Get, Put, and Scan would frustrate users—real workloads are richer. For orders: “this user’s orders in the last three months” needs at least (1) filter by user and (2) filter by time range, combined with AND. Scanning the whole table on the client and filtering locally would crush the cluster. Server-side filters are required. You also see OR, e.g. orders for Alice or Bob in the last three months—AND vs OR, often mixed....

In HBASE-21879, we redesigned the offheap read path: read the HFileBlock from HDFS to pooled offheap ByteBuffers directly, while before HBASE-21879 we just read the HFileBlock to heap which would still lead to high GC pressure. After few months of development and testing, all subtasks have been resovled now except the HBASE-21946 (It depends on HDFS-14483 and our HDFS teams are working on this, we expect the HDFS-14483 to be included in hadoop 2....

HBase Off-Heap Today HBase is a widely used distributed NoSQL database. Many workloads—feeds, ads, and similar—demand high throughput and low latency. HBase 2.0 off-heaped the core read and write paths: allocations go to JVM off-heap memory, which is not GC-managed and must be freed explicitly. On the write path, request buffers are allocated off-heap until data is written to the WAL and memstore. The memstore’s ConcurrentSkipListMap holds references to cells, not cell bodies; actual data lives in MSLAB chunks for easier off-heap management....

HBaseConWest2018 was held on June 18 in San Jose, California, hosted by Hortonworks. Attending HBaseCon West in Silicon Valley each year has become routine for the Xiaomi HBase team—our community presence is well known (seven HBase Committers, two PMC members), and the company is willing to share a year-in-review of internal practice and community contributions. In 2018 we submitted the talk “HBase Practice at Xiaomi,” spent considerable effort preparing it, and rehearsed in English three times internally....

On October 20, I accepted an invitation from the Apache HBase community and became an HBase Committer. At Xiaomi I maintain our internal HBase branch and production clusters, and the company already had six HBase Committers—including one PMC member—so becoming a Committer was a natural next step rather than something extraordinary. Compared with someone doing HBase at a company with no Committers, it takes considerably more time and effort. Some observations about the community:...

HBase is a distributed key-value database that supports automatic load balancing. With the balance switch (balance_switch) enabled, the HMaster process automatically selects regions according to a specified policy and assigns them to RegionServers with lower load. The official distribution currently supports two region-selection policies: DefaultLoadBalancer and StochasticLoadBalancer, both described in detail below. Because all HBase data (including HLog, meta, HStoreFile, and so on) is written to HDFS, region moves are very lightweight....