Database

I recently read Microsoft’s 2015 VLDB paper, Real-Time Analytical Processing with SQL Server [1]. SQL Server was among the earlier products to ship and productionize an HTAP row/column update design. Row-store (row-wise index) tradeoffs for OLTP are well understood; efficient millisecond-scale column store updates have several designs in the wild—Kudu [2], Positional Delta Tree [3], and others—but they receive far less discussion than row stores. After a careful review of SQL Server’s approach, I find it worth sharing....

I joined TiDB’s Hackathon 2019 and finally have time to write this recap. A 48-hour hackathon around TiDB: build a demo, present in six minutes. Scoring weights practicality, ease of use, and performance for TiDB (40%), completeness (30%), innovation (20%), and presentation (10%). Our team of three: captain Yi Wu from PingCAP’s US office (ex-Facebook RocksDB, now RocksDB at PingCAP), and Bokang Zhang in Beijing (TiKV). All three work on storage, so we aimed at the bottom of the stack....

How good is HBase 2.x write performance? Let’s benchmark it. Test environment: five-node cluster; each node has twelve 800GB SSDs, 24 CPU cores, 128GB RAM. HBase and HDFS are co-located—RegionServer and DataNode on the same host for better write locality (at least one replica local). Software: HBase 2.1.2, HDFS 2.6.0, OpenJDK 1.8.0_202. Per RegionServer we use 50GB heap and 50GB off-heap (100GB total)—heap mainly for memstore (~36GB), off-heap mainly for BucketCache (~36GB)....

Nebula held its second Beijing meetup in Jingdong Beichen. CEO Xiaomeng Ye led Geobase, Ant Financial’s graph database. Two technical directors—Heng Chen and Fenglin Hou (dutor)—drive storage (similar to TiKV) and the query engine (similar to TiDB), respectively. The architecture separates storage and compute. The compute layer is SQL-like, with Go syntax for edge hops and pipelines to simplify nested subqueries. Storage is key-value with static hash partitioning by key (no dynamic partitioning yet)....

In HBASE-21879, we redesigned the offheap read path: read the HFileBlock from HDFS to pooled offheap ByteBuffers directly, while before HBASE-21879 we just read the HFileBlock to heap which would still lead to high GC pressure. After few months of development and testing, all subtasks have been resovled now except the HBASE-21946 (It depends on HDFS-14483 and our HDFS teams are working on this, we expect the HDFS-14483 to be included in hadoop 2....

HBase Off-Heap Today HBase is a widely used distributed NoSQL database. Many workloads—feeds, ads, and similar—demand high throughput and low latency. HBase 2.0 off-heaped the core read and write paths: allocations go to JVM off-heap memory, which is not GC-managed and must be freed explicitly. On the write path, request buffers are allocated off-heap until data is written to the WAL and memstore. The memstore’s ConcurrentSkipListMap holds references to cells, not cell bodies; actual data lives in MSLAB chunks for easier off-heap management....

HBaseConWest2018 was held on June 18 in San Jose, California, hosted by Hortonworks. Attending HBaseCon West in Silicon Valley each year has become routine for the Xiaomi HBase team—our community presence is well known (seven HBase Committers, two PMC members), and the company is willing to share a year-in-review of internal practice and community contributions. In 2018 we submitted the talk “HBase Practice at Xiaomi,” spent considerable effort preparing it, and rehearsed in English three times internally....

This article covers transaction isolation in TokuDB. The source implementation is complex; for clarity, we focus on the most essential parts and omit minor details. Background In traditional relational databases (Oracle, MySQL, SQL Server, and others), transactions are central to both engineering and discussion. The core properties of a transaction are ACID. A (atomicity) means a transaction’s sub-operations have only two outcomes: all succeed on commit, or all are undone on rollback....

This article analyzes TokuDB’s index structure—the fractal tree—from an engineering implementation perspective. It describes the on-disk storage layout of ft-index in detail, how ft-index implements point queries, range queries, and insert/delete/update operations, and throughout the discussion attempts a detailed comparison with InnoDB’s B+ tree from multiple angles. Introduction to the Fractal Tree The fractal tree is a write-optimized on-disk index data structure. In general, fractal trees offer good write performance (insert/update/delete) while still providing read performance close to that of a B+ tree....

Online version: PPT This is a technical talk on Vitess I gave internally at my company. The main topics are: Vitess feature modules, capabilities, and system architecture Vitess sharding approach SQL syntax supported by Vitess How Vitess resharding works Vitess data backup internals Pros and cons of Vitess compared with traditional relational databases and NoSQL A side note To try building an online PPT with Markdown, I experimented with a few options....