Hi there 👋

I have been working at a US company for almost a month recently. An interesting fact is that I have counted the SaaS services I have used for specific scenarios since joining, and surprisingly, there are as many as 23 of them. All of these SaaS services are provided by third-party companies. In other words, Databricks is a customer of all these companies. This aroused my curiosity and made me want to learn about the situation of these companies....

I recently had the pleasure of joining Databricks, but I often reflect on my profoundly rewarding experience at ByteDance. Before joining ByteDance, I was a software engineer who had written code for a decade. My core responsibility was to tackle system design and code development for complex software while ensuring robust operations in production environments. At ByteDance, however, I led a 16-person technical team (mostly Senior+ engineers) responsible for building the competitiveness of ByteDance’s public cloud EMR open-source engines—a role that was immensely challenging yet transformative for me....

This is the second article of my thoughts for Iceberg Summit 2025 (Here is Part-1), which is not limited to Iceberg but focuses on Data Lake. I am trying to share what these creative teams are doing in this field, why it is a problem, what the solution is, and how it will develop in the future. These insights come from various sharing, discussions, and debates, but are limited by my personal understanding....

I attended the Apache Iceberg Summit 2025 in the US in the past two days. The conference was held at the Hyatt Regency Hotel next to Union Square in downtown San Francisco. This time, two floors of the hotel were rented as the venue. Mainstream companies in the Data field are basically sponsors of the conference, including AWS, Databricks, Snowflake, Cloudera, Dremio, Microsoft, Airbyte, CelerData, Confluent, eltloop, Google, IMB, Minio, Qlik, Redpanda, Starburst, StreamNative, e6data, daft, PuppyGraph, RisingWave, Telmai, wherobots....

I recently read Microsoft’s 2015 VLDB paper, Real-Time Analytical Processing with SQL Server [1]. SQL Server was among the earlier products to ship and productionize an HTAP row/column update design. Row-store (row-wise index) tradeoffs for OLTP are well understood; efficient millisecond-scale column store updates have several designs in the wild—Kudu [2], Positional Delta Tree [3], and others—but they receive far less discussion than row stores. After a careful review of SQL Server’s approach, I find it worth sharing....

I joined TiDB’s Hackathon 2019 and finally have time to write this recap. A 48-hour hackathon around TiDB: build a demo, present in six minutes. Scoring weights practicality, ease of use, and performance for TiDB (40%), completeness (30%), innovation (20%), and presentation (10%). Our team of three: captain Yi Wu from PingCAP’s US office (ex-Facebook RocksDB, now RocksDB at PingCAP), and Bokang Zhang in Beijing (TiKV). All three work on storage, so we aimed at the bottom of the stack....

How good is HBase 2.x write performance? Let’s benchmark it. Test environment: five-node cluster; each node has twelve 800GB SSDs, 24 CPU cores, 128GB RAM. HBase and HDFS are co-located—RegionServer and DataNode on the same host for better write locality (at least one replica local). Software: HBase 2.1.2, HDFS 2.6.0, OpenJDK 1.8.0_202. Per RegionServer we use 50GB heap and 50GB off-heap (100GB total)—heap mainly for memstore (~36GB), off-heap mainly for BucketCache (~36GB)....

HBase 2.0.0 shipped on April 30, 2018—about 15 months ago as of this writing. HBase 2.0.x is EOL with no further releases. HBase 2.1 is at 2.1.6 and likely will not be maintained much longer. Stable 2.x lines will be 2.2.x and 2.3.x, especially 2.2.x as the version battle-tested at large shops. Work I think deserves focus on HBase 2.x: Procedure V2 and Assignment V2 give distributed workflows atomicity through a framework—a pain point in 1....

After working in the Apache HBase community for a while, I noticed a few relentlessly productive leaders: Michael Stack, contributing day in and day out for a decade, and Duo Zhang, recently promoted to HBase project chair. Stack—a senior engineer I could call “uncle” by age—works across my entire day: morning mail shows a JIRA reply, afternoon another patch, late night comments on someone else’s patch. Duo Zhang—“Duo God” in Lei Jun’s blog—sits to my right....

The third Apache HBaseConAsia summit takes place in Beijing on July 20. As the Apache HBase community’s premier user conference in Asia, HBaseCon has run since 2012. More than 20 experts and community leaders from leading internet and big-data companies will share the latest on HBase and the surrounding ecosystem. Apache HBase is a highly available, high-performance, multi-version distributed NoSQL database on Apache Hadoop—Google Bigtable’s open-source counterpart. It delivers fast random read/write at scale on commodity servers....