Scott Lamb

Summary

Experienced backend developer and hands-on tech lead (TL) who builds reliable, fast, secure distributed systems and strong teams.

Experience

Staff Software Engineer, Google (Jun 2016–Jan 2021)

Senior Software Engineer, Google (May 2010–Jun 2016)

Software Engineer, Google (Jul 2007–May 2010)

Some favorite accomplishments during my time as a Site Reliability Engineer (SRE) then Software Engineer (SWE):

• For five years, led an Identity Infrastructure SWE team of ~6 responsible for C++ servers in the critical path of every signed-in request at Google. We focused on security, reliability, latency, and efficiency. I reviewed and authored designs and code. I set and documented our coding and review standards, deployment procedures, and diagnostic techniques. I mentored teammates. I gave talks and led live debugging workshops.
• Saved the company many times my salary each year in efficiency improvements. I wrote tools to easily manage production quotas and job sizes. I found CPUs "stranded" during a company-wide migration. I load tested and fixed problems that had prevented safely increasing my servers' CPU utilization. I found large opportunities in my own teams' CPU profiles and small opportunities in company-wide CPU profiles. I optimized code and mentored others as they did the same.
• Migrated Gmail's last user from our older storage to a new, during my three years as a Gmail backend SWE. I led a subteam of three in writing a more reliable migration tool that allowed us to shut down multiple poorly understood legacy systems and delete millions of lines of code.
• Led Google Accounts SRE for two years. My seven-person team was responsible for reliability of a multi-tiered system including a distributed database (older paper) in the critical path of many Google products. I trained teammates, proposed projects, gave technical advice, and liaised with other teams.
• Participated in primary on-call rotations for many products and was an escalation contact for key systems. I set up automated white-box and black-box monitoring. I diagnosed production problems by a combination of tools and code inspection.
• Designed, reviewed, implemented, and ran software written in C++, Java, Python, and Go, including both critical path and management systems.
• Became a skilled user of generations of Google technology including Chubby, MapReduce, Sawzall, GFS, Colossus, Bigtable, Spanner, Dremel, Borgmon, Monarch, and GWP.

Software Developer, 2Wire (May 2005–Jul 2007)

• 1 year as member of ~10-person system software team for the HomePortal, a high-end home DSL router. Our FreeBSD-based system contains three million lines of C. I developed new userspace code, such as a feature to retain arbitrary integer statistics gathered at frequent intervals in a ring buffer for less frequent collection by 2Wire's management server (CMS). I adapted open source code to run on unforgiving environment—no MMU, limited RAM and CPU, long uptimes. I created infrastructure for our new white box testing group. Configured and enhanced Buildbot, a distributed continuous integration framework. Created robust framework for flashing new code and launching commands over a serial console after every checkin.
• 1 year as member of ~6-person developing CMS, a Java-based management server for millions of HomePortals and other devices. Replaced complex threaded handler for proprietary protocol (OGMP) with much smaller asynchronous design, doubling performance and eliminating many races and deadlocks found in production. Refactored existing OGMP load tester to verify these claims, reusing new protocol code. Replaced broken 10,000-line threaded Java load tester for HTTP-based protocol (CWMP) with working 500-line Python-Twisted design. This tool has since been used by our QA team to find many performance and synchronization bugs.

Database Programmer, UI Hospitals & Clinics, Internal Medicine (Sep 2001–Mar 2005)

Member of five-person team supporting Oracle database needs of over 100 active users. Gathered requirements from users, designed schemas, implemented front ends, documented systems with writing and diagrams, presented systems to users, and created database-driven reports. Maintained production systems. Trained new developers. Led others. Migrated data.

Open source

Moonfire NVR (github.com/scottlamb/moonfire-nvr, 600+ ⭐s, 10 contributors)

Primary author of this security camera network video recorder. Robust Rust/SQLite-based backend supports 16+ 5MP RTSP cameras on inexpensive hardware such as the Raspberry Pi 4. Javascript/React-based frontend.