Rails World 2024
I attended Rails World again this year, this time in Toronto. A quick recap while it’s still fresh. What a great event. Both this year and last the organizers went out of their way to pick some of the most incredible venues I’ve ever seen. Many places are adequate to the task of containing a conference for a few days, but few make your mouth go wide with a “wow” as you walk into the place. This year’s was held...
16 days ago
TIL: Variables in custom VSCode snippets
This blog is entirely driven by Markdown, TOML, and Git. Publishing an atom or sequence involves popping open a TOML file, adding a new item to the top, committing to Git, and pushing to origin to trigger a CI action that deploys the site: [[atoms]] published_at = 2024-10-04T10:24:22-07:00 description = """\ Hello, world! """ This generally works quite well, and in this developer’s humble opinion, far preferable to something involving a web UI with a little text box, but when...
18 days ago
A few secure, random bytes without `pgcrypto`
In Postgres it’s common to see the SQL random() function used to generate a random number, but it’s a pseudo-random number generator, and not suitable for cases where real randomness is required critical. Postgres also provides a way of getting secure random numbers as well, but only through the use of the pgcrypto extension, which makes gen_random_bytes available. Pulling pgcrypto into your database is probably fine—at least it’s a core extension that’s distributed with Postgres itself—but while testing the RC...
28 days ago
Direnv's `source_env`, and how to manage project configuration
For years I’ve been using Direnv to manage configuration in projects. It’s a small program that loads env vars out of an .envrc file on a directory by directory basis, using a shell hook to load vars as you enter a folder, and unload them as you leave. A typical .envrc: export API_URL="http://localhost:5222" export DATABASE_URL="postgres://localhost:5432/project-db" export ENV_NAME=dev The beauty of Direnv is not only that it’s 12-factor friendly, but that it’s language agnostic, and unlike its language-specific alternatives that hook...
about 1 month ago
Your Go version CI matrix might be wrong
We had an unpleasant surprise this week in River’s CI suite. Since the project’s inception we thought we were supporting the latest two versions of Go (1.21 and 1.22), but it turns out that we never were. As per common convention, we had a GitHub Actions CI matrix testing against both versions: strategy: matrix: go-version: - "1.21" - "1.22" That looks kosher, right? Wrong! Builds were happily passing this whole time, but upon closer inspection of the install step, we...
2 months ago
Elden Ring
So the original Elden Ring went on sale after its DLC release, and I bought it. I almost hate to admit it, but I like it. After the terrible mistake of buying the PS5 Demon’s Souls remake, I swore to myself that I wouldn’t buy anything FromSoftware or FromSoftware-adjacent ever again. It’s too hard, too obnoxious, and too unforgiving. Like, really hard gameplay I guess I can understand, but then add in a weapon durability element whose only effect is...
3 months ago
Sqlc: 2024 check in
It’s been almost three years since I wrote about the use of sqlc in our Go stack. The other night someone asked me if I still thought that this was the right direction, so I thought I’d do a quick check in about it. We use sqlc for every database call in our app, and we’re now up to a little under 700 queries across 101 files (we use one model per file, so that’s ~100 domain models), with ~7,800...
4 months ago
Go: Don't name packages common nouns
If there’s one single, overriding request that I could make to Go package authors, it’s this: Don’t name packages after common nouns. Let me pick on time/rate as an example. Great little package, but absolutely awful name. I see why they did it. In Go, exported names in a package are referenced with their package name, and rate.Limiter sure has a nice ring to it. But it’s also bad. Why? Because rate is simple, obvious name that we might want...
5 months ago
The romance of Europe
Something I’m very guilty of at times is romanticizing Europe, even when consciously trying to correct for it. I went to a concert the other night, and one that was so Berlin. Address in a random part of the city, walk half a block down a private-looking alley, enter a nondescript door, climb four floors of stairs passed empty landings with graffiti everywhere, and enter through a door with an enormous devil’s head above it. It was a tiny venue....
5 months ago
ICQ
ICQ stops working on June 26th. I remember the first day I found out about ICQ. My friend Nick told me about it in the library at Fairview Junior High, and he pulled off an absolutely masterful job of Hitchcock-style suspense (especially given that we were seventh graders). He refused to tell me what ICQ was, only saying, “It’s amazing. It’s incredible! I can’t explain it. You have find out for yourself.” It drove me crazy. What was this thing?...
5 months ago
Notes on implementing dark mode
As you can see from the pretty new toggle at the top, I recently added dark mode to this site. I thought this was something that’d never happen because over a decade I’d built up an inescapable legacy of fundamentally unmaintainable CSS, but for over a year I’ve been slowly making headway refactoring everything to Tailwind, and with that finally done, the possibility of dark mode was unlocked. The internet is awash with tutorials on how to implement dark mode,...
5 months ago
Use of Go's `cmp.Or` for multi-field sorting
The fragment I wrote on a value helper yesterday was upstaged entirely by the fact that I’d missed a new helper that came in with Go 1.22, cmp.Or. (These minor embarassments happen occasionally, and is part of the reason I write these pieces. It’s better than be corrected than wrong!) While I’d expect cmp.Or to be mainly useful in cases like setting defaults as a one-liner, its docs suggest that another place where it slots in nicely is for use...
5 months ago
ValOrDefault
Update: It turns out that everything I wrote here has been superseded as of Go 1.22, which added cmp.Or, which works almost identically. It can be used to easily assign defaults: config = &Config{ CancelledJobRetentionPeriod: cmp.Or(config.CancelledJobRetentionPeriod, maintenance.CancelledJobRetentionPeriodDefault), CompletedJobRetentionPeriod: cmp.Or(config.CompletedJobRetentionPeriod, maintenance.CompletedJobRetentionPeriodDefault), DiscardedJobRetentionPeriod: cmp.Or(config.DiscardedJobRetentionPeriod, maintenance.DiscardedJobRetentionPeriodDefault), } Thanks @earthboundkid for the correction. Especially with respect to certain major pain points, Go’s traditionally lived by an accidental butchering of Larry Wall’s old quote of, “Make easy things easy, and hard things possible” to the...
5 months ago
Heroku on two standard dynos?
One of the aspects that I appreciate most about Go is its incredible economy around CPU and memory. Bridge’s API runs on two standard 512 MB dynos. Even with these very modest containers, only a fraction of available resources are used, with memory hovering around 40 to 50 MB, level as a laser beam. It actually runs quite well on only one dyno too. Having two of them is an availability hedge to protect against Heroku itself. We’d been running...
5 months ago
Activating cached feature flags instantly with notify triggers
In our feature flags system, flags can operate in one of three modes: Fully on. On randomly (e.g. 10% of checks made or for 10% of users). On by token (an ID of an account, cluster, or team). As an optimization, each API process keeps a local cache of the state of activation of all flags. The alternative to that would be to reach out to the database to get the current state on every flag check, and at our...
6 months ago
Shiki
Last year I redesigned most of this blog. It’d been a long time since it’s last facelift, but the more pressing problem was the rat’s nest of CSS which had become fundamentally unmaintainable – anything I changed had a high change of cascading unintentionally and breaking something else. Migrating to Tailwind was a must, and soon. The rebuild was a huge project that had to be broken up so I could work on the pieces incrementally. One that fell to...
6 months ago
Plumbing fully typed feature flags from API to UI via OpenAPI
Our architecture consists of a backend Go API layer that handles state and domain logic, and a frontend TypeScript app that provides the UI. We’re allergic to too much novelty, so like many shops, we roll out features via feature flags. Almost every new feature involves checking flag state somewhere in the backend. For example, given the flag to roll out of our recent analytics product: flags: analytics_enable: description: | Roll out flag for Crunchy Bridge for Analytics. kind: feature...
6 months ago
Digital detox
One of the things I was most looking forward to when hiking the John Muir trail last year was the opportunity for a digital detox. Like other members of my generation I was an early, enthusiastic adopter of the smartphone, but like Jonathan Haidt, over the intervening decades I’ve become increasingly skeptical of them. The digital world’s enabled fantastic things, and is convenient and entertaining beyond all reason, but its totally ubiquitous use is producing negative byproducts of world-altering magnitude...
6 months ago
Histograms worked
I wrote a few months back about how I was looking into using histograms for metrics rollups to power Crunchy Bridge’s metrics dashboards. Previously we’d been aggregating over raw data points, which worked surprisingly well, but was too slow for generating series for longer time ranges like a week or a month. I’m writing this follow up because in the original post, I said I would. In short, this approach worked unreasonable well. Our raw metrics are partitioned into a...
6 months ago
Ruby typing 2024: RBS, Steep, RBS Collections, subjective feelings
I was writing a new Ruby gem recently, and being a strong proponent of a type checking step, I wanted to do right by the ecosystem so that anyone using it would get the full benefit of type checking against my gem’s API in their own projects, so I dug into the current state of the art to find out how that’d be done. I used Sorbet for years at Stripe, and although a little unwieldy in places, it was...
6 months ago
Modals and mysterious macOS failures
I have a Mac Mini that I try to use as a headless server. The last few years have trained me such that if there’s ever a mysterious problem or failure, its most likely explanation is that there’s a new pop up back in the GUI that I’m trying not to use. The other day I upgraded Sonarr and irkingly, it stopped moving completed files. Sure enough, the next time I logged onto the computer via screen, I had a...
8 months ago
Stubbing degenerate network conditions in Go with `DialFunc` and `net.Conn`
While fixing a bug in River’s pgx driver’s listener the other day, I got to a point where I was trying to write a regression test, but found it difficult because getting the Close function on a pgxpool connection to return an error is practically impossible through normal usage. I thought I’d be able to do it with a prematurely cancelled context, but it didn’t do the trick. Reading pgx’s source, errors on Close are generally avoided in favor of...
8 months ago
Prepared statements for psql operations
Veteran database users will be familiar with prepared statements, useful for performance by storing a known query so that it only needs to be parsed once. They have some benefits beyond performance, and I’ve come to use them in an unexpected place: psql. We’re still running a pretty lean operation, and internal tooling is occasionally lacking or non-existent. Some operations require the use of a psql session, like enabling a feature flag. When doing so, we copy a query out...
8 months ago
Thoughts on ONCE + Campfire
37signals introduces its first product for ONCE, a $299 self-hosted version of Campfire. We used Campfire at Heroku in 2011. I never particularly liked it. It felt slow and had a lot of rough edges. We moved to HipChat briefly, which might’ve been slightly better at the time, but also nothing to write home about. Then we switched to Slack, and it felt like the sky opened up and a ray from heaven beamed down upon us. Truly a quantum...
9 months ago
Hard media
It’s the end of an era: Best Buy will stop selling physical media like DVDs and Blurays in 2024. Having not bought a physical disk in over a decade, I can’t rightfully complain about this, but I’m getting nostalgic thinking about it nonetheless. Like used book stores, browsing Blockbuster and Future Shop 1 are a fond part of my childhood and teenage years. On nights when company would come over we’d head off to Blockbuster and rent a movie largely...
9 months ago
Discovering histograms, kicking off metric rollups
A great post from Circonus on why aggregating percentiles doesn’t work, and explaining the use of histograms to get it right. I’m just breaking ground on a project to produce rollups for our metrics. Until now we’ve been using only the raw metric data and Postgres’ percentile_cont to aggregate it, which considering the bluntness of the instrument, has been working amazingly well. But as you might expect, it’s slow to load metrics for longer timeframes like a week or a...
9 months ago
The peanut gallery
I love these words: Simple. Minimal. Elegant. Consistent. Small. Besides sharing a theme, the other thing they have in common is that they’re regularly used in the programming world, usually as aspirational properties to pursue. At Heroku we used to throw them around all day long, and I’ve been complicit in promulgating their use ever since. A few weeks ago I was describing our internal architecture to an ex-colleague. While I consider it to be pretty good – a balance...
about 1 year ago
Optimizing row storage in 8-byte chunks
When Postgres stores physical rows, it aligns fields in 8-byte chunks 1. For 8-byte data types like a UUID or bigserial you don’t have to think about field ordering because it doesn’t matter, but when a type’s size isn’t a multiple of eight, space is lost to padding unless fields are tetris’ed to maximize storage efficiency. Say we have CREATE TABLE foo (a int4, b int8, c int4). int4s are 4 bytes, so they’re padded to get to 8-byte alignment:...
about 1 year ago
Being a good web denizen: Don't strip EXIF metadata from photos
A few days ago after posting photos from my trip on the John Muir Trail, a colleague asked what camera I was shooting with. It’s an easy question to ask with a direct line over Slack, but also one that’s quick to answer using a program like ImageMagick’s convert or any number of other tools that can read EXIF tags: $ identify -format '%[EXIF:*]' john-muir-trail-1_large@2x.jpg exif:ApertureValue=393216/65536 exif:ExposureTime=1/320 exif:FNumber=8/1 exif:FocalLength=72/1 exif:LensModel=RF24-105mm F4 L IS USM exif:LensSerialNumber=9234003630 exif:LensSpecification=24/1, 105/1, 0/1, 0/1 exif:Make=Canon...
about 1 year ago
A Postgres-friendly time comparison assertion for Go
Three major problems with trying to compare time values in Go: Times in Go carry a monotonic component, so attempting to compare them directly will more often than not produce the wrong result (e.g. using testify’s require.Equal helper). Times in Go are precise down to the nanosecond, whereas Postgres stores times to microsecond-level precision, so comparing a time to one round-tripped from Postgres will fail (say with the use of the built-in time.Equal). Times in Go are structs, so comparison...
about 1 year ago
Getting Postgres logs in a GitHub Action
I have an admission to make: I don’t read Postgres logs all that often. The majority of the time I get enough feedback in-band with Postgres operations that I don’t need to look at them. But occasionally, they’re absolutely indispensible. Recently I’ve been debugging an upsert deadlock problem in our test suite, and the deadlock detected (SQLSTATE 40P01) error message that comes back with the failed operation is entirely unactionable in every respect. The logs on the other hand, contain...
about 1 year ago
Why to prefer `t.Cleanup` to `defer` in tests with subtests using `t.Parallel`
Go’s defer statement and its t.Cleanup test function overlap each other in functionality, and at first glance seem to be interchangeable. Some time ago I came across the tparallel linter, which enforces the use of t.Cleanup over defer in parent test functions where t.Parallel() is used in subtests. I didn’t initially understand why, and the project’s README was light on details, so I opened an issue asking about it. Recently @telemachus responded to with translated content from the original Japanese...
about 1 year ago
An email redaction function for Postgres
We recently shipped a saved queries feature that shows the results of a SQL query in browser and which can produce shareable links. When sharing data, it’s crucial that PII (personally identifiable information) be redacted. PII comes in many forms, with one possible kind being email. Emails should never be shown anywhere wholesale, but even a partial email can add a lot of context, like showing the domain of an account that owns a resource. I wrote an email_redacted function:...
about 1 year ago
Rate limiting, DDOS, and hyperbole
Last week in the whacky real world sitcom that is now normal life in the 2020s, we had the brief episode of rate-limit-exceeded-gate, in which Elon added a daily limit to viewable tweets for Twitter’s users, and one that was so miserly that just about every active user hit it in the first hour, seeing their requests fail with a “rate limit exceeded” (429) error. The ostensible reason for the change was to discourage scraping, which was also the rationale...
over 1 year ago
A `TestTx` helper in Go using `t.Cleanup`
I’m a big fan of the use of test transactions. They’ve got some downsides (which I’ll dig into elsewhere), but they’re extremely fast, allow practically limitless parallelism, and remove the need for complicated and expensive cleanup subsystems. Previously, I’d been using them with a block-style test helper into which a function is injected to provide the inner body between transaction start and rollback, but I was pointed in the direction of Go’s t.Cleanup() helper which runs an arbitrary operation after...
over 1 year ago
100% test coverage
We run one of our projects not only with 100% test coverage, but 100% branch coverage. It’s deeply unpleasant. Every single change you make is likely to accidentally break test coverage, so if a normal development process is 1. add feature, and 2. write tests/fix bugs, here you have a three-step process of 1. add feature, 2. write tests/fix bugs, and 3. hunt down every omitted branch and write increasingly convoluted tests to hit them. And I know I’m not...
over 1 year ago
PGX + sqlc v4 to v5 upgrade notes
Last week, we upgraded from pgx v4 to v5. PGX v5 brings in some nice incremental improvements, with the major one being that packages have been reorganized so that pgtype, pgconn, and pgproto3 all move into the pgx Go module instead of living on their own, and that has the major advantage of making the versioning between them much less confusing. Upgrading pgx was the easy part. The hard part was upgrading sqlc to use the new pgx v5 driver,...
over 1 year ago
PG advisory locks in Go with built-in hashes
Postgres has a feature called advisory locks that allow a client to take out specific locks whose meanings are defined by an application using Postgres rather than within Postgres itself. They’re useful for app coordination, like ensuring that only one instance of a certain program starts. Because Postgres tracks lock IDs as integers internally, the advisory lock functions take a 64-bit integer as a key: pg_try_advisory_lock(int8) But on the application side, it’s common to want to use a string as...
over 1 year ago
Lean = fast
A feature notably missing from Bridge for too long was multi-factor authentication, but as of last week, it’s finally available. The reason it took so long was largely a historical accident – a decision was made early on to use Keycloak for identity management, and for a variety of reasons this tended to make account-related features slow to ship. Multi-factor was one of them, and we recommended use of Azure/Google SSO as a security stopgap until we had it. Keycloak...
over 1 year ago
Stay mainline
GitHub writes about how they stay current on mainline Rails. Takeaways: A pull request is opened automatically every week targeting Rails main. Weekly incremental updates avoid huge monolithic projects to bump a big dependency across many versions. GitHub developers always get access to the latest Rails features. Security posture is improved. All Rails monkeypatches have been unwound. Instead of monkeypatching, GitHub developers should contribute changes upstream to the public Rails project. The practice becomes a huge regression hedge for Rails...
over 1 year ago
Meta layoffs
Last week Meta announced another 10,000 upcoming layoffs. Layoffs are terrible, of course, but a couple points I found interesting. Firstly, they’re explicitly flattening the company’s structure: In our Year of Efficiency, we will make our organization flatter by removing multiple layers of management. As part of this, we will ask many managers to become individual contributors. We’ll also have individual contributors report into almost every level — not just the bottom — so information flow between people doing the...
over 1 year ago
Policy on util packages
Ah util, the ubiquitous fixture of practically every project since the invention of code. I spent my early years programming expecting a util package in every project, oftentimes being the one to create it myself. It wasn’t until my third job out of school that I ran into someone who had an allergic reaction to the addition of one and argued strongly against it. I later came to realize that they were right. The problem with util is that it...
over 1 year ago
`CL.THROTTLE` implemented in DragonflyDB
Way back when Redis Modules were first introduced I wrote one called redis-cell. I wrote it because in my experience you couldn’t invent a more perfect fit for Redis than rate limiting, and we’d used it for such at practically every job I’d ever had. The command provides easy limiting in a single Redis call: CL.THROTTLE user123 15 30 60 1 ▲ ▲ ▲ ▲ ▲ | | | | └───── apply 1 token (default if omitted) | | └──┴───────...
over 1 year ago
RFCs and review councils
Squarespace writes about The power of “Yes, if” on the topic of RFCs and review councils (from 2019, but recirculated recently). Major takeaways: RFCs can never have a “rejected” status, only a “not yet”. Councils like an Architectural Review meet to work through RFCs and towards consensus: So we introduced Architecture Review, a twice-weekly meeting where a small group of our most senior engineers review an RFC in depth. Each major RFC gets a one hour session and, since the...
over 1 year ago
Findings from six months of running `govulncheck` in CI
Last September saw the release of govulncheck, a tool that uses that phones into a central database maintained by a Go security team to check for known vulnerabilities that your code might be susceptible to. It’s pretty cool. Instead of doing the easy thing of checking go.mod against a list of known modules, it knows specifically which functions are liabilities, and resolves a full call graph of where your code calls into them. It also prints handy links back the...
over 1 year ago
Honest health checks that hit the database
In web services, a common pattern is a health check, often used for something like a reverse proxy (e.g. ELB) to know whether a node is online. Our API’s health check is used as a target for our status page to show uptime. We’re running an HA-ready premium-0 database on Heroku 1, and a few weeks ago it has some brief downtime as its leader was lost and HA standby rotated into place. The downtime wasn’t well represented on our...
over 1 year ago
Migrating weaker password hashes by nesting them in an outer hash
Yesterday, I wrote about moving to Argon2id for password hashing. We couldn’t perform a full migration because we don’t have access to the original passwords by design, but were opportunistically upgrading hashes on any successful login. But this still meant a long tail of relatively weak-ish hashes 1 for any accounts that hadn’t logged in in a while. Predrag brought to my attention on that despite no access to original password values, there’s still a good way to bring existing...
over 1 year ago
Adventures in password hashing + migrating to Argon2id
Last week, I accidentally revamped our password storage scheme. A small initial cut cascaded into a major incision, and it was beneficial in the end because it got me to look at a branch of tech that I hadn’t reevaluated in ten years. An advantage of being an old programmer is that I actually got to live through all the various iterations around best practices of password management. Way back in the old dinosaur days of the internet, we (and...
over 1 year ago
PartialEqual
File this under “maybe a good idea, maybe a bad idea”, but it’s something I did. In our Go codebase, we often find ourselves with long, unsightly laundry lists of assertions like this (using testify/require): resp, err := apitest.InvokeHandler(svc.Create, ctx, req) require.NoError(t, err) require.Equal(t, dbsqlc.EnvironmentProduction, *resp.Environment) require.Equal(t, req.IsHA, *resp.IsHA) require.Equal(t, req.MajorVersion, resp.MajorVersion) require.Equal(t, req.PlanID, resp.PlanID) require.Equal(t, eid.EID(version.ID), resp.PostgresVersionID) require.Equal(t, req.RegionID, resp.RegionID) require.Equal(t, req.Storage, resp.Storage) require.Equal(t, req.TeamID, resp.TeamID) They’re not only ugly, but they have a bad tendency to quickly fall...
over 1 year ago
Revisiting my two-year SF predictions
Two years ago in the dark depths of early 2021 I wrote Predictions for San Francisco over two years, containing guesses for how the city’s extended Covid policies would impact its civic landscape. I said to revisit these in 2023, so here we go. 7 out of 9 by my count, which isn’t half-bad. But also one of these situations where I really hate to be right, since none of these are good things. FiDi/SOMA businesses gone ~Every customer-facing business...
over 1 year ago
Content authored by ChatGPT front pages
An article written by ChatGPT hit the front page of Hacker News this morning. Maybe not the first time it happened, but definitive, and only the beginning of what’s surely going to be a continuing trend. Timeline: “The creator economy: the top 1% and everyone else”, front pages. It’s a subject many HNers are interested in, and produces a healthy comment thread. Someone points out that the prose seems stilted and repetitive. Oddly AI-like. Another user runs it through an...
almost 2 years ago
The static rip cord
November 28th 2022 was the Herokupocalypse, where the platform engaged in its most aggressive cost-cutting project to date by deleting free apps and free databases. Always leaving things until the last minute, the day before I cycled through the apps in my account to see whether I needed to save anything. Luckily, there wasn’t much. I had ~50 apps, but precious few that were doing anything, strongly suggesting the tendency to use in abundance where there is abundance, regardless of...
almost 2 years ago
Notes on iOS live photos with ffmpeg and <video>
After a visit to the zoo, I played around with converting iPhone “live” photos to short videos that could be uploaded to the web with reasonable ease. For anyone not familiar with the “live” feature, it’s one in which iOS captures a short video around a photo to give it a more dynamic feel (think the moving newspaper photos from Harry Potter movies). Live photos are convenient compared to video because their video component is captured automatically on a normal...
almost 2 years ago
Short, friendly base32 slugs from timestamps
I recently added Atoms to this site, which are short, tweet-length posts published via Git and TOML file. One of my goals was that they should be similarly easy to publish as a tweet, which is partly accomplished using a VSCode snippet and shell alias that copies the current timestamp to clipboard. Beyond that, I wanted each atom to have a short, stable slug that could be included in a permalink for lasting posterity, but didn’t want to have to...
almost 2 years ago
Easy, alternative soft deletion: `deleted_record_insert`
A few months back I wrote Soft deletion probably isn’t worth it (referring to the traditional strategy of putting a deleted_at column in each table), an assertion that I still stand behind. I’ve spent the time migrating our code away from deleted_at, and we’re now at the point where it’s only left on a couple core tables where we want to retain deleted records for an exceptionally long time for debugging purposes. Nearer to the end of the article I...
almost 2 years ago
The unpursuit of clout
I’m old enough that I still remember when “likes” on Twitter were called “favorites”. It initially sounds like a distinction without a difference, but especially in retrospect, the change was highly intentional, and maybe one of the smartest things to increase engagement that Twitter’s ever done. A “favorite” implies it’s some of the best content you’ve seen on the platform. It’s more like a bookmark – something you want to go back and look at again by perusing your personal...
almost 2 years ago