OKRs for Evil and Good
My work does not reduce to measurable outcomes. Much of what I accomplish as an engineer and as a developer advocate amounts to creating conditions that make it more likely for the company to succeed. I resist and resent most metrics, yet I don’t mind OKRs the way Honeycomb does them. How not to OKR: manage to them OKRs (Objectives and Key Results) were popularized by Google, and many companies use them for evil. If meeting the numbers in the...
4 days ago
Productivity v Impact
Here’s a familiar 2×2, the Important & Urgent quadrants: Very early in my career, I was taught to beware the “Not Important & Urgent” quadrant. I’m used to working in the “Important & Urgent” quadrant most of the time, getting the important stuff done just on time. This is the quadrant of productivity. A friend pointed out: The “Important & Urgent” quadrant is hard on our brains. In urgency lies stress. The more time we spend there, the more we...
11 days ago
Why is every startup’s website so darn useless?
I don’t think I’ve worked at a company whose website satisfied me. It never gives me a clear explanation of what our product does. After a few years working closely with Marketing, I finally understand why. As a developer, my goal is to provide useful capabilities to people through the software I work on. To succeed, I need the software to be running… and people have to be using it. Which means they need access, which means people have to...
19 days ago
Command vs Control
I think of “command and control” as one thing. Is there a difference? That common phrase notwithstanding, command and control are two alternative structures, more opposites than synonyms! Command (like in the military) allocates responsibilities to parts of the hierarchy. So a general is tasked with winning a war, and each officer under them is assigned some objectives that contribute. The officers understand what they’re trying to achieve and why. At each level, leadership decides how to get there, breaking...
28 days ago
Communication Structures in a Growing Organization
When Honeycomb was a small company, All Hands meetings had announcements and discussions. Discussions mostly took place in the meeting chat, where everyone can answer anyone’s question. Between that and the announce channel in Slack, information got out to the whole company. Or did it? It’s hard to check for understanding in that big a forum. We’re past 150 people now, past the number of people who can know each other personally. Most new people aren’t comfortable asking questions in...
about 1 month ago
Who turns software into money? GTM
In a startup that sells software, what is the “business” side of the company? When I worked in retail, developing software for internal use, there was a clear division between the engineering teams in IT and the Business, where people understood retail products and how to run a store. But now that I’m in a developer tooling company, software is the business, right? Nope. Selling software is the business. That creates the flow of money that keeps the company afloat....
about 1 month ago
How hard can it be to buy software?
Last year I got my first experience of implementing software internally. My team found a SaaS (software as a service) product they wanted to use, so we bought it and used it. Sort of. It was NOT SO EASY. There’s money to pay, and that has to get through several rounds of budget approvals, and negotiations with the vendor’s enterprise sales rep, and I get to talk to Finance. This is the obvious part. I thought it was the hard...
about 2 months ago
I can write the code. Getting something done is another matter.
a keynote from CppNorth This one is about everything else that a team needs to accomplish, besides writing the code, in order to provide useful capabilities to people.
4 months ago
Going Deep on Gamification
a keynote for YOW! London Playing games, we can focus deeply and work really hard and also have a good time. This is what I want out of work, too. Traditional gamification adds points, competition, and badges to make work appear more game-like. This corrodes collaboration, replacing a deeper meaning with superficial characteristics of games. We can go deeper: let’s talk about how game designers craft an experience for the players, and then apply those principles to the experience of...
4 months ago
Bridges Summit
This is a small talk for Bridges Summit, 28 August 2024. Boundaries_for_Bridges_Summit_2024-08-28_11.27.21Download References and Recommendations: Combining, by Nora Bateson. this is about seeing the second and third-order effects of what we do. It helps me live in a world that is so far from perfect, and work to make it better without despair. (I don’t plan on referencing it in the talk but it’s still my top recommendation!) A Radical Culture of Culture Building, by Beth Anders-Beck Abeba Birhane: A...
4 months ago
Stupid bash tricks: a self-updating script
For a demo app, I want a simple-as-possible deploy script. The simplest thing is to hard-code the version number (which gives my docker image a unique tag, which gets kubernetes to pull the new image1). But I’m tired of updating the version number. So I made the script update it for me. Why not use a real deployment system, Jess? I don’t want to. Some abstractions are out of scope for a freaking demo that nobody is allowed to deploy...
6 months ago
Working with others is different from working.
Today, I’m working on our new deck. It’s a little uncomfortable, because there’s no furniture. I’m sitting on a blanket. The furniture is in the yard, being spray-painted. Before my husband left for the day, he said “If you have time, you could spray some more paint on the chairs.” Seems like I could contribute to that, since I wish I had chairs. Which chair needs painted? I don’t know, I guess the ones on the tarp. Where is the...
8 months ago
Fear and participation
The court system is not created for “justice” as we tend to think about it. It’s a system that can only enforce the laws created by legislators whose interests often don’t align with the most vulnerable who are looking to be protected by the court. Sarah Galloway, Staff Attorney, Arch City Defenders … something something Israel is going to attack a city in southern Gaza regardless of whether it reaches a deal news on the hotel lobby TV Seeing the...
8 months ago
talk: Experimenting in Sociotechnical Systems
Video from DevOpsDays Rockies, September 2024.Keynote at SwiftCraft, Folkestone, UK, 24 May 2024.also given at Explore DDD (video), March 2024; also given at Emerging Tech East, April 2024 At Honeycomb, we have a company value: Everything is an Experiment. Sometimes people take that too literally, and try to make every UI change an A/B test, every process adjustment measurable by some number.The scientific method is fantastic for debugging–and completely inadequate for the unique sociotechnical systems we each work in. Yet...
10 months ago
Double Loop of Value
I want to create value in the world. That’s why this blog is public, my code is open-source, my newsletter is free. This means almost none of the value I create returns to me. It’s a tradeoff — the cheaper it is, the more value the world gets, and the less money I get. That’s OK, thanks to my day job. At work, this tradeoff is made differently. Is the purpose of a company to create value, or to make...
10 months ago
Being the Best
The other day my partner said, “Thank you. You’re the best.” I thought: Yeah, I’m the best for a lot of people these days. My kids, work, you. When do I get to be my best? And then I remembered: people are people through other people. Meaning comes from between us, not within us. Connection makes us whole. Right! what could be more fulfilling than being The Best for the people around me? This is my best, right here, right...
11 months ago
Sending OTLP Logs from JavaScript
TL;DR: right now OTel Logs are a little harder to use than span events, but otherwise they are superior. Today I wanted to send OTLP Logs and see them correlated with my traces. First problem: logs aren’t available in the OTel JavaScript SDK. They’re still in development. (It is the end of 2023.) There is an implementation under experimental/packages. npm install @opentelemetry/api-logs npm install @opentelemetry/sdk-logs npm install @opentelemetry/exporter-logs-otlp-http First, define this environment variable. You can say ‘grpc’ instead if you...
about 1 year ago
OpenTelemetry in a script tag
When I make a website, even a toy one, I want to get telemetry. Has anyone visited it? Did my JavaScript crash? Little things. I want to send this with OpenTelemetry, because it’s the industry standard and all. Currently (November 2023), that requires a whole JavaScript build chain to package up the libraries etc etc. For a toy site, I want to include a script tag and have that do the work. OpenTelemetry isn’t distributed that way (yet), so I...
about 1 year ago
Enabling Gaps
Sometimes my calendar looks like an LA freeway: 100% utilization, ultimate deadlock. This is not one of those times. November through January are quiet at my current job, no big launches or deadlines. I can take a breath. My instinct is to fill it. Sign up for more! Commitments! Deadlines! Quick, pull in the next ticket while we wait for this build to finish. But no, this is a different time. While the build runs, let’s add that test or...
about 1 year ago
Software is Complicated, Given Enough Time
I often say that software these days is complex. But is it? In Cynefin terms, it’s more like an airplane than a family: it’s complicated. Given enough expertise, all the code, lots of data, and enough time, we could analyze the causal structure of any particular system behavior. We can reason about complicated systems. When a complicated system has a problem, it’s time to dig in. We narrow the problem to a component that is behaving unexpectedly, and then we...
about 1 year ago
Working Skillfully in Complexity (for VDDD)
For: [Virtual open Space] Systems Thinking and Skillful Interaction 20 September, 2023 Keynote by Jessica Kerr, jessitron.com These are my notes, publishing for people who were there (or anyone who wants to read them) Working Skillfully in Complexity Plan: about half on the technical side of sociotechnical systems, half on social. I can’t define what Systems Thinking is. It is many things to many people. Personally, my focus area is symmathesy in software teams. I can tell you some things...
over 1 year ago
Testing an OpenTelemetry Collector deployed as a Daemonset in Kubernetes
TL;DR: skip to this readme for the easiest path When you’re running applications in Kubernetes, and you’re using OpenTelemetry for observability, then one recommended practice is to run the OpenTelemetry Collector on each node in your cluster. Then each of your applications can send telemetry data (traces, metrics, logs) to that local collector. To run on each node, the collector runs as a daemonset (instead of a deployment). It is not a Kubernetes service, so it does not get a...
over 1 year ago
A Developer’s Starting Point for Integrating with LLMs
Here’s what I know so far, from a lay-developer’s perspective (no AI or ML experience). So you’re a coder, and you’ve been asked to integrate with ChatGPT. What do you need to know? ChatGPT is an example of a Large Language Model (LLM). LLMs are a kind of machine learning (call it AI if you want to) that are really good at responding to language with more language. You give it some input (a prompt) and it gives you some...
over 1 year ago
Gamification: we can do better.
here’s an interview from YOW! London, where I did a keynote on what can be better than gamification
over 1 year ago
Observability for Speed and Flow
Check this out, I have a chapter in this free mini-book: https://landing.confluxhq.com/optimising-fast-flow This came out of a talk at QCon London… the in-person version was better, but they don’t record them anymore.
over 1 year ago
Participation without complicity
“I participate in this system, so I’m complicit in it.” All the cruelty, the racism, the poverty, the abuse of power in US capitalist patriarchy–this is my fault too? The guilt can crush us. TL;DR: no, it isn’t our fault. It is ours to do something about, while participating. Participate in the system to fill our responsibilities, not to maximize our own gain; try to make it better; and the bad karma of unjust suffering is not ours. This comes...
over 1 year ago
No More Feedback (book summary)
We’re talking here about “feedback” as commentary from people you work with, telling you what you could do better—in particular, 360° Feedback as part of a performance review. I’ve capitalized “Feedback” in this post in reference to this particular use of the word. The benefits of Feedback are purely mythological. Carol Sanford, No More Feedback In “No More Feedback,” Carol Sanford notices that formal Feedback programs are destructive to organizations and people. This post summarized its core points. Summary of...
over 1 year ago
Why no traces? Traceflags 00 edition
Today we wrote some code, told it to send a trace to Honeycomb, and it didn’t. No traces, no error message, why??? This happened to us on AWS Lambda, and it could happen other places too. These debugging techniques can be helpful anywhere you’re struggling with OpenTelemetry. Here I’ll demonstrate them in a tiny Node app. Here’s the least code that should send a span: const opentelemetry = require("@opentelemetry/api"); const tracer = opentelemetry.trace.getTracer("jessitron play"); tracer.startSpan("your favorite span").end(); But, it doesn’t send anything....
over 1 year ago
Accountability for Effective Teams
Accountability is about performance and behavior. Patrick Lencioni, The Advantage Accountability for measurable outcomes is “not the most important kind.” It’s the easy kind. The kind [of accountability] that is more fundamental, important, and difficult is about behavior. Lencioni, The Advantage It’s fundamental because behavior is what we can legitimately ask someone to change. Business outcomes aren’t under our control; we aim to influence them. Accountability for behavior is important is because it’s what leads to later outcomes. If our...
over 1 year ago
Alignment gets expensive. Don’t skimp on it.
Honeycomb has doubled in size since I joined less than 2 years ago. More people means we can do more things at the same time. Both “more people” and “more things at the same time” lead to increased overhead. There’s coordination overhead to get timings right on work that’s independent. Like, to release a feature we need press releases from marketing, documentation ready to go, and functionality lined up behind a feature flag. We were first observability vendor to release...
over 1 year ago
Using ChatGPT in Anger (not real anger)
For years, I’ve been wishing I had Honeycomb events for pageviews on my web site. But doing that in WordPress is hard, it probably involves PHP, and I don’t have time to dive into that world. Forget diving; I can dip my toe into it with the help of ChatGPT. With its help, I got something working in like 10 minutes! It isn’t good code, but it is better than nothing. And I feel powerful.
over 1 year ago
Setting up ALB Access Logs in EKS using Terraform
My objective is to troubleshoot 502 errors in my load balancer. The first step is to set up access logs. That says to create an S3 bucket, create a policy for it, and then configure the load balancer. I refuse to do this in the console; I’m using terraform to create resources and kubernetes to create ingress. TL;DR: here’s a gist. Then skip down to step 5 to configure the k8s ingress. Here’s what finally worked: Create an s3 bucket....
over 1 year ago
when the world feels wrong
sometimes when i wake up, there’s a dark tightness in my chest, like the right side of my ribcage is twisted. Everything is terrible: the world is full of wrong. Our society is in decay, and actively destroying the environment we depend on. My state government is ignoring these problems and persecuting trans people instead. Down the street, a mother was hospitalized for COVID and they took her children away, maybe for good unless she somehow gets a lawyer she...
over 1 year ago
Error: missing ‘x-honeycomb-dataset’ header
TL;DR – go create a new environment in Honeycomb, and get a new API Key. If you get this error from Honeycomb, it means that you’re sending data, and your API Key is working, but… One Honeycomb account can have multiple teams. Each team can have several environments (such as “production” or “dev”). Each environment has a dataset for every service that has ever sent trace data to it. The API key in the x-honeycomb-team header points to a team...
almost 2 years ago
How to search for requests (vs all events) in Honeycomb
Have you ever looked at someone else’s Honeycomb query and thought “what’s up with ‘trace.parent_id does not exist’? Or wondered how Honeycomb’s home screen counts Total Requests? This is totally confusing. Here, I made a video about how to do it and why it works like that.
almost 2 years ago
bash: get the directory this script is in
${BASH_SOURCE%/*} This is useful when you want to read a file that’s in the same directory as the current script, for instance. $BASH_SOURCE holds the current script, including a path. That could be a relative path or a full path. % says “strip off a glob from the end of the value, the smallest glob that matches” /* is a glob, which will match the / at the end of the file’s directory plus the filename. So this expression gives...
almost 2 years ago
Book Review: Software Design By Example
People can “learn to code” on so many websites, in little in-browser environments. Sites instruct on the basics of ‘for’ loops and ‘if’ statements. In advanced courses, learn algorithms and data structures. Writing toy programs is eminently teachable. Writing useful programs is something else. And writing them well is a mysterious art. Greg Wilson‘s new book tackles that. (free online!) The prerequisites are: basic programming constructs in JavaScript; a little HTML and CSS; basics of Git; install and run Node....
almost 2 years ago
Resilience and Waste in Software Teams
Announcement: “All passengers for Chicago at 11:45, please come to gate A14 immediately for an earlier departure.” This came through at 10:35 am, after that 11:45 to Chicago was delayed by three hours. United has an earlier flight with seats available, and they are holding it for 30 minutes. I will get to Chicago early. Flights everywhere are delayed today because of this morning’s FAA outage. Their venerable NOTAM software (roughly: status page, designed for teletype) glitched and grounded everything...
almost 2 years ago