On the Coronavirus

The last few weeks has been tough for all of us. I wanted to share with you my personal experience, and the mindset I’m relying on to move forward.

Looking Back

The calm before the storm

I remember first hearing about this in mid January. My friend was about to head out to China, and she was worried about it. She tends to over-worry, so I reassured her and jokingly told her how she would be fine.

One week in, my coworker visited from China. From his eyes and his stories I could tell this was serious. I called my friend and found out she came back early. Since I was 18, I constantly thought about exponential growth, tail risks, and black swans (1). This fit the bill — I understood it conceptually. But it stopped there. Conceptually.

From February to mid March, I was going through the motions of preparation. Though I thought I understood that the world could shift in a few days, my understanding was only hypothetical. In early February I told my parents to buy up food, and ordered food in San Francisco as well. In some respects I was preparing, but in another, I was simply fitting preparation to the amount of time I had. I didn’t think it was important enough to change priorities.

As things escalated, I increased my attention, but never to the level that this deserved. I canceled my plan to go to New York and tried to distance more. Again this was going through the motions — even with all this happening, the most important thing on my mind was my existing work and personal projects. Even when we were told to work from home. Even as I saw the markets begin to crash, and a significant portion of my personal wealth disappear with them.

The storm

Towards the end of the week, it began to hit me. I realized that we were in much worse shape than China. Complete social isolation was on the way. We could enter a serious recession.

In the same day, two of my friends and I decided to move to a cabin and isolate there. We thought we’d go for a month, starting Wednesday. By the evening we decided to go for two months, the very next day. The timing was on the nose, as the very next day shelter in place was announced in San Francisco.

The next 48 hours was a blur. We got everything together, I ended up liquidating my entire portfolio, and we got out of San Francisco. I remember feeling like I was in a war zone — making multiple drastic, high impact decisions a day. After a night of being stranded, we arrived safe and sound in the cabin. After those 48 hours, my eyes cleared up.

Looking Ahead

As we move towards the present, there’s uncertainty all around us. Many of us are worried about our loved ones. We’re worried about the future. Will hospitals flood with patients and will military cars carry coffins? How long will this last? In a matter of days, many have lost their jobs and many have lost significant wealth. Are we about to experience the great depression?

That’s a lot of uncertainty, but we can come together and manage it. Here’s how I’m thinking about it:

Short Term

  1. Amor Fati (2)

Character is forged through adversity and judged by action rather than thought. Will you let the panic consume you or will you strengthen your resolve? Will you focus inward or will you focus outward? Many of us have felt fear and when we feel fear, the reaction is knee-jerk. As you act, keep this top of mind: how you behave now, no matter what you think inside, is what determines your character.

Let this idea guide you gently: you can feel fear of course, and you can make mistakes, but keeping the idea top of mind will gently evolve and shape your behavior.

  1. Come together

Some have experienced a significant loss of wealth, yet still won’t have to worry about their livelihood. Others have lost their jobs. Some have families that are in trouble. We have experienced pain, we are all in different circumstances, and we all have some ways that we can support our community.

We can’t fix this overnight. No big brother can make sweeping changes. Let’s do what we can as individuals, whether that’s financial support, a phone call, or a kind word--it all counts. Use this adversity to come together.

  1. Roll with the punches

When there’s volatility and change, the panic and fear can make it hard to adjust. Yet, we must adjust. If you’re in quarantine — what can you do because of it? How can you grow and how can you be helpful? Adjusting will calm you and give you mental clarity: whether that’s adjusting what you do at work, reading new books, or finding new ways to connect — make the change.

First time I’ve made a dish in 7 years

Long Term

I think long term, we face two primary fears.

The first, health: will we lose lives? This is fundamentally up to us. What we do to today will ultimately decide tomorrow. Physically isolate, wash your hands, and stay safe. This is directly under our control, so let’s give ourselves completely to it.

The second, wealth: will we enter a great depression? We may feel that the world will change and we won’t keep up — what if we lose our wealth, lose our job, and our skills aren’t relevant anymore? What if we’re not safe, never mind that we may never achieve our dreams?

Let’s break this down.

Kill the fear: Even if you lose all our wealth, your job, and your skills aren’t relevant anymore, you still have your wits. The skills you have today didn’t just pop up when you were born. You learned them. You will learn and adapt with whatever is next.

Evolve the vision: Instead of judging your future by outcome (how much wealth you have), judge it by character: what kind of person will you be? You will be the kind of person who generates value, who is strengthened by adversity. Your character and your behavior is under your control.

Putting it together

Thinking about it, both the short term and the long term fall under one idea: focus on what you can control. You can only control your character and your actions. So focus on that, and judge yourself only by that.


Thanks to Bipin Suresh, whose stoic ideas inspired the realization that all of these actions fit under one umbrella.

Thanks to Jacky Wang and Luba Yudasina for convincing me to include my personal story.

Thanks to Bipin Suresh, Victoria Chang, Luba Yudasina, Mark Shlick, Jacky Wang, Aamir Patel, Nino Parunashvili, Daniel Woelfel, Avand Amiri, Abraham Sorock for reviewing drafts of this essay

(1) Black Swans: Rare events in certain domains, where their magnitude is so large that in the long run they are all that matter. See Nassim Taleb’s Incerto for the concept and some of the most profound essays on risk

(2) Amor Fati: “To love one’s fate” — from Nietzche

❤️ 1

Blanket Solutions and Microservices

Much of my system design philosophy was forged during my time at Facebook. We did a lot of things differently that I think was responsible for our technical success. I want to share one of them with you. It’s the most common pitfall I've see prevalent in our industry, and it relates how we solve problems at the system level. I’ll illustrate it with a story:

Beginning

You kick off your startup. One service, one repo, one database. Version zero ships quickly. You iterate and iterate until you sense product market fit.

All of a sudden, you’ve got it and you’re on a rocket ship. You grow your team, you’re onboarding customers, and you’ve got some great problems on your hands.

Emergency

One of those problems: your systems break.

Your build infrastructure slows down. Deployment becomes an all-day, panic-ridden affair. Commits start to break unrelated components more and more frequently. The development environment becomes slower and slower. You get incidents left and right. Oncall becomes more demanding than a newborn at night.

Slow down productivity and you’ll get grumpy engineers, but take away their sleep and you’ll get rioting engineers. Now you have an emergency on your hands.

Light in the Darkness

You look around, and the technical debt is overwhelming. You think: it’s time to grow up. You research and land on microservices as the answer. It seems to solve all your biggest pain points:

Speed up deployments

Teams can manage their own deployments. Because deploying smaller systems is easier and safer than larger ones, your teams can now deploy faster. Heck, they could even roll back now.

Separate concerns

You’ll force stronger code boundaries. No more commits breaking unrelated components

Empower teams

You’ll be able to use the right tools for the job. No more need to have the same language or stack. Teams can use whatever will make the most productive at solving their particular problem.

Ensure quality

You’ll be able to clarify ownership. No more spaghetti code because service owners will ultimately be responsible

Uh Oh

This is the dream. You kick off that initiative. But…you see some serious costs:

Hardened boundaries are harder to change

Those strict code boundaries come with their own costs. If you split incorrectly, you’re prone to dupe your data and your logic. But, even if you split correctly, what happens when your business needs change? What if you get new regulation that requires system-wide changes? All of a sudden, the way you split those services won’t make sense anymore, and evolving those boundaries become a magnitude more difficult.

Hardened boundaries make it more difficult to ship unanticipated changesets

What if you need to make changes that your org and service structure didn’t anticipate? These changes won’t cleanly fit into one service, or even groups of services. All of a sudden a change that could have been done by one engineer before requires teams to align to deliver.

Hardened boundaries reduce the potential impact of your engineers

What if one of your engineers comes up with a performance improvement, or a new idea for the business? Their impact is now constrained by the services they have control over. If the languages are different they won’t be able to actualize that performance improvement for the whole company. Even if we force all services to use the same language, delivering that to all services becomes a difficult affair.

More Incidental complexity

Sharing code, managing deploys, logging infrastructure, service orchestration, rpc are all made more difficult by a change to multiple services. None of the added difficulty helps you move faster or ship with higher quality.

Root Cause

Why so many unanticipated issues? Because microservices as a strategy is a blanket solution.

Blanket solutions don’t specifically address any one issue, but try to address a multitude of issues. This is often done with philosophy (new principles we will follow to build things) then with technology (how exactly we will solve problems). Whenever a solution addresses a multitude of issues with philosophy, it’s likely to come with a multitude of unanticipated problems.

An Alternative Path

The alternative path is evolution. As a rule of thumb, the changes you introduce should concretely solve the problems you care about. This doesn’t mean that all solutions need to be iterative, but it does mean that all solutions need to be under strong selection pressure. Every solution should have immediate wins in sight.

For example:

Deployment, CI, logging issues?

What if you built a centralized team that owned that complexity, and built infrastructure so product engineers didn’t have to worry about it? For example, most product engineers did not have to worry about deploys, observability, or logging at Facebook. Infra was already in place that they could leverage, managed by an underlying team.

Code boundary issues?

What if you evolved your system to use immutable structures and smaller interfaces between boundaries**? What if you pushed some of these problems down the stack? For example, there may be a lot of complexity introduced with privacy. You can centralize that concern, into building infrastructure with a small interface, that product engineers can use. Facebook did this with viewer context

Scaling issues?

This may be a true concern, but instead of applying a general philosophy, could we focus on solving this problem directly? Do all systems need to scale independently, or does it only matter that just a few things scale independently? exactly are the problems? Could you address with only changing the hottest, most intensive paths?

Complexity issues?

Where is the complexity exactly? Could we evolve modules so you could localize the reasoning behind them? Could we abstract the difficult portions? Sophie Alpert gives a great example of this here

This kind of thinking leads to a philosophy of system design based on simplicity:

You address problems concretely with a view towards evolving your system. At each step of the way, you constantly optimize for engineering velocity. Engineering velocity is a great metric to use, because it implies correctness and quality alongside with speed. It’s impossible to ship if you don’t have confidence in your system. You push and centralize complexity down the stack. You evolve your system so product engineers can think locally within the module they’re working on. You make changes that empower any engineer to drive impact throughout the stack.

Doing this won’t be a magic pill — it will look like your system is constantly broken and in need of improvement — but that is its secret weapon. You’re constantly evolving it.

Thanks to Daniel Woelfel, Jacky Wang, Kam Leung, Joe Averbukh, Phil Nachum, Alexandre Lebrun, for reviewing drafts of this essay

👍 1❤️ 1🚀 1

School, Nietzsche, and Comfort for High School Kids

I have four fierce and independent nephews. Three of them are over 13, which means they’re now “serious” about academics. They also have equally fierce moms with high expectations, and that means report card season can get tough.

To help them get through the ordeal, I repeat a few ideas over and over. I realized these ideas apply to all ages, so I thought I'd share them with you.

Here’s what I tell them:

Your goal is curiosity

I don’t think grades are important.

(This gives them a bit of a shock, and their mom raises an eyebrow, but I forge on)

You can do so much more than what’s expected of you. Just getting good grades won’t have you feeling like you’re becoming the person you want to be.

I want you to make great friends and explore. Create games, music, launch businesses, play, investigate, talk to that girl, do whatever drives you. Those experiences forge your character and teach you in uncountable ways

Adult Aside — Think about how much of your time is spent on status games. The most leveraged, creative work comes when you’re following your taste. Are your goals just about the status game, or something bigger?

But grades are your day job

You still want to get good grades though. This keeps your options open, gives you chances to learn new things, and keeps people from asking too many questions.

At this point, they invariably say: But…some courses I’m just not interested in — when will X be useful, etc

Treat this as your day job and a challenge. What’s the minimum time you need to spend to get the grade you want on those courses?

If you focus on the challenge, a lot of your time will be spent figuring out creative solutions and productivity hacks. This itself will be more fun than the course, help you build discipline, and learn techniques that will be useful to you throughout life. Remember, you just want to do a good job so you can be that inventor, renaissance explorer, hacker, or artist that you want to be.

Adult Aside — Before google came around, search was boring: a “simple” technical problem that no one wanted to solve anymore. Google turned it into a very interesting problem. Think about how you could do that to some of your grudging must-do tasks. At the very least it will strengthen your discipline

Be wary of slave morality

They may interject with a platitude: What’s important is to be a good person, I don’t kiss up to the teacher, who cares about grades.

Have you heard of Nietzche’s slave morality? Be wary of combining bad characteristics with good ones. You can get good grades and be a good person.

Learn to catch yourself doing this, because it will happen all the time in your life. Now it’s grades, but tomorrow it may be “Rich people are evil, I’m not evil”.

You may feel good about yourself, but if you follow this idea, you won’t get what you want. Instead, ask the question: “How can I get good grades and *not kiss up?” Find people like that and learn from them.

Adult Aside — Anytime you want something that you don’t have, this thinking will materialize. “Oh people who are fit spend so much time in the gym, I’m not that kind of person”, etc etc. Be careful about the constraints you set with your ideas.

Be wary of all or nothing thinking

Slave morality can get hard to swallow, so they may say — but grades are unfair. Even if I do well the teacher can knock me for “participation”, that’s why the kiss-up people get good grades.

You’re right. This is frustrating and unfair. And there’s a reason some people kiss up: it’s easier to do that. But, just because some parts are unfair and some people cheat, doesn’t mean you should give up on what you want. The world isn’t so black and white.

Tell yourself: you will do what’s under your control. Learn from the people you admire who behave the way you think is right. Yes some people have an advantage, yes some things are plain unfair, but you can still get what you want. If you do all of that and it doesn't work, you'll know you tried your best, and something better will certainly happen regardless.

Adult Aside — I remember Ramit Sethi telling the story of someone who wanted to start working out, but only had one day free a week. Instead of working out once a week, they didn’t work out at all. They thought it had to be 5 times a week. The more you can let go of the black and white nature of childhood, the more you’ll flourish

Be wary of comfort

A final interjection may come from comfort — I want to learn this thing, but I don’t want to look dumb, etc.

If you lived in our village, you’d be prettyy cool. Wow you study in the big city, wow, you know so much math, wow you are good at fighting.

But imagine if you met a navy seal from Afghanistan. How do you think your fighting skill would stack up?

If you actually want to become great, you have to get outside of your comfort zone. Actively look for those opportunities. You’ll have an urge to drop them because you want to feel cool…but hey it’s better to be cool then to feel cool

Adult Aside — Ask yourself constantly, where are the people I can learn from? It can be uncomfortable to go from the best person in the room to a noob, but this is the path forward

Appreciations

Many of these ideas crystallized by reading two people have inspired me: Nassim Taleb and Paul Graham. Nassim for elucidating the life of the flaneur, and Paul for his essay “What you’d wish you’d known”

Thanks to Nino Parunashvili, Elene Asanidze, Alex Reichert, Julien Odent for reviewing drafts of this essay

🎉 1❤️ 1🚀 1

The price we pay

I recently read Paul Graham’s Raising Kids. I loved the piece in its entirety, but a few sentences sparked something within me.

He wrote about what he missed about life when he was younger:

I remember perfectly well what life was like before.

Well enough to miss some things a lot,

…like the ability to take off for some other country at a moment’s notice.

I smiled with excitement when I read that. Yes! We have so much freedom.

But…there was a twist. In a few poetic sentences:

That was so great. Why did I never do that?

…most of the freedom I had before kids, I never used.

…I paid for it in loneliness, but I never used it.

Yes, yes. So much yes.

We have freedom. Yet we burden ourselves with faux responsibility or self-inflicted urgency.

Think about it: what feelings come to you, when you imagine doing something for no reason at all?

It can feel wrong on so many levels. Yet, where where are we hurrying too?

The time when we have real responsibility will come.

The time where we absolutely must focus on one thing, in one place, will come.

We’re already paying for the freedom we have. How can we embrace it?

Crazy Ideas

Note: This piece is specifically directed to people who work at mid-large companies. The ideas will still apply to you if you work at smaller companies, but they’ll come as less of a surprise.

At any point in time at a company, you can find a bunch of ideas to make things better.

You can create features that make your customers happier. You can create tools to make your engineers more productive. Or…you can launch completely new products.

The questions to ask are “Who gets these ideas” and “How do they get these ideas?”

In larger companies, there’s an implicit understanding that ideas come top-down. It’s as though there’s a line for innovation: the best ideas come to the CEO, and maybe one day some of them trickle down to you.

Yet, is that how things work?

Thankfully, reality doesn’t work that way. If you look around you, you’ll see that some of the best ideas in the world come from some of the most unexpected places.

For example:

Memcached — did it come from a senior architect at google? Nope…young kid working on scaling their project (LiveJournal!).

Gmail — did it come from an ex-hotmail VP? Nope, 24 year old who tried creating an email client before, and decided to do that again at Google.

Airbnb — did it come from veterans in the hospitality industry? Nope…just a few friends trying to make rent.

Stripe — did it come from veteran payments execs? Nope…just two brothers frustrated with taking payments online.

This is happening all over the place externally. And, even though it may not seem like it, it’s happening all over the place at larger companies too.

Don’t believe me? Try this: look over a few of your favorite tools and products at work. Try to find the origin story. You may be delighted with what you find! (If you do this, I’d love to hear the origin stories that speak to you in the comments!)

We begin to see that ideas are not top-down. Well, how do ideas come then?

The more stories you see, the more you notice a pattern: Ideas come to people who face problems, talk to customers, and tinker. Paul Graham goes over this quite well in his essay on startup ideas, and they apply at larger companies just as well.

That’s really all you need to do. The beautiful thing about this? Notice what’s missing: title, pedigree, experience. None of that is required. Just face problems, talk to customers, and tinker.

At this point, you may be facing some resistance. You may think — it’s just not how it works at my company.

Yet…I encourage you to try and test your assumptions.

Most leaders want people like you. They know ideas can come in the most unexpected places. Now, there will be a spectrum of resistance. At most smaller tech companies you’ll find sympathy and support, while at more bureaucratic places you’ll face much more resistance.

However, If you try, you’ll contribute to the culture within your company that nurtures innovation and employee engagement. If you do this, you’ll give your company the ultimate competitive advantage and discover a bunch of like-minded people too.

Two action items, and a note

1. Action Item: #crazy-ideas

At Airbnb, we have a slack channel called #crazy-ideas. This is a space for people to get together and spitball any idea. So far we’ve found it great for encouraging innovation and engagement. I think Stripe has something similar (I was inspired to create this channel from a conversation with some of the engineers there). If you don’t have this, try starting one off at your company.

2. Action Item: origin stories.

In this essay I asked you to look into origin stories of some of your favorite products. What came up? If you’re up to share it, I’d love to hear!

3. Note: On facing resistance

As you test your assumptions, you’ll be surprised with how much you can do, but at some point you will invariably face resistance. There’s a whole trove of strategy here: aligning with teams, finding buy-in, managing up, building momentum, the list goes on. This will come for another essay. Meanwhile, if you start facing these problems, ping me and I’d happy to riff.

Thanks to Avand Amiri, Daniel Woelfel, Eric Ning, Jacky Wang, Julien Odent for reviewing this essay

Advice on starting out

From time to time newly graduated hackers ask me: “Do you have any advice for beginning my career?”

I’ve found myself repeating 3 principles over and over again. On reflection, I realized these principles apply more broadly. I think they might answer the question “Do you have any advice on for beginning my life as an adult?”

I thought I’d share these ideas with you here. The style and the examples I wrote are career-oriented, but I think the core ideas will still help in your decision making. I hope you enjoy them : ).

Principle A: Follow your taste

What should you do? is one of the hardest questions to answer when you start out.

The default path is to follow what’s popular or prestigious. That can lead to a bunch of problems: What’s prestigious is already highly competitive. When you compete with smart people in a game that has established rules, just keeping up will take most of your time. That leaves little time to explore what interests you. When you don’t explore what interests you, you won’t understand things as deeply, and that leaves you with an undifferentiated skillset.

The solution here is to follow your taste. Popularity and prestige are lagging indicators. Your taste, especially as a young, curious person, is a signal for what will be popular in the future. Regardless though, when you follow you taste, you’ll find yourself much more interested in the problems you’re solving. When you’re interested in the problems you’re solving, you’ll discover deeper truths, build deeper expertise, a differentiated skillset, and have more fun along the way.

Now, you may be wondering: What if you don’t even know what you’re interested in?

Don’t worry. You just need to tune into and strengthen your taste. To do that, just do anything. Once you do anything, ask yourself, what part of that thing do you find interesting?

Start going deeper there, and all of a sudden, you’ll discover a wealth of interests.

Next up, you may be wondering: What if your interests keep changing?

Don’t worry. It’s totally fine. Just follow it. You’ll be surprised how things will serve you down the road. As long as you’re working hard and you’re learning, it’s a good use of time.

You may also wonder: What if your interests are what’s popular and prestigious today?

That’s great. Follow that, but instead of competing, start sensing what actually interests you in that field and go deeper.

The ideas may seem career oriented, so you may be wondering: What if I just want to travel and explore the world?

That’s great too! It’s your taste, telling you that you want to travel. Follow it and see where it leads you.

As you discover your interests, you may find that they’re not necessarily popular. Yet, when you go deeper and do your thing, something magical will happen…which leads us into the second principle:

Principle B: Find your community

As you explore and follow your taste, you’ll discover a subset of people who share your core interests and values.

Nurture those relationships. Some of these people will be your friends for life. Many of those people will become very successful in the future.

The people you know will show you what’s possible and work alongside you in deepening, exploring, and innovating on your shared interests.

You may be wondering: what if you live in the middle of nowhere, and there aren’t people who get you? Do your best to go to the center of wherever your community is. That may be SF for hackers or Los Angeles for entertainers or Paris for artists, the list goes on. You know where it is for you.

Now, this may sound impractical to you, but you’ll be surprised with what you can do. You can start off online in the meantime.

One common pitfall here is focusing only on successful people in your field. This misses the point. Sure, reach out to those successful people, but remember that the young, curious, interested people right next you, will become those successful people in the future. You have tons of time to spend together, so it’s great way to deepen your relationships. I’m not saying that you shouldn’t reach out to successful people (you definitely should), but to remove “success” as a criteria altogether for choosing who you spend your time with.

Now, you may be wondering, how do you deepen those relationships? That leads to us to the third principle:

Principle C: Take risks

As you form your community, new ideas and ways to collaborate will spring up. Take those opportunities: work together, support each-other and play. Over time, which you have a lot of right now, you will form a tight tribe of friends.

As a happy side effect, you’ll become exposed to new fields, new technologies, interesting problems, and interesting opportunities. Start taking them, and you’re well on your way to a career that’s made for you.

One common issue here is a lack of self belief. This may lead you into choosing what you think “you are capable of”, rather than what you’re interested in. One way to get around this is to just experiment: let yourself dream and work with your friends, even if the problems seem fantastically hard.

Final comments

There are a lot of examples I can give you if you’re interested in what I’m interesting in: From my favorite books to Paul Graham’s essays to Norvig’s Paradgims of AI programming, to Rich Hickey talks. But, instead I encourage you to think: what interests you right now?

If you’re up for it, I’d love to know your answer: what interests you right now? Feel free to leave your answer in the comments or to email me directly, I’d be curious to hear : )

Gratitude

There’s a slew of people who have influenced and mentored me. These ideas are just as much them as me.

On the idea of taste specifically, the big shoutout has to go to Paul Graham. A lot of my thinking about this was inspired by reading his essays over and over when I started out. I highly suggest reading them.

Thanks to Daniel Woelfel, Talha Baig, Martin Raison, Irakli Safareli, David Magaltadze, Joe Averbukh, Julien Odent, for reviewing this essay

Macros by Example

I was in a conversation recently about the power of macros, and the use of syntactic abstraction in building simpler systems.

We quickly realized though: it’s tough to convey in a conversation what’s so special about macros. What can you do with macros that you couldn’t do with functions?

In this essay, we’ll use 2 examples, alongside some imaginary javascript syntax and lisp [1] to explore that question!

Note: This tutorial assumes you have a light understanding of lisp syntax. Go through this tutorial to brush up if you haven’t gotten to explore lisp yet.

Note: I’ve been meaning to write this for weeks, but was worried that it would be confusing. I am going to apologize now if that’s how you end up feeling when you read this. There may be a better way to explain it, but I needed to get this out of the head. If you have any feedback on how I could make this simpler, please let me know 🙂

[1] The specific language is Clojure, but anything done here can be done with any lisp

Example 1: nullthrows

With this example, let’s gain an intuition for when macros run and why that can be powerful.

Context

In any language with nulls, there’s a nullthrows abstraction: If some value evaluates to null, throw it.

Here’s how we could implement that as a function in javascript:

function nullthrows(result) {
  if (result === null || result === undefined) {
    throw new Error("uh oh");
  } 
  return result;
}

So if we run it, and it evaluates to null, we’ll throw an exception

nullthrows(getUser(db, 'billy'))
// if it's null, throw Exception

This works great…but there’s a problem. What would our stacktrace look like?

index.html:700 Uncaught Error: uh oh
    at nullthrows (index.html:700)
    at someStuff (index.html:1325)
    ...

When some value is null, the stacktrace won’t have much helpful information. It will say which line threw, but we’d have to do some digging each time to find out where the code was.

One way we can fix that, is to pass in a message argument

nullthrows(getUser(db, 'billy'), 'expected billy');

function nullthrows(result, message) {
  if (result === null || result === undefined) {
    throw new Error(`uh oh: ${message}`);
    ...

This could work…buut

Challenge

What if I told you: I don’t want to have to pass in a message.

Instead, when the source code that nullthrows wraps is specific enough, I’d be just as happy if the error printed the offending piece of code.

For example, with nullthrows(getUser(db, ‘billy')), nullthrows is wrapping the source code getUser(db, ‘billy’))

If the error printed out “Uh oh, this returned null: getUser(db, ‘billy’)”, it would be specific enough, and I wouldn’t need a custom error message.

Problem

Well, by the time nullthrows is run, getUser(db, ‘billy’) will be long gone: all the function will see is the evaluation of getUser(db, ‘billy’). Since the evaluation will be null, there’s not much information we can gain.

Javascript Solution

To actually capture and work on source code, we need a new kind of abstraction.

This would be some kind of function, that does two things:

  1. It would take snippets of source code as input, and return new snippets of source code as output
  2. This abstraction would be called at the build step, and replace the source code snippets that it takes in, with those new source code snippets.

Let’s say Javascript had that. instead of function nullthrows, we would have macro nullthrows. It could look something like this:

macro nullthrows(sourceCodeSnippet) {
  return `
    const result = ${sourceCodeSnippet}; 
    if (result === null || result === undefined) {
      throw new Error("Uh oh, this returned null:" + ${stringify(sourceCodeSnippet)});
    } else {
      return result;
    }
  `;
}

Here, the input would be the actual source code.

Whenever it’s called, we would replace that piece of code, the source code snippet that this abstraction generates.

For example, during the build step nullthrows(getUser(db, ‘billy’)) would be replaced with:

const res = getUser(db, 'billy'); 
if (res === null || res === undefined) {
  throw new Error("Uh oh, this failed:" + "getUser(db, 'billy')");
} else {
  return res;
}

Now, you might see some potential problems here:

Snippets are just text! It’s really had to programmatically add/remove/edit text without causing a bunch of syntax errors. Imagine if you wanted to change the source code, based on what it was — is it a functional call or a value? — there would be no way to tell with just text.

With javascript, you can work on the abstract syntax tree itself with babel transforms, but that will make the implementation quite different from what we want our code to be doing.

We really want to use some better data-structures to represent our code. Turns out, this idea isn’t new: there’s a whole family of languages — the lisp family — that wanted code to be represented with data-structures so we could read/generate code snippets more easily.

It seems like a lot of work just to make a built-in code-snippet-generator for a language, but let’s see what our challenge looks like if we use lisp’s approach:

Lisp solution

Since all code in lisp, are just lists, our lisp macro takes in a list of code, and returns a new list of code

We would write nil-throws like this:

(nil-throws (get-user "billy"))

The function variant would look like this:

(defn nil-throws [res]
  (if (nil? res)
    (throw "uh oh")
    res))

Now, I’m going to show you how the macro variant would look like: (don’t worry about a few of the symbols you’ll see, they’re all simple and I’ll explain them in just a few words below)

(defmacro nil-throws [form]
  `(let [result# ~form] ;; assign the evaluation of form to result#
    (if (nil? result#)
      (throw
        (ex-info "uh oh, we got nil!" {:form '~form})) ;; save form for inspection
      result#)))

Here’s how we can think about it:

  1. Similar to how we wrote ` in javascript, the backtick here does the same thing: it says, hey, here’s the code I want to return, don’t evaluate it right away.

  2. # is a handy way to generate some symbol, that won’t interfere with any other symbol when this code gets replaced in a different scope.

  3. ~ is like our interpolation ${} in javascript, but for lists

  4. is a way to say: hey, I want to treat something as a list, and don’t want to evaluate it

This would make it so when we write: (nil-throws (get-user db “billy”))

It would be replaced (~approximately) with:

(let [result# (get-user db "billy")]
  (if (nil? result#)
    (throw (ex-info "uh oh, we got nil!" {:form '(get-user db "billy")})) 
    result#))

Wow…we just wrote code that wrote more code…that’s pretty cool

Lessons learned so far

Macros take code as input, and return code as output. They run during the build step

Example 2: pipe syntax

Now, let’s explore the kind of power this can give us.

Context

The pipe operator is quite common in a bunch of languages.

It takes code that you would normally write like this:

createBill(addToCart(cart, updatePrice(item, 100)))

And let’s you invert the flow visually:

item |> updatePrice($$, 100) |> addToCart(cart, $$) |> createBill

Challenge

What if our language didn’t have this, and we wanted to implement it? Maybe we’d want our syntax to look like this:

|> [
  item, 
  updatePrice($$, 100), // updatePrice(item, 100)
  addToCart(cart, $$), // addToCart(cart, updatePrice(item, 100))
  createBill, // createbill(addToCart(cart, updatePrice(item, 100)))
]

Problem

Now, we could do this by implementing a pipe function:

pipe(item, (item) => updatePrice(item, 100))

But we would need to introduce anonymous functions, and the code would be less concise.

The only way to do this to spec, would be to change the syntax itself.

Javascript Solution

Now, with our imaginary javascript syntax, we could write something like this:

macro |> (listOfForms) {
  return listOfForms.reduce(
    (lastForm, thisForm) => {
      if (isFunctionalCall(thisForm)) {
        return `
          let $$ = ${lastForm};
          ${thisForm};
        `;
      } else {
        return `${callFunction(thisForm, lastForm)}`;
      };
  });
}

Here, would start with a list of the forms, un-evaluated:

[item, updatePrice($$, 100), addToCart(cart, $$), createBill]

Reduce would start with the arguments lastForm = item, thisForm = updatePrice($$, 100)

Now, we would need a way to know: is this a form of a function call updatePrice($$, 100), or just a function: createBill

If it’s a function call, we can create new code, which defines $$ as the last form, and evaluate the function call within that scope.

Otherwise, we can create new code, which calls that function with the last form.

Lisp Solution

What we want would be something like this:

(|> item
    (update-price $$ 100)
    (add-to-cart cart $$)
    create-bill)

And our macro could look like this:

(defmacro |> [form & forms]
  (reduce
    (fn [last-v form]
      (if (seq? form) ;; am I being called: (update-price $$ 100)
        `(let [~(symbol "$$") ~last-v]
           ~form)
        `(~form ~last-v))) ;; or am I just a function: create-bill
    form
    forms))

Our lisp code would follow the same idea as our Javascript solution. Let’s see the code we didn’t have to write:

(create-bill 
  (let [$$ (let [$$ item]
             (update-price $$ 100))]
    (add-to-cart cart $$)))

…that’s pretty cool. We get the best of both worlds: efficient, well-erroring code that’s short to write and — most importantly — clear to read.

Lessons learned so far

Macros let you change the language itself. You can transform code and change the syntax.

Conclusion

Macros let you change your language to suit your problem. This is extremely powerful: You can build up your language so you can express your problem as clearly as possible. This makes your code more concise and simple, which in turn makes your system more malleable.

At the same time, macros…change the language itself. There are few moments where this level of abstraction is warranted, so if you use them when simpler abstractions would do, you risk adding unnecessary complexity.

Yet… when they are warranted, having them as an option can change the game.

Further Reading and Practice

If this got you interested, here’s some reading and practice you may enjoy:

  • Read Norvig’s Paradgims of AI Programming, and do the homework
  • Read Clojure for The Brave and True’s Macro Guide, and do the homework
  • Look into how Clojure uses macros to define the language itself (when, and, or, etc)
  • Write async await syntax for promises

Credits

Shoutout to Daniel Woelfel: I saw his nil-throws macro years back when we worked together, and it opened my eyes to the power of syntactic abstraction.

Thanks to Sean Grove, Daniel Woelfel, Martin Raison, Alex Reichert, Mark Shlick for a beautifully deep review of this essay.

Thanks to Paul McJones, perfunctory and tzs, for their feedback on the code examples.

Two stories I share with my nephews, to help them take risks and follow their curiosity

Story 1: The Dice

I ask them:

Imagine there’s a game of dice. If you roll any number but 5, you give me one dollar. But, if you roll a 5, I’ll give you 600 dollars.

Would you play the game?

They invariably say yes.

(Some say, “depends on how much money I have”, which case I pat them on back, as they discovered expected utility. I then tell them though they have a hundred bucks at hand, and they say yes again : ))

Now I tell them:

Imagine someone walked by you and saw you play the game.

Most of the time, other people will see you lose. They’ll think you’re dumb.

Here they get heated, and say — heck, that person doesn’t know how big I’ll win! I agree, and then ask them:

Imagine now, it’a a 60-sided die. You still lose a dollar each time, but if you get a 5, you win 6 million.

They get excited by the millions, and say they’ll take it. Yet, I remind them, now if someone sees you, they’ll almost always think you’re dumb.

And from there, we get to the lesson:

It’s important to try for things, even if you have a small chance of winning. So…apply for that scholarship, try out for that team. Yes, maybe you’ll lose, and heck, you’re likely to lose, but that’s the whole point.


Story 2: The track captain and the climate change speaker

Note: I discovered this story as a kid myself from one of Cal Newport’s blog posts. I can’t find it, so will reproduce here. Highly suggest checking out his blog!

I start with:

Imagine two stories:

A. One person is the captain of their high school track and field team.

B. One person gave a speech on climate change at the United Nations.

Who is more impressive?

They invariably say B: climate change.

(If I tell this to Eastern European parents, they say A. track captain, because they think the UN person got in through insider connections. I tell them to ignore this though, and imagine it was earned :P)

I then ask: why?

After some minutes, I explain:

With track captain: you can _imagine_ how they did it.

If someone wakes up every day at 6am and works very very hard, they can become the track captain.

But with climate change it’s hard to imagine — how the heck did they get there?

From there comes the remarkable lesson about remarkable things: Remarkable things can’t be planned. If you can plan something into the future, surely it’s easy to imagine how it can be done looking backwards.

So…then, how do you do remarkable things?

The only path is to follow your curiosity. If you follow what you’re interested in, no matter how wacky it looks, you’ll look back after some time and say “huh”, how did that happen.

I then give them an example of how it could have happened for the climate change person: maybe they were into it, they started a blog, they got a group together, they wrote something that got shared, and boom, they made it to the U.N. That would have been impossible to plan.

This means follow your curiosity. Don’t get caught up by the urge to be #1 in some list at your school. Instead, talk with your like-minded friends, play, and see the world unfold around you

How to get those 9s: on improving service uptime

What do you do when you inherit, or just as commonly, create, a service with low reliability?

Improving reliability can be daunting. You have low visibility into the system, and sometimes, if you inherited it, low expertise. The system is ever-changing, so if you just metric on success rate, you’ll consistently come across new errors, as soon as you fix old ones.

But…with the right strategy, improving reliability can be fun… maybe even thrilling! You’ll get to dive deep and solve some hairy problems. You’ll create solutions that not only improve your service…but improve all services in your company.

Now, to do this, I’ve come down to a strategy of three steps:

  1. Stop the bleeding: Represent the service as it is and hold the fort 2. Bring Observability: Identify the root causes weekly 3. Incrementally Improve: Retro and improve based on the most important root causes

1. Stop the bleeding: Represent the service as it is and hold the fort

This service is often already goal on uptime: maybe with that 99.99.

Most likely though, the service is failing that so consistently, that the goal no longer has any signal.

Our goal here: bring back the signal. To do this,

Action Item 1: Update the uptime goal to represent the current state of the system

Update the uptime, so that this metric to represent the current P99 uptime and latency values. If you do this, the goal should start to go green. This will help you become intentional about uptime slips moving forward.

This brings you back that signal. Now, just because it’s green, doesn’t mean . that you stop here and go on with our day. The next step is to improve it. To do that we need to

2. Bring Observability: Identify the root causes weekly

To have a meaningful approach to improving uptime, we’ll need visibility into your system.

Visibility in this case means the answer this question:

Question 1: What were the root causes of incidents last week?

The key here is root cause. You want to know: each week, what where the actual errors that brought down the system.

Action Item 2: Answer “what were the root cause incidents last week?”

You’ll need to invest in your tooling, to create a list of the big bugs that brought the system down.

Sometimes, especially in distributed systems, it can be very hard to get a sense of the root cause — a root cause could be a leaf node service error, and your infrastructure may not be aggregating by leaf nodes.

Nevertheless, it’s possible. You can look into observability tooling — the most promising one I know right now is www.honeycomb.io (no affiliation) — but more likely then not, you can use your existing tooling and an hour of manual work / scripting, to get a good sense.

Once you know that answer, you can

3. Improve Incrementally: Retro and improve based on the most important root causes

As you get this information, you can begin to run a “mini incident management” series:

Action Item 3: Create a “retro” series for fixing root causes

For the biggest error, take the steps needed to fix and remediate the root cause. This includes fixing the bug, as well as adding a test or an alert, or turning a hard dependency into a soft dependency.

If you do this consistently enough, you should begin to see our uptime improve. You can goal yourself on the top-line metric (uptime reduction), but also the number of root cause errors removed, remediations added (tests, process changes, etc), etc.

I read once in Nasism Taleb’s books: one plane crash, makes all planes safer. This is because the aviation agency investigates the plane crash, and makes sure that kind of error doesn’t happen again. This applies to your service too. When you do these retoes, you all of a sudden make your service better and better and better. It becomes…antifragile!

Summary of Action Items

  1. Update your uptime to represent the current state of the system
  2. Answer “what were the root cause incidents last week?”

  3. Create a “retro” series for fixing root causes

On leadership, self-esteem, and confidence

Joe and I have a bit of a reputation. We’re known to have a high level of self-esteem — so much so that we’ve been asked “how do we do it?” quite a few times.

In this essay, I’ll try to answer how I think about it:

A. Your goal is to convince yourself that you are a confident leader with high self-esteem

Your inner critic knows you best, and they’re the hardest to convince. Yet they are the most important person that we need to convince: all our behaviors and beliefs stem from there.

Now, you could try to convince your critic by describing yourself as a confident leader with high self esteem: “Oh I’m confident, oh I’m a leader”

But your critic is too smart for that. If you try this the words will seem hollow. After all your inner critic knows what you really think.

To truly convince our inner critic we’ll have to get more concrete: what do these words actually mean?

To convince your critic, first

B. Identify the behaviors behind self-esteem, confidence, and leadership

Take a step back and ask yourself: what do these words mean to you?

Self-esteem: To me, self-esteem means you have strong core values, which you won’t change to appease other people. You are okay with who you are. You are capable of being vulnerable and risking rejection, because you value your self-opinion higher than those of the crowd.

Confidence: To me, confidence means, you go after ambitious goals because you believe you are capable of achieving them.

Leadership: To me, leadership comes from taking care of other people, empowering other people, thinking of others before yourself, doing the hard work, inspiring others.

Now you can start to describe yourself with these behaviors

Instead of I have high self esteem, you can say

  • I have strong core values

  • I stand up for what’s right.

Instead of I’m a leader, you can say:

  • I am a person who looks out and takes care of people.

  • I am a person who coaches and empowers people.

At this point your inner critic will start to listen…but we’ll need to do more before it catches on. To truly cement those beliefs:

C. Prove out behaviors with your actions

As life goes on, you’ll come across many opportunities to help others, to stand up for what’s right, to dig the ditches and do the hard work.

To illustrate how frequent these opportunities are, let’s think about something very simple. Say you want to meet your friends

  • Instead of waiting for people to invite you, take initiative and invite others
  • When you’re choosing where to eat, instead of asking yourself where you specifically want to eat, ask yourself, where your friends would enjoy eating the most
  • When you’re about to order, instead of thinking about what you want to eat first, think, would the group like some appetizers or drinks?
  • Now say you arrive, and you notice one of your friends drops a fork: take the initiative and pick it up, ask the waiter to help
  • Say you’ve been at this restaurant a few times, introduce yourself to the staff, and thank them for taking care
  • Say at the dinner there’s a pretty person you’re interested in, but they say something you disagree with. Stay strong and say what you think (I still struggle with this one :P)

These opportunities are everywhere. Notice them, pick them up, and all of a sudden, your description of yourself gains strength. With that you can begin to make the jump and tell yourself “I’m confident”. This time the words have depth.

The best part of all of this is that your internal critic will change roles: from critic to advocate. Initially, all they did was hold you back. But, as you train them to describe you with positive and worthwhile behaviors, the tone will shift: when something bad happens, they’ll start giving you reasons why everything will be okay, why you have the strength to deal with this, why you will behave with class.

This will reverberate throughout your life.

End Notes

1) One thing that helps: role models for these behaviors.

Personally, adventure novels (Scaramouche, Captain Blood) and anime (Naruto, One Piece) left an imprint on me. I often think about what some of my favorite characters would do in certain situations 😄.

2) One thing to watch out for: ego

It’s easy to take this thinking too far and begin to think you are special, an exception. You don’t have actually pay the price, you’re born this way. This is a recipe for disaster and it is very easy to fall into, especially when you receive praise from others.

What helps me:

  • Remember that we are all one, a part of a community. It’s weird to say this, but in case you don’t already believe it: All human life is worth the same, no matter what we do.
  • Tie your self belief to your actions: it’s easy to stop being a leader: all you have to do is let it get to your head and stop caring for other people.

3) This may generalize

Thinking about it, this generalizes to a whole bunch of other characteristics. How do you become a great communicator, or how do you become a great writer, an exceptional engineer? The steps seem the same. Something to think about!

Prior Art

  • James Clear talks a lot about the effect describing yourself can have in his book Atomic Habits. Highly recommend reading

  • My previous co-founder Sebastian Marshall would always tell me: judge yourself by your actions. This definitely played a part in my philosophy

And with that, we have a rough roadmap for leadership, self-confidence, and self-esteem 🙂

Thanks to Alex Reichert, Giff Huang, Joe Averbukh, Luba Yudasina for the review