The rails that power our financial system—the infrastructure beneath our everyday transactions—are transforming very quickly.

By the end of this post, hopefully you will walk away excited about how Payment systems are fundamentally evolving, what the future holds for how we transact and the technical pieces to get us there. Crypto is a big part of this future, but to see that we have to look beyond Bitcoin. Bitcoin is only the tip of the iceberg, like gold is in our traditional finance. What’s below is much more massive!

To ground this post in a customer problem, let’s follow Anika, as she navigates her day through the shifting landscape of money—from the early systems to current fintech apps and what’s coming next.

Let’s dive in. 🚀.

Payments 1.0 : The Era of Traditional Banks & Clearing Houses

Anika’s parents run a family-owned bakery and relied on paper checks from the custom cakes to wedding planners. They wait for days for checks to arrive and several more days for checks to clear. If a check bounces, it creates additional week of delays. Anika, is studying to be designer and goes abroad to study. Her parents use a traditional bank wire transfer which takes several days, incurs a high transfer fee, and results in an unfavorable currency exchange rate.

Early Financial Services were largely the domain of Traditional Banks.

Technology: To understand the technology behind this phase, let’s look at an e.g. ACH (Automated Clearing Houses). To facilitate the movement of money between people who had accounts in different banks - ACHs were created. Early on it was physical and with paper.

With advent of software over the past 30 years, these are largely implemented through batch processing. The originating bank groups all ACH requests in a specific format.

e.g. "Bank A is initiating a debit (pulling money) of $50 from an account and sending it to John Doe’s account at Bank B." into a specific batch file with rows for e.g 62712345678098765432100000050001234567890 John Doe 0123456780000001.

These are then sent at the next processing time to an ACH operator (through sftp, api ), which verifies transactions and routes to receiving banks, which batch process and settle on next business day to the Customer John Doe.

This phase of Payments 1.0 was dominated by large brick-and-mortar banks and card networks (Visa, MasterCard, American Express). Paper checks, Wire transfers, ACH were primary ways to transfer money. Settlement often took days. There were high barriers to entry and entrenched institutions held monopolies, with regulatory frameworks restricting competition.

While there was trust and oversight through federal regulations, the customer experience was slow (branch-based, paperwork-heavy), little to no innovation in user experience for decades and cross-border transfers were costly and slow.

US Payment Systems is a fantastic book to go deeper into these Payment Systems.

Payments 2.0: Fintech makes it seem “Instant”

Anika is now back in San Francisco. Signing up for new fintech apps is very easy and quick - she can do it on her phone and add to her balance in minutes instead of days it used take in 1.0 to get a bank account. When she goes out with her friends for dinner, they can“Venmo” her their share of the bill immediately. This is instant, however if Anika’s wants to move her Venmo balance to her bank account to pay the credit card bill, its 1-3 days or a fee.

During summer, Anika signs up for a neobank ahead of a trip to Europe. There are no foreign transaction fees on purchases and no need to carry large sums of local currency. This is a huge leap forward in customer experience (easy app, multi-currency wallets) but the underlying international transfers still rely on older rails with fees and delays.

Apps & Neobanks like Venmo, PayPal, Cash App, Chime pioneered this phase largely through slick UX. The biggest impact of this phase is moving us towards a cashless society and using our mobile phones for transactions.

Technology in this phase is largely focused on 3 areas:

1. Slick Interfaces and Social/Network effects- e.g. discover people, send money, split tab. However, when you bring money into the app e.g. by connecting your debit card or bank account, its using same card network or ACH pull to get money into your Cash App balance.

2. Integrating with traditional payment Rails to bring money in and out of system, & management of internal balances: When you transfer money from one user to another on CashApp, thats largely managed through internal account balances, private ledgers, and reconciliation - a process for making sure the balances internally match the net in/out through all the different channels e.g. Banks, Card networks.

3. Investments in Risk/Fraud Detection- with ease of technology and quick signup (access) came increased risk of fraud. Previously you had to physically show up at your bank with paperwork to create an account. It was slow but more reliable. Now fraudsters can create 100s of accounts through a mobile farm using fake digital ids.

In most apps in this phase, to gain benefits of Payments 2.0, both sender and receiver have to be in the same closed system e.g. Cash App. This is because the underlying rails are not fundamentally changed in this phase and they are still slow. (There are some recent improvements e.g. with FedNow). Cross-border still remains slow and expensive.

Payments Rails 3.0: Blockchain, Stablecoins, and Programmable Money

Anika is now a content designer in Kenya working for multiple clients globally. She has setup a crypto wallet using Coinbase and receives payment from clients in USDC. She can choose to keep it in stablecoin or move it offramp to local currency seamlessly as her fintech app leverages the global integrations that Coinbase has invested in. She can also send the money directly to her family’s crypto wallet, who can then convert locally or spend at merchants accepting USDC.

Anika has started a firm that has multiple AI content generators that produce articles for a small fee. Each AI agent has a wallet that once a client agent confirms delivery, contracts release micro transactions automatically. AI can technically “own” assets on-chain (though legally a business still typically backs it).

This example seems distant but the underlying technology is almost here. In the Payments 3.0 phase, the old legacy rails are replaced with the new blockchain stack. You can now build the slick UX of Payments 2.0 and more while leveraging new rails underneath which overcome the limitations of Payments 2.0. What are these rails?

Technology: The foundation of this technology is blockchain - which is described eloquently in this paper. It uses a novel system for conducting electronic transactions without relying on trust - using coins made from digital signatures and a peer-to-peer system leveraging a public history of transactions, proof-of-work and computational complexity to prevent attackers.

Over the past years, we made foundational improvements in underlying blockchain technology which have reduced costs and increased speed of executing crypto transactions, including stablecoins by over 99%. Sending USDC on Base (Coinbase’s L2 network) - costs cents versus $25+ to send international wires!!

These investments enables anyone to have a digital wallet, move money from fiat currency to their wallet into stablecoin like USDC, and transfer that almost instantly to another global wallet either manually or programmatically through contracts.

The main components of this are:

• Decentralized Infrastructure: The underlying “rail” is a distributed ledger and a novel way to make payments without trusted entities.

• Stablecoins: Digital assets (e.g., USDC or central bank digital currencies) pegged to a fiat currency, aiming for minimal volatility while using blockchain’s efficiency.

• Real-time, Global Settlement: Transactions settle in minutes on-chain, potentially 24/7/365, bypassing slow clearing houses.

• Programmable Money: Smart contracts enable automated financial operations (e.g., escrow, micro-payments, conditional releases) without intermediaries.

Companies like Coinbase have invested significantly to build onramp & offramps from fiat currency (e.g. $) globally to digital assets (e.g. USDC) in a wallet. This is an open platform for e.g. you can transfer money from your wallet to any other wallet in the world.

This platform will power the next generation of slick fintech apps that want to build a 3.0 experience - with the above benefits. e.g. you could be using next gen CashApp to move ‘money’ to your friend, it might just be a stablecoin instead of $.

Over the past years, many of the challenges holding this phase back have started to be mitigated both through technology and government action - regulatory uncertainty, addressing volatility (stablecoins) & costs (L2..).

Most people in the US who are dealing in USD, will be skeptical of the benefit of this phase. However, it is worth keeping a couple of things in mind -

• most people living outside the US will benefit from putting their money in USDC for inflation hedge and easy transfers for business.

• Stablecoins are not a fad. Stablecoin activity has been steadily increasing uncorrelated to crypto trading volatility.

• Stablecoins have hit product-market fit with over $xxB in USDC alone. In this world, USD is the primary reserve (over 99%) - even better than its fiat position currently. If we do not move quickly in this space, we risk another currency becoming the reserve currency in this digital asset world e.g. euro.

• AI agents having a wallet that they can get paid to by another human or agent is not possible in the 2.0 world. A CashApp account requires a human entity it.

As the bitcoin paper proposes - the eventual goal is to build a new system for payments & transactions.

Payments 4.0:

Is there really a 4.0? Maybe - A World With No Fiat

Imagine there’s no “off-ramp” or “on-ramp” because we don’t need fiat currency at all. Where everything —your rent, your groceries, your entertainment subscriptions—is priced in native digital tokens like USDC or CBDC. No conversion. In this world, we don’t talk about crypto or fiat. You don’t wonder how do I convert this to dollars? - because no one around you needs dollars. We have a single, digital monetary world.

There are a lot of questions and work that needs to be done to get here. If you want to help create this future and have an outsized impact on future of Payments - please reachout.

A quick comparison of the different phases:

Where Do We Go From Here?

Most of us live somewhere in 2.0 —tapping apps like Venmo or CashApp and maybe dabbled in stablecoins or crypto. Is 3.0 mainstream? Not yet. Will it be one day. Very Likely!

For now, the next time you transfer money overseas or pay a friend back for pizza, pause for a second. You’re on a ride that started with paper checks and is barreling toward a future where money might be as universal—and invisible—as Wi-Fi.

When Alex, a senior engineering lead, joined NextAI, he eagerly implemented the strategies that were successful at his previous FAING company. He rolled out similar workflows, planning meetings, and introduced his favorite tools, confident they'd lead to success. But instead of thriving, his new team stumbled—projects slipped and morale dipped. Alex was puzzled that his old playbook didn't fit this new game.

That reads like a fictional excerpt from Three Dysfunctions of an AI Executive but the story is not unusual.

I worked at multiple startups before I worked at AWS. It was a big change to go from a startup to AWS - there were more people on my first Large Scale Event operational review (250+) for DynamoDB’s outage than number of employees at a small startup! Similarly as I moved to Cash App, the culture was quite different from AWS.

Across these startups, two large services in AWS, and Cash App more recently, I found the 50→500 framework useful to keep in mind as you develop mechanisms at a new org.

The 50→500 Framework

Mechanisms (e.g. Operational Reviews) should be evaluated in context. Outside of people reasons, a big reason new mechanisms fail is that the context they previously operated in, for e.g. at Microsoft, is significantly different than at the new company. Another reason they fail is that they are not effective at this size. Let’s dig in to each.

1. What boat are you operating?

If you are operating a small speed boat, you likely don’t need the same navigation techniques used by the captain of a large cargo ship. The cargo ship captain will rely on advanced GPS systems, sophisticated radar equipment, and navigators to plot the course and avoid hazards. However, these tools and strategies may not be needed or even useful for a small speed boat captain, who may be able to rely on simpler navigation tools, and their own intuition and experience to navigate.

In a large company, there are more resources, more stakeholders, and there is more complexity to manage. They are also hard to pivot quickly. Smaller companies have fewer resources, fewer stakeholders, and significantly more flexibility to pivot and iterate. Processes that are designed to optimize and standardize operations in a large organization may not be suitable for smaller teams that needs to remain lean, agile, and adaptable.

As an organization leader, understanding the boat you are operating and it’s context/strengths is important to the mechanisms you apply to operate it. For e.g. unlike my previous startups, big services at AWS were like large ships, it took time to turn them but once you get them pointed in the right direction, they can travel with speed and have massive impact. When I joined Cash App, we were earlier in lifecycle, able to pivot quickly with a strong culture of bottom up innovation. Any mechanisms we employed had to embrace that.

Do these mechanisms have multiplier effect on your team’s strengths and align with objectives at this stage? Mechanisms that you apply to your team should leverage and play to your organization’s strengths. E.g. Startups or small teams win because they can be nimble / quick to react. As a result, the mechanisms that you implement when your team is small should leverage the shared context and ability to react extremely quickly. These practices should build greater flexibility, facilitate quick customer feedback and drive agility in development, experimentation and decision making.

2. Does the mechanism work at this size?

As your team size changes, mechanisms that work at one order of scale may not work at next (or two) order of size. The initial temptation for most teams and leads is to pedal harder on the mechanisms that have worked so well and optimize them. Initially that helps but it stops working with continued growth and then additional effort does not make much difference. You need a different set of mechanisms.

The reason it stops working is that the complexity grows non-linearly with size, and you just need a completely different mechanism to deal with different stages of that curve.

Let’s take one example mechanism (1:1s) and walk through this. When your team is very small, you can do 1:1s with everyone to share context, catch issues and gauge team health. As your team grows to say 35 people, you start to stretch this mechanism by ‘pedaling harder’. It’s harder to do 1:1s with everyone but still possible to meet everyone and build context.

As you grow past 50 …. 100..300 people in your team, pedaling harder on the same mechanism (1:1s) does not work - you need to rely more on mechanisms (and organizational structure) that provide signal at larger scale than direct connections. The key reason this mechanism needs to change significantly is that as your team grows - the number of people you directly meet will be a small fraction of the total number in your team. You need to start developing other mechanisms to get a signal similar to when your team was smaller.

As you go from 50 —> 200, you go from direct 1:1s to leveraging your organization structure to share context, and build in multiple other ways - 1:1s, skip 1:1s, group chats, office hours, broadcast channels, all hands and you start to rely more on metrics (e.g. attrition rates..) and data (e.g. surveys) to determine team / people health.

You still need to dive deep (more on that in another post) so you are grounded but you just cannot dive deep on everything. You need your mechanisms to give you good signal for when to.

Similarly for other mechanisms, think about the 50→500 framework and see how to evolve the mechanism as the complexity grows. We will cover this in the next post.

When I started in DynamoDB, our team was getting paged ~50 times a day ie 300+ times a week! Instead of the typical weekly oncall rotations, the team moved to a daily rotation to deal with this pager load initially, and then to a 12 hour shift between US and Dublin.

Shipping new releases would take months as we did not have reliable automated tests. Some features were way behind. Engineers would constantly get pulled into operational issues in every sprint. Some of our operational metrics were going the wrong way and we had one of the biggest outages in AWS.

Over the next year, we were able to get out of that downward spiral. Pager load reduced to 10-15 pages a week - while the service itself scaled 5x (75 million requests/sec, 5 nines of availability) and we also delivered multiple new products like backup/restore. The first time we went a full day without a single page, my manager, asked me if the ticketing system was broken!

How did we get out of that oncall spiral?
Many products and services go through phases - a phase where they are innovating significantly and accumulate ‘debt’. Some amount of debt is healthy, otherwise you maybe over engineering early on. Too much debt will eventually result in a massive drag on the team. DynamoDB had gone through the initial innovation phase and had accumulated ‘debt’.

1. Diagnosis: The first step is identifying (diagnosing) the state your team is in. That will usually dictate what strategy to use. In this situation, we were continually falling behind (i.e lack of predictability) and oncall load was unsustainable -fixes like going to 12 hour shifts usually only address symptoms and don’t address the root cause. The team was not able to make progress on the feature backlog or address root causes of issues. As a result, they were falling behind. This diagnosis drove the strategy.

2. Strategy and framework: Given the situation and challenges, our strategy and tactics, were largely focused on 3 areas. These sound simple but it requires business / leadership support and explicit hard conversations to enable this.

   1. Create Focus and Predictability:

      1. We had conversations with product and business to identify must-have business priorities or features. The rest were deprioritized. The product team was supportive as they recognized that the current situation was not sustainable. In some cases, what they needed was predictability to better manage customer expectations. On the engineering side, running more things in parallel adds to cognitive load and requires leadership (technical, management, product) bandwidth. Doing fewer things at a time helped the team conserve and put more behind important arrows.

      2. We then expanded the box that was allocated ‘to keep the lights on’ in each sprint. For e.g. instead of just one primary oncall, we created a secondary oncall focused on root cause. This allowed us to focus on fixing root causes when issues happened instead of trying to get off the 12 hour rotation as quickly as possible. While this was more ‘planned’ operational oncall allocation than previously, it led to better predictability in sprints versus having to unpredictably pull people every sprint to deal with operational issues. As we made progress in fixing issues, it had a nonlinear impact in quickly driving pager load down.

   2. Fix Root causes: We also analyzed all the categories of issues to create a plan to address the areas that would reduce the most pager load. This plan and progress was reviewed regularly with leadership. Sometimes fixes were simple like tuning, in other cases it required building new automation or solutions. For e.g. we found a category of issues that paged on StorageNode failures and we invested more in our auto-healing process. Similarly we invested significantly in automating the testing to bring down manual processes from 6 weeks to less than a week.

   3. Additional staffing: After some analysis, we also identified that the team was quite understaffed to handle both the maintenance and feature development. Running a service like DynamoDB which was required for every other AWS service was a significant operational load. The service had attrited senior engineers previously due to high load and struggled to hire new engineers, who are understandably reluctant to join a service with high operational load. The most common question that internal transfers asked me was pager load (more on how we hired for the service in a later post). We did a lot of work to hire and build out the critical areas of the service, which allowed progress in multiple areas simultaneously.

3. Measuring and Rewarding progress: As we made these changes, we tracked progress closely and celebrated wins in this area. In many companies, new feature work gets attention and the work to improve system and operational performance does not get the same level of recognition even though keeping systems running smoothly and sustainably is perhaps more important. We made a conscious effort to reward work in this area including in promotions, calibrations and team communications. If you are interested in building a culture of operational excellence in your teams, this post covers those aspects more.

Through a lot of hard work, the team eventually not only got the service load to a manageable level but also built significant innovations as well.

Please share with your friends!

Share

If you woke up Andy Jassy (when he was still leading AWS) in the middle of the night and asked him the top challenges in one of the AWS services e.g. Redshift, he would likely be able to tell you immediately!

One of the reasons he can do that, is AWS’s mechanisms that work at a massive scale across 100s of services  - e.g. PE reviews, operational reviews, WBRs, OP1s etc. These practices allowed AWS to scale the business organically from 2 services with less than 100 people to 100s of services with 20,000+ people in less than 10 years.

The most effective SCALING mechanism (and secret sauce) is their leadership principles. If someone told me that a set of principles is the secret sauce for a $1T company, I would be skeptical too!

Most companies have core principles. When I joined AWS, I was surprised by how often folks internally referred to Amazon’s leadership principles. Most of them probably could tell you all the principles (all 14 16) instantly!

So what.. Why does it even matter and what does it have to do with scaling?

Designed correctly - Tenets/principles capture and guide behavior at the company that is valued. It is the fabric of the culture at Amazon. Folks who are not aligned with these principles find it hard to thrive at Amazon.

How has this helped Amazon scale? Amazon’s principles guide how decisions are made in the company at every level. You will see these principles internalized in calibrations, in feedbacks, in promotions, in hiring and used in many project conversations. Some principles like Customer Obsession deeply influence the company’s decision making frameworks. Every hire is based not just on technical competencies but how well they align with those principles. E.g. Ownership is a principle and your interview will likely have a question along - ‘Tell me about a time that you acted on behalf of a team outside yours’.

Having a shared rubric that is so well understood and internalized in daily vocabulary has allowed Amazon to scale to 900,000 people rapidly and push down decision making. AWS leadership doesn’t have to be in the hiring loop to ensure that DynamoDB’s next engineering leader is tested for culture fit and has customer focus. During calibrations or hiring bar, everyone in the room knows precisely what someone means when they say ‘X has good bias for action’. Leaders understand that they need to balance ‘bias for action’ with ‘insist on high standards’ on a project.

Why does this mechanism work at Amazon and how you can use this (CAP) framework?

1. Create principles that embody and guide the culture you want reflected in your teams. Ensure these principles reflect the judgement in day to day decision making. These often requires balancing multiple ‘aspirations’ that maybe at odds with each other - e.g. Move fast and Ship Quality Products. A quick litmus test for this - if you are not in the room and the set of folks making a key decision had these principles - would that allow them to consider similar facets that you would too? If yes, then good otherwise maybe refine your principles.

2. Apply these principles in daily decision making. Teams ideally internalize and use these principles in their work. E.g. the lambda runtime teams used Customer Obsession and Insist on High standards - to establish ‘Ease of use’ and ‘highly secure’ as guiding tenets for their project. This enabled engineers to balance both needs and not ship a product that is super secure but painful for customers to use.

3. People processes: Incorporate and connect these with people processes like hiring and reward systems. E.g. If ownership is one of your principles, that should be one of the core attributes for which you hire and you should reward folks who demonstrate it.

Please share with your friends.

Share

You have an idea for a new ChatGPT based Quickbooks competitor - ‘EasyBooks’ that lets small business owners manage their books powered by ChatGPT.

A small business owner can connect to their payment systems like Square and easily chat with EasyBooks - ‘how much money did we spend last month’ and then create actions - ‘schedule a reminder email’... all in natural language.

How do you evaluate this idea for a new startup?

I like to use an onion metaphor to describe the framework - Each idea is like an onion and you peel the different layers. Before we decided on the idea at my first startup, Livemocha, we spent 4-5 months looking at multiple other ideas that we discarded. It was a learning process for me and below I describe the framework we used, which has 3 areas and WHY its useful.

1. Risks

The first area we spent time on was risks. Risks in your idea include: Product Risk - how hard/feasible is it to build this, Market Risk - do people want this product, how will you get adoption, Legal Risk - are there regulatory or patents etc. that would potentially threaten this, Funding risk - is this a fundable idea, does it require a lot of capital, can you raise funding, Business / Competition Risk - is there a lot of competition, can we build a scalable business for this, is this a big market, Team Risks - are we the right team to build this, Do you have a critical insight into this space?

• Risks by themselves are not bad. If there were no risks, this would already be done. Use this to think deeper about the risks and how you will mitigate them to build a viable business. Areas you need to dig into more.

• Not all risks are equal. Some risks require more scrutiny than others. e.g if there is a legal risk.

• Competition is not bad. It means there is a market. If you find yourself saying there is no competition, ask why? Is that because there is NO market OR people have tried and failed. If so, why will you have a different outcome? No competition is a myth. Even if its true, you will at most have 3-6 months before competitors come in. Assume other people are working on the same idea.

Most startups fail because of Market risk - enough people don’t need your product at this time.

2. Business Model & Distribution

The second area where we spent time is on the business model (tied to business risk). Not a detailed MBA model but try to answer the question - How will you make money? Often when you work in a bigger company or in functions not directly responsible for revenue - you are shielded from how hard it is to make money in a business. You start to think if you build, they will come. That is rarely the case.

One of the investors at Livemocha once told me - the two hardest things in a startup are both to do with money - getting someone to invest in your business and getting customers to pay!

This forces you to think of basics of this e.g. Is this a big market, customer acquisition (i.e. distribution) and costs, lifetime value of customer. The model and scaling path is different for consumer and enterprise companies. For most consumer apps - distribution is a big problem. It’s much harder to drive virality and engagement than you assume. This exercise also forces you validate assumptions E.g. if you model that 30% users (vs 1% norm) will create content on your app, you should have a strong reason for that.

3. Moat / Defensibility

The third area that is important to think about - How defensible is this from competitors. The easier it is to build a product/launch, the easier it is for someone else to do the same. Many of the thin layers on top of ChatGPT face this risk. Many Groupon clones faced this challenge.

If there are ways that ‘the product gets better with additional customers’ - that gives you network effects that a competitor cannot immediately emulate. E.g. while someone can copy Quora’s technology quickly, the content/SEO will take a really long time.

Why Bother?

Startups are hard, you will likely go through a lot of ups and downs. Like the Mike Tyson quote on plans and punches, you will change many things on your original plan as it meets the real world.

In some cases, the biggest risks are not apparent until you are deep in the space (e.g. how aspirational language learning is) or will emerge later. The more important thing is that you adjust at that point.

WHY bother doing any of this analysis if things will change anyways?

1. I find the lightweight framework useful. It helped us refine the idea at my previous startups as we looked at it from different dimensions. Further the time spent looking at previous ideas and recognizing risks in them later helped us connect the dots between social, learning, marketplace to come up with a new idea eventually.

2. Use this more as an aid and not an end in itself. Any framework like this has large assumptions and will likely be stale quickly. Extra time getting to high precision on this is not very valuable since it has not met the real world!

A lot of your startup’s eventual outcome will come down to market, team, space (and of course luck) INSTEAD of the exact specifics of the idea or model.

(early startup days in my basement)

Back to the ChatGPT based EasyBooks idea

The nice thing about this idea is that it leverages a platform shift (AI) to take on an incumbent. You might look at the different areas of EasyBooks and evaluate the risks to something like this: (these will be different for different people).

Good luck with the EasyBooks IPO!

Thank you for reading. Please share.

Share

It was 2am on a Sunday in 2015, when my pager at AWS DynamoDB went off. Within minutes, we started to see 100s of alerts and pages. It quickly became clear that the service was having serious trouble.

DynamoDB is a tier-0 service at AWS, a foundational service, that is among the first services to be installed in any new AWS region because every other service depends on it. The incident took us multiple hours to start to recover and almost 4 days to fully get back to normal. It was the largest outage in AWS history!

The war room experience, recovery and the subsequent weeks, months and years working through the post-event operational and scaling issues on one of the largest AWS services (60+ million requests/sec), turned out to be a great learning experience. I still recall my mentor Marco telling me at that time - ‘there are somethings you can only learn through hard knocks’.

Systems usually require re-design when usage grows by orders of magnitude or more. As a rule of thumb, when you build a new system, you should be able to handle one order growth. If you are designing for 2 orders or more, you are probably over-investing upfront. In DynamoDB’s case, the service had grown multiple orders since launch and that led to this event.

Issues are always going to happen - you don’t want to be so risk averse that you stop pushing things out. How much should you invest in Operational excellence? There is no one size fits all answer here. Depending on the stage of your service and customer expectations, you may choose to invest little or a lot at a given point in time.

At DynamoDB and Lambda , we invested in two areas to help us keep operational pace:

1. Building a culture of Operational Excellence in our teams, where we learn from incidents, share learnings and incorporate actions to prevent similar issues in future.

2. Investing in building ways to detect issues and recover as quickly as possible.

Operational Excellence Culture

• Set up operational/SLA goals for services: All services in AWS take SLA goals e.g. availability. This helps them make tradeoffs between an operational investment (e.g. active-active database) vs a feature investment. How did we set operational goals? These goals are typically set as a combination of business needs and demands that would place on team operationally. For e.g. early on in Lambda we started with SLA of 99% based on business acceptability and pager load it would place on teams.

• Regular Operational Reviews in your teams: Service leaders review operational performance across their teams weekly. It helps them stay close to the ground and also reinforces that operations matter.

• Learn from Operational Incidents: Every operational event is an opportunity to improve. Conduct a Post Mortem and take actions. For e.g. by looking across many incidents, we found that significant % of our incidents were triggered by deployments. Unsurprisingly, the most common mitigation step is to roll back. This led to an audit step for deployments that teams had to answer - Can you safely roll back what you are about to deploy?

• Share learnings: AWS runs a weekly operational review every Wednesday across all services. It is a forum to share learnings and best practices across teams. In the meeting, teams share successes, review key operational events and review service metrics for 1–2 services - chosen at random by spinning a wheel.

  

Detecting Issues and Recovering Quickly

1. Invest in identifying issues as early in lifecycle: While the mechanisms that you choose may vary, the key principle is that the cost of fixing an issue goes up by order of magnitude as you go from your dev machine to each subsequent step.

As a result, invest in systems that catch issues earlier in that cycle - e.g. automated tests in staging or canaries on production or doing a limited rollout in production. In AWS for e.g. we deploy first to a ‘onebox’ and monitor its metrics.

2. Reducing Mean Time to Detect: Similar to above, mean times to detect (and therefore recover) are typically an order of magnitude higher as they go from automated detection.

As a result, investments in proactive alarming will make a huge difference e.g. having headroom graphs in your dashboard and alarming at 75% capacity could prevent a bad capacity issue on production. Take a look at your team’s incidents from the past months and see what percent were manually discovered and start putting in alarms to bring that number down.

3. What should you monitor? Here is a great tutorial on how teams at AWS do monitoring and setup dashboards.

Every organization’s stage and culture is different. Things that work in one place may not work at another for a variety of reasons. However the core principle of ‘why something works or does not’ is often useful so you can decide how/if to apply in your situation.

My goal is to share some of the practices, war stories and narratives from the different companies and experiences - from my basement startup to AWS.. And dive into the key principles and frameworks.

Let’s get started! In the meantime, tell your friends! Welcome to the 5 Whys

(signed poster from some of the folks when I left Livemocha)

While it’s tempting to adopt practices, every organization’s stage and culture is different. Things that work in one place may not work at another for a variety of reasons. However the core principle of ‘why something works or does not’ is often useful so you can decide how/if to apply in your situation.

5 Whys? When we typically have a system issue, many engineering teams typically do an exercise later to look at the situation, ask why and identify roots and potential actions. My goal is to share some of the practices, war stories and narratives from the different companies and experiences - from my basement startup to AWS.. And dive into the key principles and frameworks.

Subscribe now