From Chaos to Clarity: A Step-by-Step Guide to Mastering System Design

I am a software engineer who has worked on many systems ranging from small and simple to large and complex. I have had the opportunity to design, build, maintain, and support those systems. In my years of experience, I have realized that quality begins in the design phase. I attempt to summarize that experience and give you some tips, even checklists you can use in your next design task. The content is intended to be general, so you need to adapt some of the steps to your specific use case.

Opinions expressed are solely my own and do not express the views or opinions of my employer.

Step 0: The Problem Statement

All successful engineering projects begin with a well-defined problem statement. Each project is different. Often, there is a gap between what you think is needed and what is actually needed. Sometimes, a Product/Project Manager (PM) can articulate those requirements into a nice document. Other times, you will have to do it yourself. In any case, here are some helpful things to do:

• Ask lots of questions. The only stupid questions are the ones you don't ask.
• Rephrase and confirm. At least for the important bits, rephrase the requirements in your own words and ensure you got it right.
• Do a feasibility analysis.
  • Good PMs are unsurprisingly good at weeding out wild goose chases. You should still conduct your due diligence to complement it.
  • You may need to do some technical analysis in some cases. Be sure to budget for that.
• Check for red flags:
  • Security risks.
  • Privacy concerns.
  • Data availability issues.
• Agree on trade-offs. Document trade-offs based on feasibility, security, and privacy aspects. Discuss and agree on the proposed trade-offs. Doing so will aid clarity and help your design process.

Step 1: Collecting

We have a problem, a good one. You know where you want to go but don't yet know how. It's like knowing you want to get to the Grand Canyon, but you haven't found the cheapest flights or the best route. There are a lot of moving parts.

You may have a lot of information on your hands or barely know anything at this point. That is normal. When I am in this situation, I do this:

1. Grab a pen and paper.
2. Draw some rough boxes for "things" that make the system tick. Don't worry about getting all the boxes right at this point.
3. Connect all the "things" with lines. Now, you have a rough flow.
4. Iterate on the flow until I have a rough diagram that solves the problem.

Step 2: Categorizing

Now that you have all the fuzzy boxes you need, it is time to divide and conquer. You can divide the rough boxes into 3 categories:

• 1. Clear:
  • There are tried and tested solutions to these, or at least a clear path forward.
  • These "basics" are available to you: Specifications, Documentation, Support plans.
• 2. Frosted:
  • These are the same as the clear boxes, except some basics are missing.
• 3. Fuzzy:
  • These are the interesting ones (in a good or a bad way).
  • Most of the basics are missing but you have somewhat of a path forward.

Step 3: Prioritizing

“If everything’s a priority, then nothing’s a priority.” - Frank Sonnenberg

You must have heard that numerous times. Think about the things that this product needs to do. Is it resource efficiency, low latency, high availability, or all of those things? Which of those comes first, which is second, and so on. This list of priorities is crucial to how you approach designing your product.

Step 4: Discovery

Now that you have the rough boxes and all the little squiggly connecting lines, it is time to dive deeper. In this step, you try or test different options for each rough box. You will need several things for each of them:

• Specifications & User Documentation.
• Contacts: If a component is from another team, you will need at least one point of contact. This is useful if you have questions while exploring the component.
• A Test Rig:
  • Chances are, whoever built the components you are using already has a test rig. Beg or borrow so you don't have to reinvent the wheel.
  • If there is none, build a simple one yourself. Do just enough to test the basic functionality you need.
• Dependency Analysis:
  • Create a list of known dependencies.
  • Check for known vulnerabilities. The presence of vulnerabilities may influence your decision to use or not to use a library.
• Notes: Keep well-organized notes you can refer to later, especially for large systems with many details.

It is okay to collect the items from the list for one box at a time. The idea is to not get overwhelmed by the breadth of the system and work in small, easy-to-handle increments.

Step 5: Assembly

By this time, you should have a pretty good idea of what everything does. Now is the time to grab a pen and paper again. This time, look at your initial drawing and make a more detailed version. This drawing will give you a good picture of how everything works.

Time for another checklist:

• List the scenarios:
  • Happy path.
  • Stuff is on fire (Error handling).
  • Transient failures (Retries/Backoff).
• Diagramming: Draw a sequence diagram or a flow chart for each scenario.
• Test Plan: Write a comprehensive test plan.
• Review Dependencies: Recheck the list of dependencies and vulnerabilities. Do you need to add or remove some? Is a dependency too hard to handle? You may have to redo discovery for that functionality.
• Documentation: Put the design together in a neat document.

Key Takeaways

The design won't be perfect in the first iteration. There is no shame in going back to it and redoing some bits. Make sure you are weighing costs, risks, and benefits wisely. Lastly, estimate and put everything into whatever work tracker you use at your company.