Hugo 1 weeks ago

Alignment, Autonomy and context

Over the years, I've met and observed teams that were competent and yet failed: working on the wrong priorities, sometimes working on the right objectives, but with unsuitable solutions, sometimes working with no priorities at all. The root cause of these problems can be articulated around these 3 words: autonomy, alignment, and context . If these words seem relatively simple to understand, in practice, their implementation is not trivial. How do you ensure that everyone in a team is trying to solve the same problems? What does autonomy mean within a group, and what are its limits? Perhaps you've heard the term "empowered teams"? And thought it was yet another new consultant concept? The expression gained popularity via the book Empowered by Marty Cagan . In simple terms, an empowered team works on a problem to be solved, as opposed to a feature team which works on a list of functionalities (the famous roadmap). We also speak of a team that prioritizes outcomes (impacts) over outputs (deliverables). The first thing I'd like to stress is that this isn't just a topic for product managers. Any senior person in Engineering should be involved in decision-making, for example : I'm writing this chapter for any technical leader looking to create impact. To do this, we'll see that creating autonomy, alignment and knowing how to communicate the context are levers for achieving this. Here, as a Tech Leader you can be in two situations, but often in both at the same time: To tackle this subject, I'm going to start from a drawing by Henrik Kniberg about alignment and empowerment. "Empowered teams" being the teams in the top right-hand corner. This drawing is part of a keynote given in 2016 that I invite you to watch. In this keynote, it's about a group, whose goal is to cross a river to settle on the other side of the bank. The lower-left corner shows a group with little autonomy, no alignment and no precise directions. No one in this part of the quadrant has taken the initial problem of crossing the river personally. The symptoms of such an organization in a technological context : What can be done about it? It's a tough job on this part of the quadrant. Even before talking about autonomy, the first issue will be to get people to agree on a common goal. The role of a tech leader is to get close to the business, or its closest representative, and understand the issues that need to be addressed. In this type of organization, the business is often far away and/or difficult to access. Breaking the distance can be difficult, and is not part of the company's culture. Technique and "best practices" are likely to be the least of your worries in this context, as the importance of connecting Engineering to the business is paramount. If we move to the right, we come across organizations made up of individuals or groups of individuals, each of whom has defined objectives, but all of which are different. If the group's objective is to cross a river, everyone is working on different things: some are growing vegetables, others are fishing, and a few leaders are wondering whether someone is working on crossing the river. There's a misunderstanding of autonomy here. Autonomy does not mean independence. A team that decides its priorities in its own corner doesn't make an impact, or does so by accident, and this success is hard to repeat many times over. And this doesn't mean that individual groups can't be highly effective. Unfortunately, impact is measured in global terms. The symptoms of such an organization in a technological context: What can be done about it? The observation is fairly similar to the previous one, but the work will be less arduous due to the highly entrepreneurial nature of the company. Here, the tech leader will be able to rely on committed teams. We'll need to work on alignment. But let's define this notion a little better. Alignment defines the ability of a group to all seek solutions to the same problem. Objective: A European company decides to enter the US market. Each team will seek to contribute to the same objective. However, in the drawing in the upper left corner, we can see a clear alignment without autonomy. One or more leaders can decide exactly what needs to be done to achieve an objective. The leader explicitly asks to build a bridge across the river. This mode of operation can be very effective, especially on a small scale with a visionary leader. But scaling up can cause problems, as the visionary leader can't spread out across the company and follow all the decisions. On several occasions, I've observed companies with very good leaders, but with great difficulty in managing rapid growth. They recruited people who were more junior than they were, more capable of carrying out a task than making decisions. These people then failed to scale up in the time available. Symptoms of such an organization in a technological context: What to do about it: If you're a tech leader, one of your responsibilities is recruitment. You have to recruit people who are better than you in certain areas. You have to learn to share your legos . The second important thing is to get out of the vicious circle: At stage 2, you have a major responsibility for coaching/mentoring to educate and provide the necessary context for more effective decision-making. If you're a tech leader, but you're in a situation where the work is delegated to you, you'll need to show that the team can be autonomous. To do this, two things are important: In our example above, a group that chose to build a catapult to hurl individuals to the other side would be showing a certain lack of understanding of the effect of gravity on human bodies :) Hopefully, the initiative would be stopped. Don't be the one to suggest the catapult . Bad: there are too few of us in the team, so we can't finish the project on time. Good: given the size of our team and our current speed, I've decided to cut back on some of the functionalities to meet the deadline. Bad: We're drowning in support. The team can't deliver project A on time. Good: We carried out a support analysis and noted a sharp increase in support cases linked to a bug in a recent feature. We decided to assign one person for a week to fix this bug and regain velocity on project A. Before I talk about the last quadrant, I need to talk a little about the "context". "Lead with context, not control" is a quote you'll find in the excellent book "No rules rules" by Erin Meyer and Reed Hastings. For a person or a group to make a good decision, they need to know the whole context. If we go back to our group, which has to cross a river, we lack the context to propose initiatives. Maybe we have to cross the river to find food. In which case, why didn't we decide to settle on this side? The leader failed to mention that we're in the territory of another tribe who has asked us to leave before autumn. Okay, and if the leader had told us that this tribe wasn't against trading with us after all, having a bridge would probably be more interesting in the future than just building a boat. Having the context allows you to make the right decisions. And in the corporate world, there are plenty of occasions when you neglect to give the full context, sometimes for the sake of speed, sometimes to protect your team, sometimes because you've forgotten. But without context, you can't expect people to make the right decisions. Context gives you the constraints that will enable you to propose the most appropriate technological solution. In this corner of the quadrant, the objective is clearly defined, such as crossing the river. The constraints are known, and each group is invited to contribute to the common goal. In theory, this is the most efficient type of organization, especially at scale. I'd like to point out two pitfalls at this stage: It doesn't make sense to hire smart people and tell them what to do; we hire smart people so they can tell us what to do. You have to accept the unexpected, accept that the solutions you come up with may not be the ones you originally imagined. If a group suggests crossing the river with a boat or a tunnel, or has spotted a ford a hundred meters downstream, it may be different from what you had in mind, but it may be more effective. Accept that not everything is as it seems in your head. BUT, if there are several groups working, one on building a tunnel, another on building a bridge, another on building a boat, you've got a problem. If everyone is working on the same problem with different solutions, that's also a form of misalignment . Again, autonomy doesn't mean independence, and there needs to be regular discussion around the initiatives launched to ensure overall coherence. Don't be the one working on creating a tunnel while everyone else is trying to build a bridge. Certain methodologies, such as OKRs for example, help to materialize this periodic discussion. Objective: The product should enable the acquisition of 10,000 prospects in the USA. Quarter 1 initiatives: Quarter 2 initiatives: The coherence of the initiatives is linked to a discussion at the beginning of the quarter of what each group has come up with to contribute to the initial lead acquisition objective. This step, designed to ensure the coherence of initiatives, can be found in many methodologies: I must stress once again that autonomy does not mean independence. If, during these discussions, a decision is made to favor a bet, and it's not the solution you had in mind, too bad but you have to commit to this decision. This is the famous "disagree and commit" . And that doesn't take anything away from team autonomy. Teams can be the driving force behind initiatives to solve a given problem, but in the end, we'll be looking for alignment. Otherwise, we're back in the bottom right quadrant. If your aim is to create more autonomous product teams, you may come up against the following difficulties: These 3 subjects create deadlocks. A good PM and/or Tech Leader will be reluctant to join an organization where he/she feels a lack of autonomy. Management will find it hard to trust a team that they don't feel is business-oriented enough. Top management who don't have confidence in a team will continue to demand a high degree of predictability in development via detailed roadmaps. As a Tech Leader, you have your hands on several levers that I've already covered in the first chapters: The aim of all these actions is to increase the trust placed in Engineering's actions. This confidence will be transformed into the delegation of autonomy . Would you say that you have all the constraints in mind for your current projects? How much time do you spend explaining the context to the people who work with you? What is your team/department's objective for the year? How are you contributing to this objective this quarter? an empowered team's mission is to work on acquiring new users a "feature team" has to come up with a sponsorship program an "empowered team" is tasked with improving the developer experience by reducing dev cycle times a feature team is tasked with implementing a new CI (continuous integration) solution. avoiding the construction of unnecessarily complex solutions, demonstrating a misunderstanding of objectives checking that these priorities are built with the right constraints in mind ensuring that these priorities contribute to the group's objectives. the person seeking to develop his or her group's autonomy the person who belongs to a group with limited autonomy, and is seeking to gain independence. reaction-only teams, i.e. whose work is driven by support and bugs, upgrades to end-of-life components, and so on. no hyper-clear objectives, but a kanban (a big todo list) that feeds itself organically micromanagement, with tasks assigned by a manager/project leader a technical team that works in isolation from users, often in supplier mode for a team that defines the product, but itself far from the business. several groups work on defined topics, but without collaboration with other teams. a large number of projects are underway, stretching out over months. This is linked to the "one-man army" phenomenon: if a single person can move fast, he or she ends up burning out faster than a larger group. This slowness makes teams weary, as they continue to pile new subjects on top of the list. the multiplicity of topics creates vagueness throughout the organization, which becomes a black box for the rest of the company the perceived impact is low The sales department will invest to open a sales office in the USA. The finance department will open a subsidiary in the USA and modify its invoicing model to comply with current regulations. The engineering department will study how to reduce product latency through a better geographical distribution of its infrastructure the leader is the most experienced member of the group, with a significant gap between them there is a strong imbalance between junior and senior staff, to the detriment of the latter the team is not involved in decision-making, and the backlog is fed by a single person the people I work with don't make the right decisions or don't know how to make decisions, so I step in to make them for them for the sake of speed I don't spend much time on the explanation the people in question end up concentrating more on their expertise than on decision-making. Develop your business understanding . You need to step outside the team to better understand the challenges and constraints of the company and/or your customers. Develop your decision-making skills. To re-establish the symmetry of exchanges with your management, you need to keep one thing in mind: you need to provide more solutions than problems . You won't be trusted if you only come up with problems to solve, and your discussions will systematically turn into basic reporting, so you'll have to make choices. Team A works on improving SEO through multi-domain management and regionalization. Team B sets up several new domain names for multi-domain management Team C works on SEA to refine geographic targeting of campaigns. Team A works on a sponsorship program Team B works on sharing mechanisms on social networks Team C works on affiliation mechanisms The objective is defined in advance. Teams propose initiatives to contribute to it After discussion, a set of initiatives is chosen. in OKR via the OKR Quarterly review fixing appetites in the ShapeUp method the construction of "bet boards" by Henrik Kniberg Program Increment Planning in SAFe recruitment. To get started, you need very good PMs and Tech leaders who have an entrepreneurial mindset as well as a technical one. Transformation is all about people. top management buy-in and a change of mindset are essential. They must agree to delegate the "how" to product teams, and focus on the "what" (objectives). And that's a lot harder than it sounds. Many say they want to move to autonomous product teams, but are very frustrated at not being able to tell them what to do agreeing to abandon traditional roadmaps in favor of strategies based on objectives (OKR or other, it doesn't matter). Develop your understanding of the business Measuring everything Know how to prioritize Create time for ideation Know how to communicate Give away your legos and other commandments for scaling startups No rules rules keynote from Henrik Kniberg en 2016 Empowered from Marty Cagan

Agile Business

Nick Khami 1 weeks ago

Replace Your Standup with a Todo List

Note: You can watch me write this blog post on video here on x.com! Standups create anxiety. You hoard updates during the day because you need content for the meeting. You forget your blockers. You adjust your schedule to be in early, even if you work better at night. I ran into this when I started my company. People join startups to escape corporate bureaucracy. When I tried to introduce morning standups, the early hires pushed back hard. It didn't feel like a startup move to them. Given that, I went back to the drawing board to break down the actual utility of the meeting. A standup exists to distribute context so a team can parallelize work and maintain synchronization. If that's the goal, you don't need a meeting. Here's what one of those messages looks like. As you work, you come back and edit the message to cross things off. Add timestamps if you want to track how long things are taking you or otherwise provide more context. That's it. Everyone sees what you're working on. No meeting required. Post your task list in a channel like first thing in the morning Edit the message to cross off tasks as you complete them throughout the day

Agile

Chaotic-Good Engineering Management 2 weeks ago

Let Your Teams Own Their Processes

When it comes to team operating principles, I'm sure we've all encountered the leader who values uniformity seemingly for the sake of it. Every team has the same ceremonies, on the same days. Status reports are via a templated slide deck with the same format and the same information as all the other teams. I get it, this can be comforting. I worked at a place like that. Every team used Scrum, estimated with story points, sprints all started and ended on the same day, and the last day of the sprint featured a full day of sprint-reviews presented via PowerPoint decks that all basically looked the same save for maybe some screenshots of whatever feature the team was working on. One of my teams, however, was really struggling. Because of the nature of their scope of ownership and who their stakeholders were, they could rarely get through a sprint without some new information or requirements coming to light that either injected new, urgent work or made some of the work they had planned pointless. Going by the book, this should have caused us to cancel nearly every sprint (we didn't, we just soldiered on). During a retro, we all concluded that this team would be better served by using Kanban and tracking Cycle Time as the primary metric for doing estimations. The story points method was proving pretty wasteful because we'd often break down and estimate work that we didn't end up doing, then have to point some new work on the fly mostly because we needed the stats for our sprint review deck. I could talk all day about how we could have handled things differently, but that's not the point of this post. The upshot is, the proposal to switch to Kanban was shot down by the execs since it proposed reporting cycle time and throughput metrics rather than the points-based velocity they were used to. (We agreed to keep doing "Sprint" reviews on the same cadence even though we weren't planning to do sprints). The perceived control they got from seeing a graph showing whether or not velocity was going up or down was too valuable to them to give up and having two systems of measurement was a bridge too far, I guess. Ultimately, a path that could have improved the satisfaction of both the team and its stakeholders was cut off to them. This is the failure mode that uniform process creates. A concept I first heard from Jean-Michel Lemieux when he was leading engineering at Shopify was creating an "API For Teams". Like an API in software, the API For Teams defines a contract for interfacing with the rest of the business, but leaves the implementation up to the implementors. A good basic interface would be something like: This information should be available any time, with reasonable expectations on around staleness for things like estimates and health. (Put it in a Slack integration or some other info delivery system and people can even query it without interrupting anyone). In an organization with good, aligned incentives , something like that can give stakeholders the information they need to make decisions without over-specifying the processes in a way that removes so much autonomy from the team that they cease taking any interest in finding ways to improve. The less you demonstrate that you trust your team, the less they'll try. The API concept lets teams decide for themselves what processes best match their skills, the type of work they do, the way they deploy software to customers. I've had teams choose to switch to Kanban from Scrum because in the world of continuous delivery they found the starting and stopping of sprints was a weird artificial construct that caused things like code reviews to bunch up until later in the sprint. I've had other teams fully embrace sprints because they could divide up a couple week's work among themselves and mostly stay heads down without a lot of coordination overhead. No one system or way of working ever emerged as "the best" in a way that I would have imposed it on every other team. Keep the API minimal and be mindful that you're not adding things on to the point where you've dictated the implementation. This is where the API concept shines. The information that's expected should be clear, non-negotiable, and where metrics are involved, not easily gamed. The incentives should be aligned around extracting truthful, honest data. Don't give teams the incentive to lie, don't punish them if they report out data you don't like. Try to help when teams are struggling, and encourage a culture of sharing so that when one team finds something that works, they can share that with other teams who might also benefit. For their part, teams need to be accountable to the outcomes, not just providing the data. If they keep missing their target dates, that's a contract violation and they probably need to change their implementation. They should hear that. Again, I understand the comfort that uniformity can bring but taking it too far will be stifling and counterproductive. On the flip side, getting too into the details s a bad use of a leader's time and they'll never have the amount of information needed to make decisions that the team doing the work has. The chaotic-good manager embraces some ambiguity and loss of control and in exchange they'll often find they get a better outcome. " Tangye instrument panel " by Elsie esq. is licensed under CC BY 2.0 . Like this? Please feel free to share it on your favourite social media or link site! Share it with friends! Hit subscribe to get new posts delivered to your inbox automatically. Feedback? Topics you want to hear about? Get in touch ! Project Current Project or Deliverable Health of the project. Phase of the project (preview, beta, GA, etc) Current estimate for next delivery People Open roles on the team Who is on-call and/or handling support escalations? Risk Current level of Planned vs. Unplanned work. Any escalations from their last retrospective that need addressing.

Agile

The Coder Cafe 2 months ago

Speed vs. Velocity

☕ Welcome to The Coder Cafe! Today, we discuss the difference between speed and velocity in team productivity, illustrating that tracking speed alone can be misleading. Get cozy, grab a coffee, and let’s begin! We often celebrate teams for moving fast. But speed alone can be a trap. A rush of fast changes that barely move the product toward the real goal shouldn’t count as a win. When we talk about team productivity, we should understand that speed ≠ velocity : Speed is how quickly a team ship changes. Velocity is speed with direction, the movement toward a defined goal. Let’s look at three teams to illustrate these definitions. Team A has ideal velocity. Each iteration is represented by an arrow: its length shows speed, and its angle shows direction. Every iteration moves the team consistently closer to the goal. Team A shows ideal velocity: steady speed and consistent direction. Team B has a speed problem. Each iteration is well-aligned (correct angle), but the team delivers too slowly (small length), resulting in a lower velocity. They finish only halfway to the goal. Team B has the right direction but low speed, so they only reach halfway to the goal. Team C ships as rapidly as team A (same arrow length). However, various factors such as frequent bug fixes and changing targets make their direction inconsistent. Despite high speed, their velocity is low, and they end only halfway to the goal, just like team B. Team C moves fast but changes direction often, so velocity stays low and progress stops halfway. As we can see, velocity requires both speed and direction. A team moving too slowly or in inconsistent directions will make little progress, even if they’re busy . Only when speed is high and direction is aligned do teams reach their goals efficiently. Measuring team speed isn’t useless, though. We can track speed by considering various metrics such as deployment frequency, average time in code review, or mean time to recovery (MTTR) following a production bug. These metrics are interesting to track and provide a certain perspective to understand a team's productivity. Yet, speed shouldn’t be the sole dimension to track. The danger of tracking speed only is that a team might become organized in a way to optimize short-term delivery . The team might focus on delivering many changes that, together, do not move the product in a meaningful way. Instead, teams should track speed and velocity. As we said, velocity is speed with direction. We already discussed metrics to track speed; what is missing is monitoring direction. Setting clear and factual objectives that align with the business strategy helps us track direction. For example: Payment success rate above 99 percent. Signup to activation above 50 percent within 7 days. Retention after week 4 is above 40 percent. The easier a metric is to measure, the easier it is to track the direction over time. One caveat is how to report progress. In a past team, we used to set OKRs (Objectives and Key Results) every semester. Some objectives were difficult to measure, so we tracked progress differently. Say we created 50 tickets and closed 45. In that case, we reported that we reached 90% of the OKR. That number said nothing about real progress toward the objective. Key results should be outcomes, not ticket counts. Something else to mention, I read here and there that we should track velocity by counting the number of story points delivered within a timeframe (e.g., during a sprint). I strongly disagree with this. Let me give an example: A team ships a first change ← 3 story points The team finds a bug and ships a fix ← 2 story points Later, the team learns the initial approach is not the best, so it ships another change ← 5 story points On paper, the team delivered 10 story points. That is only speed, though, not velocity. If we care about direction as well, the team only delivered a single feature. Story points measure effort; they don’t measure progress toward the objective. Speed is how fast we ship. Velocity is speed with direction toward a goal. The goal of a team shouldn’t be to reach high speed; it should be to reach high velocity, where rapid iterations translate into real system-level improvements. Tracking story points, for example, is a measure of speed, not velocity. Objectives tracking should use outcomes, not ticket counts. Reporting 90 percent of tickets done is not a good measure of progress toward the objective. Missing direction in your tech career? At The Coder Cafe, we serve timeless concepts with your coffee to help you master the fundamentals. Written by a Google SWE and trusted by thousands of readers, we support your growth as an engineer, one coffee at a time. Keeping a Mistake Journal Streetlight Effect Survivor Bias Sprint Velocity in Scrum: How to Measure and Improve Performance // The velocity definition I disagree with. ❤️ If you enjoyed this post, please hit the like button. 💬 What’s your take on speed vs. velocity? How do you measure velocity in your team? Leave a comment We often celebrate teams for moving fast. But speed alone can be a trap. A rush of fast changes that barely move the product toward the real goal shouldn’t count as a win. When we talk about team productivity, we should understand that speed ≠ velocity : Speed is how quickly a team ship changes. Velocity is speed with direction, the movement toward a defined goal. Team A has ideal velocity. Each iteration is represented by an arrow: its length shows speed, and its angle shows direction. Every iteration moves the team consistently closer to the goal. Team A shows ideal velocity: steady speed and consistent direction. Team B has a speed problem. Each iteration is well-aligned (correct angle), but the team delivers too slowly (small length), resulting in a lower velocity. They finish only halfway to the goal. Team B has the right direction but low speed, so they only reach halfway to the goal. Team C ships as rapidly as team A (same arrow length). However, various factors such as frequent bug fixes and changing targets make their direction inconsistent. Despite high speed, their velocity is low, and they end only halfway to the goal, just like team B. Team C moves fast but changes direction often, so velocity stays low and progress stops halfway. Payment success rate above 99 percent. Signup to activation above 50 percent within 7 days. Retention after week 4 is above 40 percent. A team ships a first change ← 3 story points The team finds a bug and ships a fix ← 2 story points Later, the team learns the initial approach is not the best, so it ships another change ← 5 story points Speed is how fast we ship. Velocity is speed with direction toward a goal. The goal of a team shouldn’t be to reach high speed; it should be to reach high velocity, where rapid iterations translate into real system-level improvements. Tracking story points, for example, is a measure of speed, not velocity. Objectives tracking should use outcomes, not ticket counts. Reporting 90 percent of tickets done is not a good measure of progress toward the objective. Keeping a Mistake Journal Streetlight Effect Survivor Bias Sprint Velocity in Scrum: How to Measure and Improve Performance // The velocity definition I disagree with.

Business Agile

The Coder Cafe 3 months ago

Organic Growth vs. Controlled Growth

☕ Welcome to The Coder Cafe! Today, we will discuss the concept of organic growth vs. controlled growth. Get cozy, grab a coffee, and let’s begin! A few months ago, I was in a meeting discussing the state of a codebase and heard myself say: This codebase has grown organically. After reflecting on my own words, I began questioning myself: I don’t even know what organic growth really means and whether it is ultimately positive or negative. Discussing the concept with various colleagues, I noticed we had different interpretations. I also created a poll on X that went viral (# irony ), showing that most people see organic growth as something positive. If Perplexity is correct, the term organic growth was used first in Bushido: The Soul of Japan , published in 1899. The author, Inazo Nitobe, describes Bushido (aka the way of the samurai) as something that developed through organic growth rather than being the invention of a single person or the result of a single event. In other words, Bushido emerged gradually over decades and centuries as a collective experience, not the product of one individual. It’s easy to draw parallels with the many codebases we maintain at work. The older the codebase and the more developers it has had, each with their own vision of how the code should look. Differences in perception arise from factors like developers’ varied mental models based on the parts of the code they know best. Developers come and go, making changes, introducing biases about what the future should be, and after years, the codebase reflects hundreds or even thousands of such incremental changes. This is organic growth in software: incremental delivery driven by agile methodology that shapes codebases, often in a bottom-up manner. So, is organic growth good or bad? Let’s use the garden metaphor: imagine a public garden where anyone can plant whatever they want and wherever they want. Over time, sure, it’s colorful, but there’s little harmony, with a lack of overall guidance or direction: Now imagine we control what can be planted and where, and periodically review and rearrange what’s growing. Our garden will likely look harmonious and well-maintained: In my opinion, organic growth carries a somewhat negative connotation, implying a lack of direction, with systems growing out of ad hoc solutions and quick fixes rather than intentional design. So, how can we move from a messy garden to a harmonious one? By transitioning from organic to controlled growth. Controlled growth is the deliberate and reflective process of evolving a codebase. With controlled growth, we should also embrace progress that is not always perfect but consistently guided by shared patterns and standards, fostering sustainable, manageable, and harmonious development over time. Here are five main rules for controlled growth: Be consistent . Follow agreed conventions and standards to build maintainable and predictable codebases. Many standards aren’t explicit, so strive to follow even the implicit ones. For example, if an entity is consistently named X, don’t be a cowboy and name it Y; stick to X. Plan ahead for large changes. Think strategically and collectively about big transitions such as significant new features. Avoid chaos and technical debt by breaking ambitious changes into smaller phases, communicating plans early, and leveraging documentation assets such as design docs (see Explore Further section). Hold regular retrospectives. After major development milestones, gather the team to review what went well and what could improve. Retrospectives foster a culture of learning and continuous improvement, a strategic pillar of controlled growth. Apply the Boy Scout rule. Frequent changes are inevitable and often rushed under pressure. Remember: “Always leave the campground cleaner than you found it.” When making a pull request, spend a little time refactoring small bits, cleaning legacy code, removing dead code, or renaming variables. This keeps the garden harmonious over time. Build a team where everyone shares responsibility and speaks openly. Controlled growth shouldn’t fall only on tech leads; it should be everyone’s job. When all team members feel heard and share responsibility for the health and quality of the codebase, the team grows on a stronger, more sustainable foundation. Subscribe now Tidy First? // We already discussed organic growth in this post. Focus on Product Ideas, Not Requirements Cognitive Load Bushido: The Soul of Japan Design Docs at Google ❤️ If you enjoyed this post, please hit the like button. 💬 When you say a codebase grew organically, what do you mean? Leave a comment A few months ago, I was in a meeting discussing the state of a codebase and heard myself say: This codebase has grown organically. After reflecting on my own words, I began questioning myself: I don’t even know what organic growth really means and whether it is ultimately positive or negative. Discussing the concept with various colleagues, I noticed we had different interpretations. I also created a poll on X that went viral (# irony ), showing that most people see organic growth as something positive. If Perplexity is correct, the term organic growth was used first in Bushido: The Soul of Japan , published in 1899. The author, Inazo Nitobe, describes Bushido (aka the way of the samurai) as something that developed through organic growth rather than being the invention of a single person or the result of a single event. In other words, Bushido emerged gradually over decades and centuries as a collective experience, not the product of one individual. It’s easy to draw parallels with the many codebases we maintain at work. The older the codebase and the more developers it has had, each with their own vision of how the code should look. Differences in perception arise from factors like developers’ varied mental models based on the parts of the code they know best. Developers come and go, making changes, introducing biases about what the future should be, and after years, the codebase reflects hundreds or even thousands of such incremental changes. This is organic growth in software: incremental delivery driven by agile methodology that shapes codebases, often in a bottom-up manner. So, is organic growth good or bad? Let’s use the garden metaphor: imagine a public garden where anyone can plant whatever they want and wherever they want. Over time, sure, it’s colorful, but there’s little harmony, with a lack of overall guidance or direction: Now imagine we control what can be planted and where, and periodically review and rearrange what’s growing. Our garden will likely look harmonious and well-maintained: In my opinion, organic growth carries a somewhat negative connotation, implying a lack of direction, with systems growing out of ad hoc solutions and quick fixes rather than intentional design. So, how can we move from a messy garden to a harmonious one? By transitioning from organic to controlled growth. Controlled growth is the deliberate and reflective process of evolving a codebase. With controlled growth, we should also embrace progress that is not always perfect but consistently guided by shared patterns and standards, fostering sustainable, manageable, and harmonious development over time. Here are five main rules for controlled growth: Be consistent . Follow agreed conventions and standards to build maintainable and predictable codebases. Many standards aren’t explicit, so strive to follow even the implicit ones. For example, if an entity is consistently named X, don’t be a cowboy and name it Y; stick to X. Plan ahead for large changes. Think strategically and collectively about big transitions such as significant new features. Avoid chaos and technical debt by breaking ambitious changes into smaller phases, communicating plans early, and leveraging documentation assets such as design docs (see Explore Further section). Hold regular retrospectives. After major development milestones, gather the team to review what went well and what could improve. Retrospectives foster a culture of learning and continuous improvement, a strategic pillar of controlled growth. Apply the Boy Scout rule. Frequent changes are inevitable and often rushed under pressure. Remember: “Always leave the campground cleaner than you found it.” When making a pull request, spend a little time refactoring small bits, cleaning legacy code, removing dead code, or renaming variables. This keeps the garden harmonious over time. Build a team where everyone shares responsibility and speaks openly. Controlled growth shouldn’t fall only on tech leads; it should be everyone’s job. When all team members feel heard and share responsibility for the health and quality of the codebase, the team grows on a stronger, more sustainable foundation. Tidy First? // We already discussed organic growth in this post. Focus on Product Ideas, Not Requirements Cognitive Load Bushido: The Soul of Japan Design Docs at Google

Agile

A Working Library 4 months ago

Everything

It’s common to talk about Taylorism—the practice of so-called “scientific management” that’s most known for it’s reviled use of stopwatches—as if it were a thing of the past, as if we had somehow moved beyond it. But like a lot of coercive practices, Taylorism didn’t so much retire as rebrand. As workers rebelled against oppressive bureaucracies, the postindustrial work ethic shifted from work as a moral imperative to work as self-realization in a process that Nikil Saval grimly calls “ self-Taylorization .” In essence, the timekeeper was internalized. Whereas, for Taylorism, the self-organization, ingenuity, and creativity of the workers were to be combated as the source of all dangers of rebellion and disorder, for Toyotism these things were a resource to be developed and exploited. The total and entirely repressive domination of the worker’s personality was to be replaced by the total mobilization of that personality. Toyotism—contrasted with Ford ism, which adopted Taylor’s model—involved a practice where small teams of people would manage a limited amount of work-in-progress through communication with teams up- and downstream of their work. Versions of it were subsequently adopted in “agile” software development and have become so engrained in product organizations that they are often barely remarked upon; it’s just how things are done. But as with most just-so stories, it’s worth considering how it came to be—and who benefits from the way things are. [The head of training at Volkswagen] first explains that “transferring entrepreneurial skills to the shopfloor” makes it possible “largely to eliminate the antagonisms between labor and capital…If the work teams have great independence to plan, carry out, and monitor processes, material flows, staffing, and skills…then you have a large enterprise made up of independent small entrepreneurs, and that constitutes a cultural revolution.” That is, by offering “elite” status to some workers, and building a system in which they monitored their own work in excruciating detail, Toyota could keep the administrators in their offices while remaining confident that the same surveillance, operational focus, and company-first perspective would be maintained—this time by the workers themselves. Giving some workers permission to perform as entrepreneurs just meant they worked harder for the company even as they became convinced they were working for themselves . Men in stopwatches are unnecessary when the worker’s own conscience will do the job. And of course, that “elite” status is, by definition, scarce. It depends on other workers continuing to toil in the old, Taylorist ways, performatively monitored and repressed. (Gorz points out that at the time he was writing about Toyota, the workers organized under the entrepreneurial model represented a mere 10-15% of the workforce; the rest were subcontractors, who were “increasingly Taylorized” as they moved down the ladder.) And, more to the point, it depends on a system in which fewer and fewer people are employed at all. It could hardly be more clearly stated that the workers taken in by the big companies are a small “elite,” not because they have higher levels of skill, but because they have been chosen from a mass of equally able individuals in such a way as to perpetuate the work ethic in an economic context in which work is objectively losing its “centrality”: the economy has less and less need of it. The passion for, devotion to, and identification with work would be diminishing if everyone were able to work less and less. It is economically more advantageous to concentrate the small amount of necessary work in the hands of a few, who will be imbued with the sense of being a deservedly privileged elite by virtue of the eagerness which distinguishes them from the “losers.” Technically, there really is nothing to prevent the firm from sharing out the work between a larger number of people who would work only 20 hours a week. But then those people would not have the “correct” attitude to work which consists in regarding themselves as small entrepreneurs turning their knowledge capital to good effect. So the firm “largely…eliminates the antagonisms between work and labor” for the stable core of its elite workers and shifts those antagonisms outside its field of vision, to the peripheral, insecure, or unemployed workers. Post-Fordism produces its elite by producing unemployment; the latter is the precondition for the former. The “social utility” of the elite cannot, for that reason, be assessed solely from the angle of the use-value of its production or the “service rendered to users.” Its members can no longer believe themselves useful in a general way, since they produce wealth and unemployment in the self-same act. The greater their productivity and eagerness for work, the greater also will be unemployment, poverty, inequality, social marginalization, and the rate of profit. The more they identify with work and with their company’s successes, the more they contribute to producing and reproducing the conditions of their own subjection, to intensifying the competition between firms, and hence to making the battle for productivity the more lethal, the threat to everyone’s employment—including their own—the more menacing, and the domination of capital over workers and society the more irresistible. That is, the existence of an elite workforce—whether it’s workers managing a kanban process in a Toyota factory, or workers driving agile development at a product company—is predicated on an underclass of people who either work in less sustainable conditions or else are proscribed from work at all. The former has come into some awareness in recent years, as workers at Google and elsewhere have organized not only well-paid engineers and designers but also support staff and contractors who are paid in a year what an engineer makes in a month. Those very highly-paid engineering roles simply couldn’t exist without the people toiling in the support mines or tagging text and images for AI training —often dreadful work that’s barely remunerated at all. But what Gorz is calling out here is that isn’t only bad work that the elite work depends on—it’s also the absence of work. The “disruption” that the tech industry has so long prided itself on is just another word for “unemployment.” But there’s also a gesture here towards another way: the less that elite identifies with their work and with their companies’ successes, the more they admit of their own insecurity and of their collaboration in creating it, the less menacing that threat becomes, the more space is opened up for different futures. I am not saying, however, that post-Fordist workers cannot or ought not to identify with what they do. I am saying that what they do cannot and should not be reduced solely to the immediately productive work they accomplish, irrespective of the consequences and mediate effects which it engenders in the social environment. I say, therefore, that they must identify with everything they do, that they must make their work their own and assume responsibility for it as subjects, not excluding from this the consequences it produces in the social field. I say that they ought to be the subjects of—and also the actors in—the abolition of work, the abolition of employment, the abolition of wage labor, instead of abandoning all these macroeconomic and macro-social dimensions of their productive activity to market forces and capital. They ought, therefore, to make the redistribution of work, the diminution of its intensity, the reduction of working hours, the self-management of the hours and pace of work, and the guarantee of purchasing power demands inherent in the meaning of their work. Abolition is both destruction and reconstruction; in abolishing work, you become able to create it anew. For too long, “work” has been synonymous with waged work, with the work we long for an escape from. And everything else becomes the “life” that stands in opposition to work, as if work were somehow an equal to the life it sucks dry. But what if work was all the change we make in the world, with all the people we make that change with—colleagues and comrades, neighbors and friends, kin in all the kingdoms. What if work wasn’t only what we do at work, but all the ways that work moves out into the world, and all the work we do elsewhere—whether in our homes or in our streets. What if our work is all the things we give a fuck about ? What becomes possible then? View this post on the web , subscribe to the newsletter , or reply via email .

Business Agile

Chaotic-Good Engineering Management 4 months ago

‘Labs’ teams, Skunkworks, and Special Projects: Beware

In a previous post , I talked about balancing ‘creating work’ and ‘destroying work’ such that the backlog does not become a huge mental burden on everyone to the point that it gives the impression that “the dev team is slow” or “we’re not making enough progress”. These are common themes at most of the places I’ve worked. One typical reaction to that is to pick a particular project and try to run it differently than “business as usual”. The projects chosen are often something other than just an incremental feature for the existing project. It might be an idea for an entirely new product or business unit or some grand re-imagining of the existing product. Often a founder is nostalgic for the days when they were coding entire features in a matter of days. Unencumbered by meetings, customers, existing code, internal stakeholders, compliance concerns, and everything else that comes along for the ride, they truly could crank out product in a way they’ve never seen all these expensive devs do since. To try to recapture some of that velocity that was ‘lost’ and get back to ‘the scrappy start-up days’, someone will inevitably propose cordoning off a special squad that can behave like they used to ‘back in the basement/garage’. They go by many names: Skunkworks , Tiger-Team, Startup-within-a-Startup, “Project Mayhem” and the like. I’m here to rain on the parade and tell you why this is almost always going to go poorly. If you’re an engineering manager or product manager asked to participate you should almost always push back against this type of idea. There is one exception though, which I will get to. The typical setup for a team like this:

Agile

underlap 4 months ago

Software convergence

The fact that such limits turn out to be members of the semantic domain is one of the pleasing results of denotational semantics. That kind of convergence is all very well, but it’s not what I had in mind. I was more interested in code which converges, to some kind of limit, as it is developed over time. The limit could be a specification of some kind, probably formal. But how would we measure the distance of code from the specification. How about number of tests passing? This seems to make two assumptions: Each test really does reflect part of the specification. The more distinct tests there are, the more closely the whole set of tests would reflect the specification. The second assumption, as stated, is clearly false unless the notion of “distinct tests” is firmed up. Perhaps we could define two tests to be distinct if it is possible to write a piece of code which passes one of the tests, but not the other. There’s still a gap. It’s possible to write many tests, but still not test some part of the specification. Let’s assume we can always discover untested gaps and fill them in with more tests. With this notion of a potentially growing series of tests, how would we actually go about developing convergent software? The key is deciding which tests should pass. This can be done en masse , a classic example being when there is a Compliance Test Suite (CTS) that needs to pass. In that case, the number/percentage of tests of the CTS passing is a good measure of the convergence of the code to the CTS requirements. But often, especially with an agile development process, the full set of tests is not known ahead of time. So the approach there is to spot an untested gap, write some (failing) tests to cover the gap, make those (and any previously existing) tests pass, and then look for another gap, and so on. The number of passing tests should increase monotonically, but unfortunately, there is no concept of “done”, like there is when a CTS is available. Essentially, with an agile process, there could be many possible specifications and the process of making more tests pass simply reduces the number of possible specifications remaining. I’m still mulling over the notion of software convergence. I’m interested in any ideas you may have. One of nice property of convergent software should be that releases are backward compatible. Or, I suppose, if tests are changed so that backward incompatible behaviour is introduced, that’s the time to bump the major version of the next release, and warn the users. I’m grateful to some good friends for giving me tips on LaTeX \LaTeX markup. [2] In particular, produces a “curly” epsilon: ε \varepsilon . Tom M. Apostol, “Mathematical Analysis”, 2nd ed., 1977, Addison-Wesley. ↩︎ I’m actually using KaTeX \KaTeX , but it’s very similar to LaTeX \LaTeX . ↩︎

Programming Testing Agile

Martin Fowler 5 months ago

Team OKRs in Action

OKRs have become a popular way to connect strategy with execution in large organizations. But when they are set in a top‑down cascade, they often lose their meaning. Teams receive objectives they didn’t help create, and the result is weak commitment and little real change. Paulo Caroli describes how high‑performing teams can work in another way. They define their own objectives in an organization that uses a collaborative process to align the team’s OKRs with the broader strategy. With these Team OKRs in place, they create a shared purpose and become the base for a regular cycle of planning, check‑ins, and retrospectives.

Scrum Agile Business

Martin Fowler 5 months ago

Impact Intelligence, addressing common objections

Sriram Narayan concludes his article in impact intelligence by addressing five common objections to this activity, including slowing down, lack of agility and collaboration, and the unpredictability of innovation.

Agile Business

André Arko 6 months ago

You should delete tests

We’ve had decades of thought leadership around testing, especially coming from holistic development philosophies like Agile, TDD, and BDD. After all that time and several supposedly superseding movements, the developers I talk to seem to have developed a folk wisdom around tests. That consensus seems to boil down to simple but mostly helpful axioms, like “include tests for your changes” and “write a new test when you fix a bug to prevent regressions”. Unfortunately, one of those consensus beliefs seems to be “it is blasphemy to delete a test”, and that belief is not just wrong but actively harmful. Let’s talk about why you should delete tests. To know why we should delete tests, let’s start with why we write tests in the first place. Why do we write tests? At the surface level, it’s to see if our program works the way we expect. But that doesn’t explain why we would write automated tests rather than simply run our program and observe if it works. If you’ve ever tried to work on a project with no tests, I’m sure you’ve experienced the sinking sensation of backing yourself into a corner over time. The longer the project runs, the worse it gets, and eventually every possible change includes stressfully wondering if you broke something, wondering what you missed, and frantically deploying fix after revert after fix after revert as fast as possible because each frantic fix broke something else.

Tdd Testing Agile

harrisoncramer.me 7 months ago

Embracing Incrementalism

Agile

the website of jyn 7 months ago

technical debt is different from technical risk

the phrase "technical debt" at this point is very common in programming circles. however, i think the way this phrase is commonly used is misleading and in some cases actively harmful. here is a statement of the common usage by a random commenter on my fediverse posts : tech debt is [...] debt in the literal sense that you took a shortcut to get to your product state. You've saved time by moving quick, and are now paying interest by development slowing down. contrast this to the original statement of technical debt in Ward Cunningham's paper from 1992 : Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite. [...] The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation. Ward isn't comparing "shortcuts" or "move fast and break things" to "good code"—he's comparing iterative design (often called "agile") to the waterfall method 1 : The traditional waterfall development cycle has endeavored to avoid programming catastrophy by working out a program in detail before programming begins. [...] However, using our debt analogy, we recognize this amounts to preserving the concept of payment up-front and in-full. Finally, I want to quote a follow-up statement from Ward in 2006, which is closely related to Programming As Theory Building by Peter Naur: A lot of bloggers at least have explained the debt metaphor and confused it, I think, with the idea that you could write code poorly with the intention of doing a good job later and thinking that that was the primary source of debt. I'm never in favor of writing code poorly, but I am in favor of writing code to reflect your current understanding of a problem even if that understanding is partial. It seems pretty clear at this point that Ward is describing something different from the common usage (I think "technical debt" is a good term for Ward's original idea). What then can we call the common usage? I like technical risk . Whenever you modify a program's behavior, you incur a risk that you introduce a bug. Over time, as the code is used more, the number of bugs tends to decrease as you fix them. Two studies in 2021 and 2022 (one by the Android security team, one by Usenix security) found empirically that memory vulnerabilities decay exponentially over time. So you have an inherent tension between minimizing your changes so that your code gets less buggy over time and modifying your code so that your program becomes more useful. When people talk about "technical debt", what I am calling "technical risk", they mean "modifying the code has a very high risk"—any kind of modification has a high chance of introducing bugs, not only when adding new features but also when doing refactors and bug fixes. Even the most trivial changes become painful and time-consuming, and the right tail of your time distribution increases dramatically. Furthermore, when we say "this program has a lot of tech debt", we are implicitly arguing "the risk of a refactor is lower than the risk of it eventually breaking when we make some other change ". We are gambling that the risk of a refactor (either in time or breakage) is worth the decreased risk going forward. Note that you cannot overcome technical risk simply by spending more time; in this way it is unlike technical debt. With sufficient technical risk, simply predicting how long a change will take becomes hard. Due to the risk of regressions you must spend more time testing; but because of the complexity of the program, creating tests is also time-consuming, and it is less likely that you can test exhaustively, so there is a higher risk that your tests don't catch all regressions. Eventually changing the program without regressions becomes nearly impossible, and people fork or reimplement the program from scratch (what Ward describes as "the interest is total"). The common understanding of "tech debt" is that it only applies to programs that were built hastily or without planning. "tech risk" is much more broad than that, though. It also applies to old programs which no longer have anyone that understands their theory ; new code if it's sufficiently complicated (stateful, non-local, "just a complicated algorithm", etc); and large programs that are too big for any one person to understand in full. In fact, most code has some amount of risk, simply because it isn't usually worth making readability the number 1 priority (and readability differs from programmer to programmer). Hillel Wayne recently wrote a post titled Write the most clever code you possibly can . At one point he says this: I've also talked to people who think that datatypes besides lists and hashmaps are too clever to use, that most optimizations are too clever to bother with, and even that functions and classes are too clever and code should be a linear script. This is an extreme example, but it reflects a pattern I often see: people think any code that uses complicated features is "too clever" and therefore bad. This comes up a lot for "weird" syscalls or libc functions, like / and . I think this misses the point. What makes something technical risk is the risk , the inertia when you try to modify it, the likelihood of bugs. Having a steep learning curve is not the same as being hard to modify, because once you learn it once, future changes become easier. Once your risk is high enough, and if you don't have the option of reducing complexity, people tend to work around the risk with feature flags or configuration options. These flags avoid the new behavior altogether in the default case, such that "changing the program" is decoupled from "changing the behavior". In my experience this can be good in moderation—but if every new change requires a feature flag, and you never go back and remove old flags, then you're in trouble, because the flags themselves are adding complexity and risk. Each new change has to consider not just the default case, but all possible combinations of flags in a combinatorial explosion. You see this with things like tmux, OracleDB, and vim, all of which tend to accumulate options without removing them. Consider this quote from someone who claimed to work at Oracle: Sometimes one needs to understand the values and the effects of 20 different flag to predict how the code would behave in different situations. Sometimes 100s too! I am not exaggerating. The only reason why this product is still surviving and still works is due to literally millions of tests! This is an extreme case, but in my experience it is absolutely representative of what happens to sufficiently large codebases over time. Once things are this bad you are "locked in" to the feature flag model—there's too many to remove (and your users may be depending on them!), but you cannot understand the interactions of all their combinations, so you gate new changes behind more new flags just to be sure. this post is kinda scary! it tells a story of codebases that grow more and more bogged down over time, despite people's best efforts, until they eventually die because they can't be modified. i think things are not actually so dire as they seem. firstly, you always have the option to do ongoing refactors, reducing the risk of changes. with ongoing maintenance like this, even extremely complex programs can be maintained for years or decades; i think the rust compiler is a good example of such a program. secondly, rebuilding systems is good, actually, because it lets us learn from the lessons of the past. oracledb, tmux, and vim all have younger competitors (e.g. sqlite, zellij, and helix) that are more nimble. even more than that, new systems have the opportunity to be built on a different paradigm (e.g. sqlite runs in-process instead of as a daemon) with different tradeoffs. this is the classic case of disruptive innovation . to some extent, people or teams can get "locked in" to existing systems, especially if they are highly configurable or changing to a new system would be high risk for the organization (e.g. migrating to a new database is extremely high risk for almost anyone), but this can be mitigated by open file formats (such as sqlite's database file ) and backwards compatibility for the old options (such as in neovim). Actually, if you read the post more closely, he is saying something even more interesting: iterative development is only possible because his company is using a language (smalltalk) that has privacy boundaries. ↩ "technical debt" as commonly understood is different from its origins. the original "technical debt" referred to iterative design. the common meaning is about programs that are hard to change, and i refer to it as "technical risk". all programs have technical risk to greater or lesser degree; you can decrease it but never eliminate it altogether. once risk grows sufficiently high, changes become hard enough that they have to be gated behind feature flags. the program eventually stagnates and is rewritten. Actually, if you read the post more closely, he is saying something even more interesting: iterative development is only possible because his company is using a language (smalltalk) that has privacy boundaries. ↩

Rust Programming Agile

Jefferson Heard 7 months ago

An unbroken chain of little waterfalls

The Buffalo River, nestled in the Boston Mountains of Arkansas, is one of the best floats in the country. It's so iconic that it's been protected as a National River, the first of its kind. Clear water, tinting to green-blue in the depths. White limestone rocks and pebbles studded with the fossil remnants of a shallow Paleozoic sea. Gentle rapids that cascade your boat downriver, each one a little waterfall that is so smooth your canoe never feels out of your control. A float from the put-in at Tyler Bend to the Gilbert General Store takes about 4 hours if you're looking to enjoy yourself along the way. But this isn't a post about floating down a river. It's a post about Agile, Waterfall, and the challenge of estimating time and complexity as a Product and Engineering leader. But it's also a little bit about rivers, journeys, and Flow . There are countless ways software and product teams use to estimate how long it will take them to ship a feature or complete a project, precisely because all of them are so bad. I suppose Point Poker sells those silly card decks, so it makes someone money. But Fibonacci points, T-shirt sizes, point poker, time estimates, and all the other idiosyncratic things people resort to under pressure to perform better at estimation than the last late-shipment are well... pointless. The Ecology of Great Tech No spam. Unsubscribe anytime. If you ship consistently on time, when has your point-poker (or whatever) exercise ever told you something you didn't already guess intuitively? If you ship consistently late or early, and you go over your estimations every time, when do you get anything but "reasonable excuses that we couldn't have known better" for why the estimate was so far off? There's a book, my old colleague and mentor James Coffos at Teamworks gave to me when I was trying to ship our re-engineered product stack on time. Actionable Agile Metrics for Predictability: An Introduction , by Daniel S. Vicanti. Despite its longwinded and dull title, this is probably the best, shortest book I've ever read on figuring out how to recognize speedups and slowdowns, how to estimate correctly, and how to intervene when the float down the river from concept to feature snags on the shoreline. First off, humans are tremendously bad at estimating time and complexity, and there's no evidence they can get much better when the approach is "narrative guessing," i.e. reading a description of a feature or ticket and giving an estimation based on your understanding of what's written. It's far better to estimate completion of a task based on past performance. Start with a well-written epic of fixed scope. Break it out into tickets with engineers and allow those tickets to be broken into subtasks. (I'll tell you in a minute how to do all that.) Then, at the end of each sprint measure: By "epic of fixed scope" I don't mean that the tickets are static. They can be added to and designs can be reworked, but the outcome should remain steady. Over time you're going to build a picture of what a good project looks like and what a troubled one looks like. From these measurements above you want to understand how fast on average your team moves through tickets vs. the amount of "scope creep" and "unexpected complexity" they discover per sprint. You won't believe me until you measure it for a while, but regardless of how they estimate it, your teams are going to move through tickets at roughly the same rate every month. The canoe-crushing boulders on your project are not velocity, but scope change, creep, and undiscovered complexity. There's some wiggle room on these rules, but: If this doesn't happen then the feature definition is incomplete or the devs lack clarity on how to build what's being asked. Further attempts to develop against it without refinement will result in wasted work. Schedule a retro and re-evaluate the scope and tasks before going forward again. If this sounds like Waterfall to you, understand that you cannot deliver reliably if you don't know what you're building . I'm not saying that you build a whole new product with waterfall process. I am saying that the most agile way to develop is to navigate a chain of tiny waterfalls that become clearer as they're approached. Rapids, if you will. An epic at a time. This of course puts a hard outline around how an epic can be constructed. It has to describe a precisely known modification. It covers maybe six weeks, not six months of work. It also can't be a title and a bunch of stories which themselves are title-only. It has to have written, fixed(-ish) requirements and end-goals when it's accepted as dev-ready. If you're building something bigger, it's more of an "initiative" and covers multiple well-understood modifications to achieve a larger goal in an agile way. A well written epic distinguishes between the Product function and the Engineering and Design functions. Typically, I think of product concerns as user stories, acceptance requirements, stretch goals, milestones, and releases. Organize what's being built in the most declarative (as opposed to procedural) terms possible. You want to give your engineers and designers freedom to come up with a solution that fits the experience of using the product, and you can't do that if you're telling them how to do it. I'm going to go with an example. I don't want to rehash how to do basic product research but your decently written epic is going to focus on a class of users and a problem they want to solve. This isn't the only way to write an epic, but it mirrors how entrepreneurs think. There are need-to-haves and nice-to-haves, and follow-ons that they deem natural, but aren't included in the ACs. Each bulleted "user story" or AC is written as an outcome that a class of user needs to achieve using your software. Designers and Engineers should talk with the person guiding the product definition (could be a PM, could be the CEO, could be the Founding Engineer) and clarify the requirements and fill in gaps before going off and creating designs and engineering artifacts. For example, missing in the above but probably necessary is " PF Coaches need to be able to cancel or reschedule an appointment individually or for a block of time / days." and "PF Coaches need to be notified if a conflict arises with a client from an incoming appointment on their calendar." A good designer or engineer will catch that requirement as missing and add it. Designers will take these ACs and turn them into full fledged user stories with screen or dialog designs. Engineers will say "oh yeah, we already have a recurrence dialog, so don't design a new one" and debate with the designers on how to get into the scheduler in a natural way. Then they come back with the person guiding the product definition and go over their approach. That approach should be complete in the sense that it covers the project from breaking ground through delivery. It's not just the first sprint's worth of tasks but an outline of how the project should go. Sure, more will get added along the way, but Design and Engineering should know how they're getting there. Also, If the product person takes the design to a customer call or two, the lead designer and engineer on the team should be present on that call, because they're the ones that need to get better over time at intuiting how a customer wants to use their software. Once everyone agrees that the solution is workable, it's tasked out: And so on. If you're thinking "This looks like a lot of work before engineers start building my product," then one of two things is true: I cannot tell you the number of times I've seen "seat-of-our-pants Agile" result in dead features or weeks or months of being stuck at "80% done" on a project. If the person doing the product definition is incentivized to commit to the work they're pushing towards engineering and product, and they're held accountable for late-changes, that person is going to get better at their job quickly. If the engineering and design functions are given creative control over a solution, then they're ultimately held accountable for coming up with a good solution for customers, making them more market and customer focused. When the above is practiced well, it encourages Flow in your team. Product is incentivized correctly to keep the process moving. Design and Engineering are given the ability to work to their strengths and become better at understanding your market and customers. Here are some "anti-patterns" I've seen that cause projects to drag out. The One Liner: This makes the design and engineering functions guess at what the product is. The worst part of a one-liner is that it's usually deceptively straightforward. It implies that the product person thinks that the result is obvious enough that engineering and design should just know what to build. Realistically they're going to think of about 40% of the actual ACs. The "stories" that get added will be a mishmash of engineering tasks, design artifacts, and guesses about what the user wants. The result will be that the PM goes back and forth with customers with the half-finished feature and saying "we actually need this change," to the building team. If it was a 6-week project done right, it's now a 12-week project that creates tension and bad blood between engineers, designers, product, and customers. Product Negotiates Against Itself: If the product manager is not also the engineer and the designer, then they do not know what constitutes complexity for those functions. If they don't understand this limitation, the temptation is to "offer" engineering and design "concessions" that actually fail to reduce complexity at all and in many cases increase it and make for a worse customer experience at the same time. For our above example, these kinds of ACs get added by product management before the feature ever reaches an engineer or designer: From the product manager's perspective, they've reduced the number of days and the time window that need to be considered, made it so you don't have to handle anonymous scheduling, and you've reduced the number of channels a message has to go through. From Design and Engineering's point of view, these are requirements, not concessions and well, now: Instead of cutting scope, the product manager just doubled it and created a series of follow-on tickets that will get "mopped up" after the initial release when the coaches complain that they can't do things like schedule evening appointments, and when clients demand a "lite" experience that doesn't require downloading the app. Prescriptive instead of Descriptive: Here we have no user-stories. We have a prescription by Product of what they want built. It might come in the form of a slide deck or Figma they sketched designs on. It might come in the form aberrant "user stories" that are actually written as ersatz engineering and design tasks. But however the work is presented, the product manager has done Engineering and Design's jobs for them albeit badly, setting up a chain of painful and slow negotiations where Design and Engineering suggest changes without understanding the underlying desires of the user they're building for. Now your builders are building for the product manager rather than the customer . The end-product will inevitably be incomplete and buggy, because the builders are playing Telephone with product management and customers. Focus your teams on laying out the course for an epic in the first sprint of dev work. A particular story may be unworkable or a design might need to be completely re-thought. Ideally that would've been found before the canoe was in the water, but the next best time is the present. You want to reduce the number of items that are reworked or added each day as quickly as possible, because hitting that fixed-scope is going to be what accelerates you to the finish line. For the product manager, an MVP contains a minimum complete set of user-stories that improve a user's experience over their baseline. "Cutting scope" at the product level is an illusion based on misunderstanding the product definition vs its implementation . Engineers and designers will outline their approach and may ask for concessions that reduce the implementation time, but the actual scope of work remains the same. Not every team is going to go through tickets at the same speed. Not every team is going to dial in the work to be done at the same speed or add engineering tasks at the same rate. Each major feature in your products is a different level of maturity and complexity, approaching a different problem for a different set of users. They're going to have different profiles and that's fine . The goal with your measurements is to establish a baseline per team or per product , not to measure the delta between your team and an ideal or compare them to each other. If a team starts doing suddenly worse than it has before on fixing the scope or it starts adding engineering subtasks at a late stage in the game, you have a way to say "This isn't normal, what's wrong?" and improve. For my money, the best unit of work to measure is the epic, but there are things that don't fit in epics. Drop-everything bugs come in. Vendors you integrate with change their products. Tooling and library dependencies age. Marketing updates branding. Sales is pushing into a new market and needs a new language pack. These tasks make up a sort of base-cost of operations and maintenance. You can categorize them differently and measure bug rates and so forth, but in the end what you want to know is how much of an average month or sprint or quarter is taken up O&M work vs. Developmental work. I've divided this up various ways over the years, but I've found it really doesn't matter how you divide it up. Over-measuring doesn't clear up the signal. If there is an uptick in how much time is being spent on this by a team or on a particular product, dig in and figure out why and whether there's an opportunity to bring it down to or below baseline. It could be tech debt. It could be that a vendor has become unreliable. It could be a communications or employee engagement breakdown. Once you have baselines you can make a prediction for new work. For our example above: We just scoped out giving personal fitness coaches a way to schedule with their clients, and we have 32 tasks. That means that at the end of the first sprint we should have 51 total tasks and at the we should expect about 65-70 tasks in the epic by the time it's completed and shipped. That's about 6 weeks worth of work to get it to UAT using the lower numbers, and accounting for the O&M overhead. You can use that in your estimates to others, or you can build in some wiggle room but keep in mind that projects usually take as long as you give them. You can even use statistical methods to build that wiggle room, and the more data you have, the better those estimates will be. I know that's a lot, so I want to summarize the main points and give you a Start-Stop-Continue. Predicting completion based on past performance. Organize work into epics. Epics are populated with user stories that describe a good user outcome. Then engineering and design work with the people defining the product (including customers where appropriate) to determine an approach and an initial set of development tasks that goes from breaking ground to shipped product. Once it's agreed upon, it goes into development and while the development tasks may change, the outcome should not. Once you have a baseline it's easy to provide completion estimates to people outside the team. It's also easy to figure out if a project is on or off track, to proactively communicate that, including how far off track it is. And you have the information to dig in with the right people on the team if that happens. Stop doing point-poker and all forms of "narrative guessing" in creating development estimates. Stop letting development start on an idea that's not ready. Stop writing user-stories that are actually development tasks. Stop product management from "people pleasing" and pre-emptively negotiating against itself with the development team. Continue to provide completion estimates, but better ones. The number of tickets and subtasks completed. The number of changes to the scope of a story or epic e.g. the end-goal, requirements, and user-stories. The number of new tickets and subtasks added. Changes to the end-goal or core requirements of a story or epic should be done by the time devs accept it and begin work. From there, tickets and subtasks should stop being added within a sprint or two. You're a tiny shop where everyone knows what they need to build and this is too heavy-handed a practice. This article can wait until you're bigger. Consider this up-front cost vs. the cost of wasting engineering and design days and weeks needing to redo work based on unforeseen changes. We can't use our existing calendar components because of the restricted schedule. The sign-up flow has to account for a new user sign up happening during the scheduling workflow. Redirects and navigation need to be updated in the mobile and web apps. The user's experience is made significantly worse because they have to complete an irrelevant task in the course of doing the thing they came to do. Our message provider Twilio already provides SMS, push, and email as a single notification package so now we write code into the system that allows us to ignore the user's notification preferences and only send SMS. Now the user is irritated because every other message from your app comes according to their preference but this one, new, "buggy" feature they used. The calendar team clears 15-25 tasks per week The number of implementation tasks will grow from the initial dev-ready set by an average of 60% in the first sprint, 20% in the second sprint, and then fall off. The calendar team spends 20% of its task clearance on O&M. Changes to the outcome (user stories or overall goal) made after development starts. The rate of development and design tasks added over time. The rate development tasks are cleared overall. The average number of "overhead" tasks within that.

Agile

Jason Fried 9 months ago

Doing what you think, not what you thought

Whenever I talk about working in real-time, making decisions as you go, figuring things out now rather than before, I get a question like this. "If you don't have a backlog, or deep sets of prioritized, ranked items, how do you decide what to do next. " My answer:  The same way you do when your made your list. You make decisions

Career Agile Go

Jason Scheirer 10 months ago

Passively Transparent

Nobody should have to ask what you’re working on. You should leave an obvious, loud trail. Encourage this pattern in others. One thing that is important to me as an engineer that I don’t think I’ve seen put in writing is something I have distilled into the term passively transparent . It should be obvious to the people in leadership what is motivating you to work and what you are working on (transparency), and it should not require an on-demand effort on your part to communicate this to them (passive access). You should be transparent for the reason that you are setting a baseline of what you are expecting to share with your manager: if they feel like you aren’t giving good status it gives them tacit permission to ask too many questions or questions that are inappropriately deep . Calibrate the level of communication by setting it yourself: keep your management at an arm’s length in how they use your time and attention. “I refer you to this query on GitHub, which you may access at any time” when someone asks about your deliverables should be a response you aspire for and your leadership accepts. You should be passively available for the reason that you are setting a baseline of when and how you are making yourself available to your manager: you, as my lead, should not have to wait for things to escalate to me openly broadcasting a crisis state before reaching out. You should also be able to do a pulse check without requiring my intervention. My time is precious, and I should not be forced to choose which time I give to you via a scheduled 1:1 if I can choose to give you the time that matters, when I have it to share progress. A lot of the agile software movement and its offshoots attempt to codify and formalize processes of behavior that boil down to expressing these values: If a manager or peer wants to see what you are working on, they shouldn’t have to ask for a status update: they should be able to figure it out on their own at any time on demand. This is something that has always mattered to me, doubly so with the forced async-by-default working world inflicted on us by the pandemic years and working with multi-time-zone teams. Make decisions (choose one end, these are opposite ends of a continuum): One way to promote this way of working is to practice it yourself: when working in a team, engage with your peers with the assumption that they are working in a passively transparent manner themselves. Quick, no-expectations messages on Slack can do wonders in a non-forceful way by gently framing your working style as conventional: Open a draft PR the second you begin coding, push to it regularly. Hourly, during a standard working day, if you can. Broadcast your current status on your issue tracking site: it is more work in the short run, but give you warm fuzzies (line go down on burndown) and lets you give high-value opportunities to course correct before going too far. Keep tickets at a 1-3 working day granularity and Move them along the started -> in progress -> finished pipeline quickly to telegraph to your management that you are working Independently On the right things At a reasonable pace. Set permissions by default to commenter or editor on every document and web site you start. Assume if someone is curious enough to want to know something that they are a stakeholder, whether they like it or not. Let them know. Pull them in. Share everything: use public Slack channels. Private channels and DMs are for shit talking and performance reviews, not for day-to-day work. As quickly as possible with minimal consensus but easy to change course on As late as possible with as long as reasonable to get input on “I was reading your draft PR, looks good” “Is there a Jira queue I’m not watching where you’re doing daily updates?”

Agile Career

Jimmy Miller 1 years ago

Dec 13: What Knowledge Isn't

I read four different papers today. None of them stuck with me. There were a few papers on social aspects of programming, all of the empirical bent. I was hoping to find a good paper that captured the sense of frustration I feel with Agile; a paper that could summarize the perverse labor relationship that Agile has forged between software engineers, product managers, and the software itself. But I've tried in vain. I looked through my absurd back catalog. None of them stood out to me as papers I wanted to discuss today. So I'm instead giving you yet another philosophy paper. But this paper leaves you with no excuse but to read it. Coming in under 3 pages. This philosophy paper turned the world of epistemology on its head. Introducing Edmund L. Gettier's Is Justified True Belief Knowledge . These are all examples of things I think I know. But how can I be sure? What exactly does it mean for something to be knowledge? Well, when we look at the kinds of things we intuitively count as knowledge, there seems to be fairly wide agreement on the core of the account. First, the things you know, you must believe. Now of course we might say something like "I know I'm going to fail, but I don't want to believe it." But here we aren't being literal. When we say that we know something we are at least saying we believe it. Or to make it silly programmer speech, knowledge is a subclass of belief. But not all belief rises to the level of knowledge. What is missing? "I know that the earth is flat." "No, you don't, because it isn't." Replace the word "know" about with the word "believe". Note the asymmetry. With "know", this logic feels sound, but replacing it with believe makes the tort non-sensical. What does this show? That we can't know things that aren't true. Philosophers call this the "factive" portion of knowledge. In order for something to count as knowledge, it must actually be true. But it's not enough for something to be true for you to correctly assert that you know it. "It's going to rain tomorrow" "How do you know that?" "I rolled a d20 and got a natural 20" Unless the above sentence was uttered during some TTRPG, we would in fact reject this person's claim to knowledge. But what if it turns out it did in fact rain tomorrow? Did they know it all along? No, they just got lucky. What they lacked was any good reason for that belief. Philosophers call this "justification". Justification means that a person is internally in possession of a good reason for their belief. When asked why they believe something, they could offer a good reason. But does this intuitive view work? Gettier shows us that it doesn't. You should read the paper. So I'm not even going to give a Gettier example. Instead, I'll use a different example. The pyromaniac (Skyrms 1967). A pyromaniac reaches eagerly for his box of Sure-Fire matches. He has excellent evidence of the past reliability of such matches, as well as of the present conditions — the clear air and dry matches — being as they should be, if his aim of lighting one of the matches is to be satisfied. He thus has good justification for believing, of the particular match he proceeds to pluck from the box, that it will light. This is what occurs, too: the match does light. However, what the pyromaniac did not realize is that there were impurities in this specific match, and that it would not have lit if not for the sudden (and rare) jolt of Q-radiation it receives exactly when he is striking it. His belief is therefore true and well justified. But is it knowledge? Gettier shows us in the paper multiple cases where true justified belief fails to be knowledge. So what exactly is knowledge? Well, this is a whole area of debate in the philosophical community. But out of this has spawned fascinating, interesting work. There are debates about whether the criteria for knowledge requires something internal (i.e. mental) or external (the world, our brain chemistry, etc). There are debates about whether there is one unified conception of knowledge, or if knowledge is contextual. Finally, there are debates on whether knowledge has an analysis at all or if we ought to take knowledge as fundamental and define things like belief in terms of knowledge. The example given above and the examples given in the paper may seem far-fetched, but it is my contention that programming is a constant Gettier problem generator. How many times have you believed you had found the cause of some bug, and had good reason to believe it, it turns out you were correct, but then when looking closer at the bug realize all your reasons were completely wrong! You had true justified belief, but not knowledge. Philosophy (well, analytic philosophy, which is the kind I enjoy) is a lot like programming. It goes deep on topics most people don't care about. It introduces concepts that seem overly technical and complicated. There are endless debates and disagreements with no hope of reconciliation. But what I see from philosophy that I find so hard to find in these computer science papers, is an awareness of the complexity of our topics. A willingness to engage with those who disagree. No pretension that your approach is the best approach. No need to have a related work section, because instead, you directly engage with others' works. My name is Jimmy Agile is a terrible way to make software The earth is round My dog sheds a ton of hair No prime numbers are prime ministers

Philosophy Agile Science

Jimmy Miller 1 years ago

Card Driven Development

Every company I've worked for has practiced what I call "Card Driven Development" (CDD). CDD is an umbrella term meant to encompass quite a few different development processes. The name comes from the use "cards" (digital or physical) found in the Scrum and Kanban processes as a means of organizing work, but CDD extends further than these two processes. It encompasses formal or informal processes as long as they have a few things in common. CDD is a way of approaching software development that involves a splitting of responsibilities. In CDD, there must be at least two groups, the first, often called "Engineering", is responsible for technical duties including writing code, maintaining code, and operational concerns like managing servers. The second group, often known as "Product" or informally "the business", is responsible for researching, detailing, and prioritizing what work needs to be done. Product gives work to Engineering, they prioritize what work will be done next, and they message out the status of the work to various interested parties. This split in responsibility means that the day-to-day work accomplished by Engineering and Product looks very different. Engineering, when operating well, is about shipping working software to customers, whereas Product's work is an act of delegation. This can often lead engineers to believe that Product does nothing, but this could not be further from the truth. Product's days are packed with work. Product's role involves being pulled in many directions, they must keep track of the work in progress, while continuously engaging with the various interested parties vying for their attention; attempting to get their party's particular needs worked on next. Product's work involves status updates, meetings, research, and above all else detailing and prioritizing work for Engineering. This last part is where the Card makes its appearance. The work needed to be done is too large to write down and detail in its entirety. Even if this task could be achieved, it isn't desirable because by the time it was written down, the facts on the ground may have changed. So a system of dividing work is adopted. In CDD this system is the card. Cards are a place to describe work that needs to be done and in what order it should be done. Cards break work into small chunks, typically focused on functionality that some particular party would like. Cards serve as a meeting point for Product and Engineering. Product writes cards, prioritizes cards, and gives them to Engineering as work that needs to be done. Engineering then works in the order given by product. Sometimes this means taking multiple cards at a time and making a time commitment for their completion (often called a sprint). Other times cards are pulled in one by one as time and resources permit (this process is often referred to as a Kanban-style workflow). As described here the card-making process only involves Product. In reality, there is usually a collaboration between Product and Engineering around cards. Typically cards are reviewed and discussed before they ever are handed to an Engineer to be worked on. Some of these cards may be "technical cards", cards created and owned by Engineering. With the exception of these technical cards though, cards are owned by Product. Product has the final say in the priority of cards and also serves as the "acceptor" for the fact that the work described in cards has been accomplished. The above is meant to serve as a neutral description of what is entailed by CDD. I imagine the process as I've described above is familiar to most reading this. CDD is a process that many companies adopt and do so for good reason. But despite the best of intentions by companies that adopt CDD, it is a disastrous process that leads to some of the worst issues plaguing modern software development. I contend that companies that adopt CDD will inevitably exacerbate these issues, leading to engineer dissatisfaction and decreased productivity. Creating software is a complex task. Features, that may seem on the surface trivial may involve multiple months of effort, minor modifications to an existing process may cause a cascade of required changes across many codebases, and seemingly innocent changes may introduce pernicious bugs. While there are most certainly ways to mitigate these problems, this is the reality the Software Engineer faces. The development of software does not exist in a vacuum, our new code must continue to work with old code, our legacy systems require continual maintenance and the interactions between pieces of our system continue to grow in complexity. Any view about how creating software ought to be done needs a realistic picture of the constraints that Software Engineers face. The software creation process at any given time is constrained by all the decisions of past engineers who had a hand in building the software in question. But this set of decisions involves not only all engineers who have worked at the company, but also the authors of the libraries on which the software is built. The design decisions of past engineers provide deep constraints on how much effort a given change will take. The choice of a relational or non-relational datastore limits the sort of data-integrity constraints we can easily enforce, the library we have chosen for our frontend constrains our ability to support things like server-side rendering that may be needed for SSO considerations, and the decision of how to split (or not) our application into services constrains our deployment and delivery strategy, just to name a few examples. Card Driven Development, as I have defined it, is concerned primarily with a division of labor. But with this division of labor comes a division of knowledge. Engineering knows the software systems needed for the business to operate and Product has knowledge of the various interested parties' desires. In themselves, these divisions aren't an issue, but as I hope to show the process of CDD limits the sharing of knowledge across these domains, causing issues for both the software being created and the larger goals of the business. What precisely do I mean by CDD limiting knowledge? As I'm using this term it implies at least two things. First that the knowledge that is shared between Product and Engineering happens at the wrong level and in the wrong order, limiting the usefulness of the knowledge transfer. Second, that the sorts of knowledge shared between Product and Engineering allow neither to accomplish their job as well as they ought. The reciprocal nature of this issue is incredibly important. CDD isn't just bad for Engineering, it is bad for Product and bad for the business. Imagine you are a novice cook looking to tackle a recipe a bit above your skill level. You follow the instructions step by step diligently, spending close to an hour on prepping and cooking the ingredients and as you come to the end of the recipe it says, "To finish the dish, poach an egg and place it on top". You immediately become frustrated. While the recipe is telling you everything needed to make the dish, you have no idea how to poach an egg and even if you figure it out, poaching takes time and your dish is already complete. Unlike our cooking example, CDD gives engineers knowledge at a far too granular level. Cards are about tiny pieces of functionality. They don't provide the context needed to place a given piece of functionality into its larger context. This focus on granularity can cause the big picture to be lost. It isn't uncommon to implement an entire card based on the details it provided just to find out that it didn't actually solve the problem it was intended to. Had the card been written at a higher level, the constraints on what counted as a solution to the problem would have been clearer. Not only is the information too granular, but it is also often delivered out of order. It isn't uncommon to work on 2 or 3 cards, make great progress, and then once you reach the next card you find out the work you just implemented was wrong. What you learned on card 4 was the information you really needed at the beginning. This new feature or constraint casts all previous work in a different light. The assumptions you made when implementing the others cards are now invalid. This means you have one of two choices, rework the previous work, or write a hack. I think we all know what happens most often. Cards detail functional requirements of our software. They are more often than not about the inputs and outputs of our systems. They detail UI components, data requirements, user interactions, etc... And yet these are not what Engineering needs to do its job most effectively. The businesses we create are not about these inputs and outputs. Ultimately as businesses, we are trying to solve problems. Our users face issues that we are providing a service to solve. These users may be our internal customer support, other companies, or consumers, regardless, they have a problem and we are tasked with fixing it. For these problems to be fixed software needs to be written by Engineering. How can Engineering know if their software is effective? Or in other words, how can Engineering know if their software solves the users' problem? By first understanding at a deep level what problem the user is facing. This is truly the fundamental problem with CDD, it distances engineers from the problems they are solving. No longer do engineers know or care deeply about these problems. No longer do they spend their time thinking about these problems and how best to solve them. Instead, they are consumed by details, consumed by sub-optimal answers. But this isn't just a problem for Engineering. Product has burdened themselves unnecessarily by adopting CDD. Product's job is to understand the user, but there is never just one user. Product has to understand all these various users with vastly different needs. This is a hard enough task on its own, but with CDD understanding the users is a small part. Instead, Product must spend their time trying to fit these problems into the system that Engineering has developed. They must figure out a way to address the problems the user is facing in a system that is largely opaque to them. The opaque nature of the system that Engineering has built is a constant frustration for Product. Constantly their best attempts at making small changes are thwarted. Even the most reasonable-sounding request gets push back from Engineering. It becomes harder and harder for product to make the changes they are requesting. Ultimately, in CDD Product is being asked to do something nearly impossible, propose changes to a system that is largely invisible to them. Try as they might, knowing the difference between what changes can easily be made to the system and what are hard are completely opaque to Product, and yet their job depends on knowing exactly that. Engineers who lack understanding of the users' problems may solve the wrong problems. This might mean shipping a feature that no one uses. It might mean removing some important feature. It may mean leaving out a feature that was sorely needed. All of these problems can drive customers away or prevent a company from gaining more customers. If the users of the system are internal, it may cause inefficiencies that cost the company money and can even affect employee morale. If Product has been tasked with designing new features and lacks the knowledge needed to do so effectively, inefficiencies in the development cycle can appear. As Product proposes more changes, estimates grow. More time is spent in meetings discussing changes than actual time implementing them. As development slows, demands for changes continue to grow. Users talking to Product get increasingly frustrated as to why their changes haven't made it in. More often than not, this means changes to internal tools are deprioritized leading to inefficiencies and frustrating process that can lead to high turnover in customer support. This lack of knowledge on both sides can lead to friction between Product and Engineering. It is incredibly common for companies to have a strained relationship between Engineering and Product. I think many of these issues stem from this confusion of roles. Both sides feel powerless to achieve the ends they have been tasked with. Over time their jobs get harder and harder and it always seems to be the other group's fault. A strained Product and Engineering relationship can spell the end for a project or even a whole company. More often than not, strained product relationship leads to frustrated engineers. These engineers leave, taking with them their hard-earned knowledge of the system they helped build. As engineers become frustrated, they often become defensive. When asked for estimates they worry about how the estimates might be used and so they pad them. Knowing that anyone can detect if they constantly over-estimate, they slow their work, meeting nearly exactly the estimates they put forward. Development time stretches on and on. The system the company has invested so much in grows worse and worse to make changes to. All the original engineers have left. Product is sick and tired of trying to force changes through. Eventually, the business decides to do the big rewrite and the cycle repeats. If we are ever to break this cycle, we must get rid of card-driven development. What I think should replace it is a topic for another post. But whatever the right process is, it must start with an understanding of what programming is .

Business Career Agile

Jimmy Miller 1 years ago

CDD and the Non-Linearity of Discovery

I have written previously about how Card Driven Development is extremely detrimental to the development process. In that essay I pegged the problem on CDDs lack of proper knowledge sharing and its incorrect splitting of responsibility. Today I want to focus on a different attribute of CDD that has been bothering me lately. This is the start of me trying to deliver on the promissory note I left at the end of the last essay. If we are to replace CDD we must start with an understanding of what programming is. That is our starting point today. Programming is Theory Building , does CDD help us build theories? Perhaps my last essay could be seen as a bit of a caricature. I don’t think it is, I have in fact seen this process play out in practice. But let’s assume I am wrong. Maybe CDD isn’t disasterous. Maybe I gave too much credit to a process for creating such turmoil. What then would the problem with CDD be? If you don’t see the social problems of CDD that I do why should you not use it? Theory Building is an activity of trying to take information about the world and it put it to some particular use. Sherlock Holmes works to build his theories to apprehend criminals. Someone decorating a home builds their theory so that they might make their home cozy, habitable, and enjoyable to live in. A software engineer builds their theory to enable them to solve the problems at hand, not only the problems of today but the problems that will arise tomorrow. Consider an instance of complex theory building, how does it proceed? Does Sherlock Holmes proceed from clue to clue, following a linear path to apprehension of the criminal? No, the investigation is full of false starts, of reimaginings, of unexpected twists. But what is most important are Holmes's thoughts on the matter. Holmes's theory involves learning lessons and responding to them in unusual, particular ways. The process of learning and responding is important to his investigation. If Holmes merely gathered the data, brought it to the police and let them decide his next steps, would he have been as effective? The same is true for all theoretical investigations. These investigations give us embodied knowledge. The fact that we worked out the theory is an important aspect of its creation. We had to go through this process of discover, we had to have the false starts, without them we would not have created the same theory. On the surface it might seem like CDD should allow for this non-linear process. CDD is common in agile circles where small iterations, retrospectives, and course corrections are advocated. Isn’t agile explictly a response to methodologies like waterfall that didn’t allow for non-linearity? In one sense of the term, yes, agile does not assume linearity. It accepts that you cannot know the destination from the beginning. But remember, the question is about CDD and the non-linearity of discover, not agile and planning non-linearity. CDD by its splitting of responsibility between two separate groups, product and engineering, makes using the results of discovery incredibly difficult. What is crucially missing is the allowance for discovery to take center stage when it comes to planning. As engineers work to build a theory (by programming) their discoveries must be incorporated into the plan. But what mechanism is available in CDD for doing so? Well, the primary mechanism is the card. This is the communication tool of choice, it is the pre-requisite for getting work sanctioned. But here is where the problem starts. The card cannot serve the task of feedback that it has been tasked with. It is inadequate for a few reasons. First, the time horizon for feedback to be incorporated and for course correction to occur is too long. As engineers are programming, they might find that their theory was incorrect, that the work they had planned to do is no longer the correct choice. What should they do? Well, given CDD the pressure is to forge ahead and file a card for future consideration. But often the correct answer is to stop the work you planned and instead go learn, go develop your theory better so you know the correct moves going forward. The card also fails to serve as a proper medium for conveying the information it needs to about this discovery process. First, cards are factual pieces of proposed works. They often take the form of a user story along with acceptance criteria. But this is not what is needed. Because CDD has split the responsibility for deciding what work is to be performed between product and engineering, what the engineer now has to do is convince product that what they have discovered is the truth. They need to help product understand their theory. Finally, the card fails because it assumes that all the work to be done has a deliverable. The process of discovery is however not like this. When working to build a theory much of the work we do has no direct outcome. We make guesses that turn out the be wrong. We pursue paths that are dead ends. But this work was not useless. If programming is theory building and the primary output is knowledge rather than a build artifact or source code, then the engineer learning new things is the best possible outcome. In the previous section, I have ignored some of the nuances that occur in the CDD process that might make it seem I am not being fair, let me address those. First, is it true that engineers are not given the flexibility to course correct without creating a card? Well, if the card is written the way many advocates of CDD recommend, it would seem no. CDD advocates often say that a card should be about detailing a concrete customer problem, rather than a particular solution. It is up to the team to determine how best to solve this problem. Here we can say a few things. First, it is readily admitted by most parties that achieving this goal is something must companies struggle with. So CDD as it is practiced, rather than in its ideal form still runs into this issue quite a bit. But we can go further than this. What problems make sense to tackle, what problems are meaningful depends on the theory we have. How many times have we done work where we solved a particular problem, and solved it well, only to find out our customers didn’t use it? Many times, our customers themselves told us about this problem, will even admit it solves the problem as originally stated, but it turns out they were wrong about their needs. Setting forth a problem constrains our theories, if in the process of discovery we come up with a new theory that sets us on a new course with new problems, what are we do to given CDD? Next, I have acted as if the card is the only medium of communication available. If we view any guide on how to practice a CDD process, we will find this isn’t accurate. The card is a starting point, it precedes the conversation. Planning is not about taking the card as given, it is about discussing the card and coming to an agreement as a team on the plan to move forward. And yet still this isn’t enough to handle the discovery process. First, the card frames the discussion as does the planning meeting. The goal is to come to a decision for action, not to contemplate a theory. But even if this isn’t a barrier, the card won’t be adequate, because the knowledge that has been gained by the engineer might not be expressible for them. We can know all sorts of things we can’t express. We can even teach others those things we cannot express. But the card is not the medium for doing so. Finally there are all sorts of different types of cards. What it might seem I am describing is a CDD process without the Spike card. The spike card is purpose built for the discovery process. It is a time-boxed card made for discover where the deliverable is the findings that the spike uncovered. And yet, this isn’t adequate. The discover process is not a phase, it is not a time-boxed part of the process, it just is the process. If programming is theory building, then what has been relegated to a “spike” is really the main function we are performing, with code as a side-effect. But CDD get this exactly backwards. It might seem like am saying that engineers ought to just spend all their time thinking, never committing to any plan, never taking actions toward concrete goals. I can certainly understand that reaction. CDD is meant to keep engineers grounded and to hold them accountable. We all have seen engineering teams who seemingly twiddle their thumbs all day, working and reworking things to no end, never delivering anything of value. Getting rid of CDD is not getting rid of accountability, nor is it getting rid of progress. It is about moving the decision making process closer to those who have to execute on these decisions. It is about accepting that knowledge matters when we need to make be sure we make the right decisions. It is about accepting that we need non-linear processes to discover what we ought to be working on. Our products today are technology. They are composed of bits. Why then should the “product” team not be experts in technology? How can they make the right decisions for the product if they don’t truly understand it at the core, fundamental level? At the same time, how can engineers be expected to make the right decisions if they aren’t given the data they need and allowed to build the theory regardless of what path reality pushes them towards? CDD’s push towards linearity is what causes this feeling that we can never accomplish anything in software. It is why year after year it seems that customer needs aren’t being met. Over and over again CDD forces us down a linear path to dead-ends. It prevents feedback mechanisms from operating as they should. If we are to make engineering more productive and help engineers have a better understand of the problems they truly need to solve, we must rid ourselves of CDD.

Programming Agile

Jason Fried 1 years ago

Appetites instead of estimates

The problem with software estimates is that they're both entirely right and entirely wrong. Yes there's a 3 week version of something. And a 6 week version. And a 4 month version. And a 12 month version. That's correct. Yet, you'll almost always be wrong whichever you pick. Because estimates aren't walls — they're windows. Too easy to open and climb through to the next one

Business Agile