Agile Architecture Requirements

This article tries to identify some additional requirements on an technical software architecture and also hints at existing requirements that possibly have to get newly weighted in the context of a project that is driven by DevOps an Agile approaches.

Definition of Technical Architecture

Before looking at the details, lets be more precise regarding the definition of “technical software architecture”.

I use the term technical software architecture to explicitly differentiate between technical and business architecture. Having some DDD background I would for example (besides other potential things) assign the actual business model based on the business requirements to the business part of the software architecture. This business part of course is important, but not the focus in this article, we concentrate on the actual technical solution.

The technical architecture is a description of concepts that make sure that functional requirements can get implemented while limiting non-functional requirements are fulfilled.

Besides instructions how to construct the actual software, the technical architecture also includes solutions for building, deploying and operating the software.

An actual technical architecture (in the context of this article) consists of multiple aspects:

• Deductions and identified constraints based on the actual requirements and resulting findings and decisions.
• The definition of the to be used infrastructure elements, libraries and frameworks together with clear implementation instructions how these elements have to be used (and how not to be used) to make sure, that the architecture is used as designed and holds to the requirements and restrictions is was initially designed for.
• Patterns and style guides with the aim to supply an abstraction layer that allows the developer to implement the functional requirements of the customer without forcing the developer to find new patterns for similar functional requirements again and again…

(there are of course many other things to keep in mind to describe an actual architecture, see Arc42)

Sources for requirements

Sources for requirements are in principle the requirements of the customer.

Besides the customer explicitly stating concrete technical requirements like “you have to use Java as a programming language” or architectural requirements like “use microservices”, functional requirements are most of the time business requirements stating more or less concrete business use cases. Non-functional requirements on the other hands define constraints that result into architectural requirements, too. Description of business requirements together with the non-functional requirements will result in further requirements for the architecture in the context of modularisation, intra/inter-process-communication, performance, scalability and other attributes of the software.

DevOps Requirements

How has DevOps influence on the architecture of a software? Let’s see. From “The DevOps Handbook” by Gene Kim, John Willis, Patrick Debols, Jez Humble:

The First Way requires that fast and smooth flow of work from Development to Operations, to delivery value to customers quickly. We optimize for this goal global instead of local paths, such as Development feature completion rates, test find/fix rations, or Ops availability measures.

[…]

By speeding up the flow through the technology value stream, we reduce the lead time required to fulfill internal and external customer requests, further increasing our work while making us more agile and able to out-experiment the competition.

[…]

Out goal is to decrease the amount of time required for changes to be deployed into production and to increase the reliability and quality of these services.

Agile Requirements

Similar from the "Agile Manifest" some quotes that should give hints regarding architecture requirements in an agile environment.

[…]

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

[…]

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

[…]

Continuous attention to technical excellence and good design enhances agility.

[…]

Simplicity–the art of maximizing the amount of work not done–is essential.

[…]

Resulting Requirements on Technical Architecture

From (especially) above quotes we see that the ultimate goal of both approaches — DevOps and Agile — is to create business value. The primary goal is not to create the best possible software architecture. A technical architecture is just a necessary evil to create (and keep) new excellent business value as fast as possible.

It also look like it is not only very important to deliver new additive business value fast but also be able to quickly change the existing software to adapt to new needs.

Minimizing the effort and maximizing the quality under the given constraints are continuous processes that act not only on the software development but on the complete software delivery process, too.

From this we have identified the following requirements and constraints for the technical architecture of a concrete project in the context of Agile and DevOps driven software development.

Not surprisingly these criteria are good criteria for a software architecture in general, too, it is just that DevOps and Agile further enforce them and even require them, claiming that else development in this context not work without them.

• The architecture has to allow the implementation of the functional requirements of the customer.
• The implementation of these requirements has to be as fast as possible.
• The architecture must allow to create new business value within one iteration (usually a scrum sprint or similar).
• The architecture must allow the within one iteration newly created and changed software
• to be released without surprises.
• The architecture must be sustainable and thus must allow to create business value
• continuously over a long period of time.
• The quality of the resulting software should be as good as possible by design to reduce the manual efforts put into quality assurance.
• Resulting software must be testable, especially using automation technics.
• The architecture must be simple to change. This is especially necessary in cases where the architecture does not allow efficient realization of business value anymore (e.g. because requirements have changed).
• The architecture has to be improvable in small iterations (“Kaizen”). It must also be possible to optimize the implementation of business value itself in iterations.
• The architecture itself must follow the iterative development paradigm itself and thus must be change- and adaptable in iterations.

Resulting Properties of the Architecture

From above requirements we deduce the following concrete properties of the necessary technical architecture.

Modular, layered and with high locality — and not monolithic

To be able to change the software itself but also the architecture in small iterations, the architecture should make use of small building blocks that are easily changeable or even exchangeable.

The architecture should consist of a small number of rather small framework components instead of one big monolithic framework. Each of this components should have as little as possible dependencies to each other, so that a change of implementation does not require adaptions of the other components.

Building blocks should be ordered and composed in some layered structure to further reduce the impact of changes.

All unlike monolithic architectures, which tend to be unchangeable because of the smaller but bigger areas of high coupling. Changes in this case often result in refactoring that is not local but covers the complete application. As such refactoring — because of its size — cannot get harmonized with the continuous flow of business value, it is often not done at all. Monolithic architecture thus creates lock in situations and tends to be not change able at all or only with big effort and result in stop the world situations.

Modular design of the Solution

To be able to apply architecture changes within one iteration, it is required that not only the architecture and frameworks has a modular structure but the actual software must have a modular structure, too. Modularity must ensure, that we can change parts of the architecture for a part of the software and still have a continuously working solution ready to deploy at any time. It must be possible to do even bigger migrations in small iterative steps.

Note that microservices are an approach to enforce this requirement.

Tight, strict Interfaces between Components

In consequence modularity and locality of the architecture and also business functionality require small, and strict interfaces between components. Contracts must be precise to be able to modify or even exchange one side of an interface.

Separation of business and infrastructure code

The goal to reach as small iteration times as possible while still generating business value suggests that the framework, infrastructure and integration code should as much as possible be separated from the actual business and domain code. It should be possible to apply changes to the technical architecture without touching business functionality and vice versa.

Interfaces between business logic and infrastructure should not expose actual technology but should reflect the necessary semantic of the interface (e.g. messaging as a pattern instead an actual messaging solution).

Note that the Onion-Architecture offers a solution for this requirement.

Size of framework

The architecture implementation should be as small as possible. “More” architecture (in the sense of “more libraries and frameworks”, “more infrastructure”, “more patterns” or “more rules”) does not automatically create business value and is not lean. The architecture has enough “size”, if it allows to comply to functional and non-functional requirements and thus generation of business value as fast as possible. A smaller framework is simpler and faster to change. If you cannot take away something without sacrificing any requirements, it is big enough.

If two frameworks supply the same quality, changeability and implementation performance, choose one — and the smaller one.

Control and master

To be able to introduce change you must be in control of the actual solution. This means, that you must be able to modify or even replace parts of your architecture. For this you must have freedom or in other words control over your architecture.

And further on: You must not only be allowed to change, modify or optimize but must also be actually be able to do so, meaning that you must be a real master of the technology you use.

Only if you are master you really understand your system, you are in control and you can understand and judge the effects of a change and thus can master the change, too.

Implementation patterns

Implementation patterns are clear descriptions together with reasoning and concrete examples that describe how a typical (business) problem should be implemented based on and aligned with then given architecture. Such pattern normally cover a number of similar problems and give the developer assistance how to implement “the correct” code.

To reduce the development time, your architecture should offer such clearly defined implementation patterns. Also functional requirements should be mapped as often as possible onto these given implementation patterns. Implementation patterns should require as little as possible code. Patterns and frameworks allow to apply optimization at central points with minimum effort.

The sum of all patterns should allow the implementation of known requirements.

The patterns should allow fast initial implementation of business code. However they should also allow to optimize and specialize with minimum effort — if required by (changing) business value.

Note that for explorative development and testing of new functionality being able to quickly offer an initial implementation (rapid prototyping) is a essential feature.

In the worst case it should always be possible to exchange a general implementation pattern with a custom solution without side effects if special requirements exist.

Implementation patterns should automatically lead to high quality and should support automatic testing as good as possible.

The number of implementation patterns should be as small as possible and should lead to an “as small a possible” architecture. Overlapping patterns or redundant implementations should be avoided as they do not create new value.

Expect existing patterns to be dropped and new to be created if requirements change over time.

Monitoring

The architecture should allow to measure and get feedback. Measure and feedback is required to make sure that the actual software does indeed fullfil it requirements (requirements match), it is required to make sure is running within its parameters (implementation match), but in a more abstract view it is also required to learn and improve the architecture itself (architecture match) and its fitness regarding it requirements and finally it is also required to measure and improve the business value (adapt requirements to business needs).

Self-serviceable

All building blocks should make use of automation where possible and where human interaction is not required. To keep developer concentrated on working on new creative and innovative concepts with the aim to maximize business value, all simple, mechanical tasks should delegated to automatism.

Handle Architecture Change

The urge to optimize (“Kaizen”) and the permanently process of adoption towards shifting needs require a continuous adjustment of the technical architecture. Note however that the change itself does not create business value in it self, but only if the changes are applied to the existing solution and do create new business value or optimize processes, actual value is created. Even worse, research and development reduce the time to create new business value if without success.

On the other hand, “The DevOps Handbook” claims, that the invest into non-user visible architecture improvements, improvements regarding non-functional requirements, process optimizations and reducing technical debt should at least be 20% of the cycle time. Improvements in complex situation may come only by doing experiments often.

Thus architecture research and development have to be strongly judged regarding actual need, the resulting risks, costs and value creation. Research and development always creates knowledge but only creates actual business value if the result is successful. Research and development should only be triggered if there is a business need or a potential for valuable optimization or a new risk to get handled.

A minimum process should be defined and possibly a WIP-limit needs to be implemented for this.

Research and development work…

• …should have a low risk of failure
• …should be iterative (start small with small costs, define goals, measure, use decision gates).
• …should either lead to a new element in the architecture to fill an existing gap or should lead to a migration from an existing element to the new element.
• …Should be limited (do not innovate faster as needed and possible do not innovate for every element at the same time to make sure that business development is not impacted)
• …should be based on common decision and should have as global use as possible
• …be lean
• …be steady
• …should document and communicate its findings to keep learnings and stimulate further experiments even in the absence of actual success.

Potential Conflicts

Above goals hold some potential conflicts between them. Not all goals can get maximized at the same time. They is judging required and some relative weighting to be done between them.

Weighting “Make” vs. “Buy”

As changeability is an important requirement, maintaining and especially changing your own framework consumes (much) time and does not create business value in itself (but only by actual application of the new functionality to create new business value).

If you thus have the option to buy and spend on top some more time (once) into making an external component exchangeable (with a newly bought component some time later) or have the option to create, maintain and continuously change your own framework, it may be wiser to give more weight to the first option.

Weighting Changeability vs. Perfection

Architecture (and especially architects) has the tendency to try to reach perfection. Reaching perfection takes time. Also reaching further perfection takes more time the more you are close to “absolute” perfection.

On the other hand, change might come every time. The more time you spend in making your architecture perfect the higher the chance that the invest you take might never realize, since a requirement enforces a change in the architecture that results in better business value but less perfection again. And even if it realizes, the invest still might not have been worth it, since change will come soon afterwards.

So it might make sense to weight changeability higher than perfection. Make your architecture good enough, but not better. Make sure that your invest will pay off. Changeability will pay off, perfection possibly not. Assume that the amount of time and money you have for managing architecture is (strictly) limited and you have to explain, too, how business value is gained by your work.

Required Forerun

Architectural changes need a higher forerun than simple functional extensions of the system. The forerun is required for conceptional work, research, actual development and for the resulting migration to the new solution. Changes higher up in the architecture stack are likely cheaper and can be applied faster, changes lower in the architecture stack (like moving from a relational database to a NoSQL database) likely need more time.

On the other hand, agile processes require quick reaction to changing requirements which will collide with the forerun required to change architecture. You cannot change your architecture every time (everyday) a requirement changes, this result in a uneven, non-aligned architecture with multiple unfinished approaches and finally chaos.

To avoid this, try to not just plan your architecture but also have a plan for changing your architecture later on. Design your architecture to be changeable and even better extendable. Be prepared not for a certain change (that may never happen) but be prepared for change in general.

Master

Everything you use you should master. Try to avoid to use libraries, frameworks, infrastructure and patterns that you do not master and likely will not master anytime soon. If you do not master your tools, you will not master your requirements.

Not that this may bring you into conflict with “buy before make”. If a “buy solution” does not fit and thus you are not able or cannot effort to master it, making you own, better fitting solution you are also able to “master” may be way out — but still a possible burden.

Obvious Architecture Solutions

Above requirements result in a number of – at first glance – obvious solutions:

• Low-Code approaches as they reduce architecture costs. Low-Code approaches however are closed solutions. The is not much room for changeability and chances are expensive. The do not allow much further optimization of the development process. This is not only true regarding improving the infrastructure or the development time but also change to improve customer satisfaction is limited.
• Code generation: Code generation allows to (dramatically) reduce the development effort by generating code that had to written by hand before. They do this by offering high level implementation patterns based on Models or DSLs. Similar to Low-Code changeability is limited (though better than for Low-Code approaches) if you are in control of the model, DSL and generator. Still make sure that you weight changeability higher than perfection and follow an pragmatic approach. Note though that code generation has its own complexity that you must be able to master. Though code is generated, make still sure that you have implementation patterns defined for the generated code and make sure that it keeps following your implementation patterns.
• Microservices solve the requirement, that the actual software solution must have a modular design to be able to apply architecture changes within one iteration. Still note though, that is is in principle also possible for a monolithic software for most architecture changes – if the actual architecture has been prepared for this.
• DDD. DDD allows you to create good models of your business requirements. The resulting models match requirements like locality, simplicity, clear interfaces and similar. DDD also defines some tactical patterns that help you to structure your code.
• Onion-Architecture helps you to separate infrastructure code from business code.