There used to be a time when programmers used to be known as scientists. They would work in labs and the one commodity that these scientists had at their disposal was time. They had enough time to completely understand the problem at hand, spend days at the drawing board trying to figure out how best to solve this problem then create an application, spending days on end in its implementation.
The scientists would also know the exact usage of the application they were working on. They would know which system would be running the application, they would know what the application would be used for and they would also know the degree of the problem they would be solving. The scientists had the luxury of having perfect information about their application from specification, to development, to deployment and all the way to support. But then technology started progressing at break-neck speeds and pretty soon things changed.
The forgotten client
As soon as technology was transferred from labs to homes, human nature kicked in. The consumer wanted more. They wanted more power, they wanted more speed and they wanted it immediately. The scientists weren’t able to keep up so roles were specialized. Scientists were now known as developers, and network engineers. Developers and system architects. Database developers and web developers. Still the consumer wanted more. They wanted their applications to be more robust and more reliable. Technology needed to be created to handle these specifications so further specialization was required. Web-developers further assimilated into back-end developers and front end developers and PHP developers and Ruby developers and Angular developers.
There was complete pandemonium.
Each cog in the machine started believing that they were somehow better than the rest. That their contribution to the entire system was more essential when compared to the other. The system of creating an application became such that there was minimum interaction between these separate entities. It was the only way to ensure harmony. To keep these factions away from each other in a bid to keep the projects from self-imploding collaboration was at a minimum and discord seemed to be part of the job requirement.
In this ego-centric race to figure out who was the best and which role was the most essential, a very important facet of creating applications was forgotten: the needs of the consumer and the clients. Projects would get delayed, or they would become too expensive or they would fail altogether. This happened with such regularity that it was almost like it was part and parcel of being a developer.
Many attempts were made to overcome this rift within the world of application development. A system known as Waterfall was developed. Waterfall worked on the principle that when one group finished their task they handed over all their work to the next group, having no say in what happened to the project after it had already been handed over. The next group would add on to the work that had previously been done and so on and so forth. This process continued till you would get to the finished product.
This seemed to work for a little while, but again the clients had further demands. Now they (the clients) wanted to have more of a say in the developmental stage of the project. They would ask for changes when it was impossible to make them because that part of the process had already been passed forward to the next group. A software project failing was now so common that it was accepted as an industry standard. Statistically only half of the projects that were ever initiated would be successful.
End of the blame game
There was no way to ignore the problem anymore. It was obvious. All these different groups of developers had to work together for the overall success of the task at hand. The need for social integration and harmony created the need for the Agile Manifesto.
The Agile Manifesto was based on twelve principles:
1. Customer satisfaction by early and continuous delivery of valuable software
2. Welcome changing requirements, even in late development
3. Working software is delivered frequently (weeks rather than months)
4. Close, daily cooperation between business people and developers
5. Projects are built around motivated individuals, who should be trusted
6. Face-to-face conversation is the best form of communication (co-location)
7. Working software is the principal measure of progress
8. Sustainable development, able to maintain a constant pace
9. Continuous attention to technical excellence and good design
10. Simplicity—the art of maximizing the amount of work not done—is essential
11. Self-organizing teams
12. Regular adaptation to changing circumstance
For the first time in a long time it seemed that there was a humanistic element to the way software was developed, while at the same time ensuring client satisfaction and collaboration between developers. People actually started talking to each other! Ideas were exchanged and clients were brought in the mix to understand what the user wanted, rather than just a source of money to fund the project.
The process was still not perfect, but had improved in a manner that had once seemed unprecedented. The Agile Manifesto meant that the process of developing software would be open and flexible enough to incorporate change. On the flipside this has had the negative effect of developers losing sight of their end-goal so along the way projects were akin to lose vision of what they were originally intended for. Another manifesto needed to be implemented, and so Lean Software Development (LSD) came into being.
LSD was derived from the manufacturing process of the Toyota Production System. A system that efficiently and effectively was used to manufacture cars was implemented in the software industry.
Lean had 7 principles:
1) Eliminate Waste
2) Build Quality In
3) Create Knowledge (Amplify learning)
4) Defer Commitment (Decide as late as possible)
5) Deliver as fast as possible
6) Respect People (Empower the team)
7) Optimize the Whole
With Agile and Lean now a tried and tested component of success when developing software there needed to be just one final piece of the puzzle to make the entire system perfect (or maybe near perfect). Enter DevOps.
Initially the people that used to be involved in software development would be business analysts, system architects, front end developers, back end developers, testers and so on. The applications after being made were then sent to the “Operational Side” i.e. systems engineers, release engineers, DBAs, network engineers, security professionals, etc. Then software optimization took into account Agile and Lean methods. Removing the wall between the developmental and operational parts of the process is essentially what DevOps is all about.
DevOps are neither hired nor is it set up as a separate department. It is rather created from within the existing team. Introduce the culture, develop a mindset and promote the trend. There you have your DevOps ready and they are the same people that you have already hired as part of your existing IT team.
DevOps is a three way lane to the main highway. You start up in lane one, shift gears to lane two and then speed even more when in lane three.
Lane one – Focuses on the performance of the whole system and emphasizes the whole over the individual element of the system.
Lane two – Assures the constant feedback is given upstream and reverted downstream
Lane three – Nurture experiments, constant improvements, and fail fast
Lane one – Getting up to speed
Here you need to develop the understanding of the system as a whole and then break it down to a priority list properly. This is the beginning of the DevOps method.
The ultimate goal is to ensure uninterrupted workflow. This has a direct impact on the work quality since no one is allowed to send any defects down the stream as this will create bottlenecks.
A productive organization ensures the minimization of time delays and is on the constant look out to find ways of increasing workflow. There are a number of methodologies available to do so, such as the theory of constraints, Six Sigma, Lean or the Toyota Production System. Opt the one which you like or the one that suits your project type or use a combination of them all, it’s all up to you.
The only rule that the DevOps approach cares about is a two point principle:
Keep the system flow uninterrupted
Increase and optimize workflow at all times
Lane Two: Gear Shift
The crux of DevOps is to ensure that there is communication between Dev and Ops upstream as well as relaying information and comments back to the source. This means that ‘upstream’ communication channels needs to be implemented from Ops to Dev.
This is simply why the Waterfall method was not effective. Unless there is a system to evaluate deliverables and a method to communicate up the flow the entire system of DevOps would be redundant. It is very important that for each step downstream there is an instant confirmation upstream to ensure fluidity in communication.
There can be a number of ways to do this. Developers can be invited to join support team meetings, or the network admin can be invited to the weekly sprit planning. As long as there is constant feedback between all parties concerned you’re more or less down the path of the DevOps highway.
Lane three – Warp speed
Now the move shifts towards the fastest lane. For this move to happen you should be geared up properly to take risks, experiment continuous and keep learning from every failure. There needs to be time allotted for continuous experimentation, almost daily if possible.
To make sure that the entire system does not crumble around you ensure that a constant feedback loop is in place that ensures that bottlenecks are clear and the system flow is uninterrupted.
To see whether DevOps is enabled in your IT organization see if the following points are validated in your company:
• The development team and operations team are both part of the same process. Not separate entities
• If one team sends work forward to the next it is always verified and error free
• Deployment and maintenance are part of the same time box so the development team should not be using Operations time for its tasks
• Bottle-necking or piling should not occur at any step
• Code should not be sent for deployment at the last possible second on a weekday leaving operations having to clean up over the weekend,
• Operations should be easily be able to reproduce and scale development environments into production
• New versions can be delivered by the development team when and if appropriate and operations are still able to deploy them into production.
• All teams should have a clear and understandable means of communication
• There is ample time to experiment and work on improving any given system
• Incident handling is routine and handled in a way that is legible to all concerned. Any lessons that are learned should be documented and shared with the relevant people on a regular basis.
Although there are many tools that encourage DevOps, such as Docker, Ansible, Chef, Troposphere, Jenkins, Consular and SonarQube etc. tis does not necessarily mean that DevOps principles are being applied effectively. This is because DevOps is more a state of mind than a process that can be implemented through software. The entire team needs to understand that they are one unit and should act like one. Individual teams should collaborate and troubleshoot together. Every worker is a part of DevOps and once that is understood tools will start magically cropping up that make the entire process much cleaner and less prone to errors.