DevOps Madness

The problem with DevOps is there are too many technology to know.  The only way to manage DevOps work is follow some principles.  Here are some principles:

Build self service infrastructure

Build evolvable infrastructure

Be anti-expertise

Build disposable infrastructure

Over communicate

Do no harm

Pay down technical debts

AWS is a totally self service infrastructure.  They have great documentation, their GUI is intuitive, and they have great support to show you how things are done.  I can pretty much compose infrastructure on my own in AWS without needing to ask AWS engineers to build stuff for me.  Granted from time to time, I need to ask them to increase quotas.  DevOps should aim to provide same level of service to internal customers.  DevOps tools are constantly evolving, GitOps has gain acceptance in the field and it is a great thing.  GitOps is a great help in moving DevOps services toward a self service model.  Just do it until a better approach comes along.

Legacy infrastructure is often not evolvable because deep rooted interdependency.   This sucks because the more you add to an un-evolvable infrastructure, the more maintenance work gets added to daly DevOps tasks.  To switch to a better self service approach becomes risky and difficult.  Service discovery should help applications and other infrastructure components to move to different cluster, different network, different VPC, different accounts etc.  If resources are able to move freely yet still stay connected, then one can build a cluster new tech and resources just migrate to it without pain.  How cool would that be.

Needing expertise means your infrastructure depends on the expert.  This is clearly bad but company do go for this dependency if the price is right.  That is a business decision but costs DevOps pain and it goes against the two principle of self service and evolvable infrastructure.  So, I am anti-expertise.

If infrastructure feels touchy, it is probably not built with the right mind set or you just don't have time to build it with thick skin.  In a cloud environment, infrastructure should be disposable objects.  You should be able to throw away any number of micro-services or an entire cluster and the thing should come backup on its own.  If this level of reliance is achieved, people's mind would be at ease.  Would that be nice?  This can probably be achieved with GitOps tools and some Python scripts.

When DevOps makes changes to infrastructure, sometime it is not seamless.  That is the reality of DevOps.  Often there are ways to made changes seamless such as build redundant traffic routes, blue/green deployment, etc.  Even if changes are expected to be seamless, DevOps still need to over communicate.  The scope, complexity, and risk of the change influences how many days ahead you need to announce the change.  A lot of human coordination might get involve as change in core infrastructure sometime requires application to also change.  So, plan on over communication.

When making changes to existing infrastructure, the goal is to do no harm.  This involves evaluating the risk associated with the change.  Ideally no downtime, no data lost, no compromise on the level of service to customer.

Last but not less, pay down technical debts.  Often, I see DevOps team structured to deliver cool stuff like new apps, new GitOps tools, new what have you.  It is kind of dirty work to clean up old infrastructures, move them, evolve them, etc.  Dirty work pays allot.  You usually only have to do them once and it pays off handsomely more often than not.  Imagine you remove improve on a piece of infrastructure that is giving you alerts on a regular bases.  The pease of mind already is a big payoff.  If it is a removal instead of an improvement, it saves cost to the business.  That is good investment you can depend on.  So, why not structure the team to do dirty work?

Voila, DevOps principles explained.