DevOps Madness

 Recently, I joined the Counter Culture Lab in Oakland CA as an intern.  It is a rather interesting maker space for bio hackery.  My mentor has assigned me to the Bio ArtBot project where I can pickup some basic lab skills.  I am working on transforming e-coli bacteria with plasmid gene which expresses color.  The competant cell we are using is call the DH5-alpha  from NEB (New England BioLabs Inc).  The product number is C2988J which costs $73 to buy as of 2022. Here is a sample transformation protocol: Thaw a tube of NEB 5-alpha Competent  E. coli  cells on ice until the last ice crystals disappear. Mix gently and carefully pipette 50 µl of cells into a transformation tube on ice. Add 1-5 µl containing 100 pg-1 µg of plasmid DNA to the cell mixture. Carefully flick the tube 4-5 times to mix cells and DNA.  Do not vortex. Place the mixture on ice for 30 minutes. Do not mix. Heat shock at exactly 42°C for exactly 30 second...

 Qiime2 is a Microbiome bioinformatics platform.  Emm, what does that mean?  Microbiome is the microorganisms in a particular environment.  Bioinformatics is the application of tools of computation and analysis to the capture and interpretation of biological data.  So, what sort of biological data Qiime2 takes in?  Is it DNA data?  If it is just DNA data, why does it has to be microbiome.  Emm. I would like to build it from scratch using dockerfile : docker build \ -t quay.io/qiime2/core: $( QIIME2_RELEASE ) \ -t quay.io/qiime2/core:latest \ --build-arg QIIME2_RELEASE= $( QIIME2_RELEASE ) docker The actual Dockerfile looks like this: FROM continuumio/miniconda3 ARG QIIME2_RELEASE ENV PATH /opt/conda/envs/qiime2-${QIIME2_RELEASE}/bin:$PATH ENV LC_ALL C.UTF-8 ENV LANG C.UTF-8 ENV MPLBACKEND agg ENV HOME /home/qiime2 ENV XDG_CONFIG_HOME /home/qiime2 RUN mkdir /home/qiime2 WORKDIR /home/qiime2 RUN conda update -q -y conda RUN...

 So, I have an CoMA3 X11 app.  It is a genetic analysis pipeline.  I can run its Docker container on MAC OS like so : docker run -it -e DISPLAY=$IP:0 -e XAUTHORITY=/.Xauthority --net=host --ipc=host -v /tmp/.X11-unix:/tmp/.X11-unix -v ~/.Xauthority:/.Xauthority -v $(pwd):/home sebh87/coma3 bash  Actually, I have to change it to this: docker run -it -e DISPLAY= host.docker.internal:0   -e XAUTHORITY=/.Xauthority --net=host --ipc=host -v /tmp/.X11-unix:/tmp/.X11-unix -v ~/.Xauthority:/.Xauthority -v $(pwd):/home sebh87/coma3 bash but before running it I have to do this  xhost + localhost . It is not too bad since I am using .Xauthority's magic cookie for authentication.  The communication between my X11 client and server is over plan text so not so good.  I would like to run this over ssh instead.  So, I edited the original Dockerfile to this: FROM sebh87/coma3:latest RUN apt update && apt install    openssh-server sudo -y RU...

 There are a number of ways to network containerized app running in separate machines.  Docker offers IPVLAND, MACVLAN, Bridges, and Overlay.  For local development or just a few machines, Bridge and VLANs seems just fine.  As the infrastructure gets more complex, then I am face with the prospect of Overlay.  In that case, I am feeling may be reaching for Kubernetes is better than going for Docker. So, how to bring an on-premise K8S into live?  I am anti-expertise so to bring up a huge chunk of infrastrure like this, I would go for reputable providers such as anthos or EKS anywhere .

Hey, a guy I respect in DevOps told me micro services that implement liveness prob and readiness prob is a good thing to do in a Kubernetes environment. That sounds obvious but if an app is in ready state, would that also means it is a live?  Why the curve ball?  Well, container state machine goes like startup -?-> liveness -?-> readiness then the pod is up and running.  It is generally consider bad to send traffic to pod that is not ready to process data.  Otherwise, it can cause data lost if proper exception handling is not in place or that reliable connection is assumed like UDP traffic.  Liveness prob is there to detect pod in bad state and restart it to gain more use out of iffy pod hosting application of questionable quality.  Emm, that sound like making the best of bad situation until someone is notified.  For a pod to transition from one state to another, there needs to be some conditions meet.  What is the condition transitioning fr...

The problem with DevOps is there are too many technology to know.  The only way to manage DevOps work is follow some principles.  Here are some principles: Build self service infrastructure Build evolvable infrastructure Be anti-expertise Build disposable infrastructure Over communicate Do no harm Pay down technical debts AWS is a totally self service infrastructure.  They have great documentation, their GUI is intuitive, and they have great support to show you how things are done.  I can pretty much compose infrastructure on my own in AWS without needing to ask AWS engineers to build stuff for me.  Granted from time to time, I need to ask them to increase quotas.  DevOps should aim to provide same level of service to internal customers.  DevOps tools are constantly evolving, GitOps has gain acceptance in the field and it is a great thing.  GitOps is a great help in moving DevOps services toward a self service model.  Just do it until a b...