I started my new job, a few months ago as a server operator. This has
I started my new job, a few months ago as a server operator. This has been
a really positive change in my life. The firm I work at is a VPS provider and my
job has all the things that excite me about server-side computing.
There's no more illusions being made about 'cloud computing' I can see the
virtualization (we use both KVM and containers) happening right in front of
me. And this has been something I wanted to dissect for a while, but never
had the chance to do so (at least not in production). To work with the bare
metal and to see the enormous underlying complexity is both exciting and
humbling, to say the least.
However, there's a long road ahead for me to walk and this post is meant to
be a note to my future self, so he doesn't get too carried away.
It's always about the problem, not the tools or
There are a lot of tools for configuration management like Ansible, Chef,
Puppet, etc. These have their own special place in the industry. To my own
surprise, I found myself shying away from them.
The reason was not because I don't want my life to be easy, but quite the
opposite. I started scripting my tasks in Bash and Python because I wanted to
know what's happening under the hood. My lack of experience can only be compensated by extra visibility into what's happening inside the system. It
may be tedious and time-consuming, in the beginning, but over the long run it
will lead to better understanding of the system.
Only then can I allow myself to use Ansible, (or go crazy editing config files
using sed and living in the woods somewhere :P ). To quote Ian
"Whatever it is you seek, you have to put in the time, the practice, the
effort. You must give up a lot to get it. It has to be very important to you. And
once you have attained it, it is your power. It can't be given away : it resides in you. It is literally the result of your discipline."
The above line, to me, is the resolution for the age old debates like
dependencies vs vanilla software, or libraries vs frameworks. And it helped me choose vanilla scripts over config management. If I understand the system intimately enough, I can then use a higher level abstraction easily, but if my underlying assumptions are wrong then things will go catastrophically wrong, as well. There will come a time when I can use config management, but it is not today. There's no magic tool that can substitute lack of understanding. And what I need to understand, is the problem that ails my precious servers. The more articulate I am about the issue, the easier it will be to find the solution.
Documentation over Stack-overflow/Forums
Reading documentation is healthy. I learned this by observing one of my
friends, who benefited from it tremendously, and I decided to give it a shot. A
lot of things happen when you stick to documentation:
You learn what the developer's had in mind when they wrote the
tools/library/API.You understand better what you are doing.Just the act of
reading and trying comprehend inculcates a little more patience into your
personality, and that's not nothing.
Not to pick on Stack-overflow or any other community. It just so happens that
if people ask obvious questions, which someone has already answered in the
documentation, community members are bound to get frustrated by it. This is
a good example: https://ubuntuforums.org/showthread.php?t=2328152
I have made this mistake before, I will try my best to avoid it in the future.
Besides the whole point of the community is to focus on the human aspect of
technology -- to show the path and not to hold my hand every step of the way.
Question the Authority
Always question the authority. In my short venture into the design and
implementation of mainstream virtualization stack I came across some
horrible design decisions and naming conventions. The fact that some of this
is part of "enterprise" grade software like RHEL made me second guess
For example, the library libvirt is not just a library. It is also a set of tools, a
background daemon, an API, and virtual device drivers as well. I thought I
knew what a library means but seeing libvirtd in my list of processes confused
The plain fact is that, a lot of software - most notoriously, the GNU/Linux ethos is just not designed. It tries to evolve. That's something software can't do very well, unless it is well thought out in the first place. And if I act like a fanatic and blindly accept something because it is mainstream, I will miss out on a lot of cool things.
I can be wrong in thinking something is broken, but unless I question them, I will never know. Now, this is something in which community interaction is actually valuable. With a little bit of confidence and little bit of humility maybe I will be able to learn more.
Doing what is Right, not what is Easy
Not what is easy. Not what is convenient. But what's challenging. I hope to
learn a bit more about the operations aspect of cloud computing and find
problems to solve. I won't mind write software for any unsolved problem I
encounter. Technologies like SmartOS, dTrace and containerization are
something that I am yet to conquer. It won't be easy. It is truly esoteric stuff,
people will call me out for it. Maybe, it will all actually be a bad investment of
my time. I can't know that beforehand without doing it.
The end result, or at least a part of my goal, is to create a stabler platform for
people to deploy their software on top of. A platform that is designed with security and observability in mind, not just convenience. Not just hype without substance.
Again, I can't know it without trying it out in the first place. That's my next milestone. Hopefully, looking back at this post I would have made some forward progress.