[LINK] Roles and Responsibilities of System Administrators

[grove at zeta.org.au]

On Thu, 13 Sep 2012, Marghanita da Cruz wrote:

> Hi All,
>
> As there are a few SysAdmins and many people who rely on Sys Admins,
> on this list, I wondered if it were possible to collaboratively write a book
> chapter proposal and chapter on Link.
>
> I have been invited to submit a chapter proposal by 30 sep and plan to submit
> this Link thread. The idea is the postings will be archived and publically
> available at:
> <http://mailman.anu.edu.au/pipermail/link/2012-September/thread.html>
>
> So, this is an open invite to collaborate on this chapter...collaboration is
> on process and technology as well as content.

Hi Marghanita, 
I have been a Sysadmin for <some_uni> since about 1995.   I have noticed that
the job has changed a lot, at least for me.  The most noticable things about
the change for me can be lumped into 3 or 4 categories.


Microsoftisation of Enterprise Services
The move to Lights Out, Colocated, Cloud and Hosted DataCentres.
The Lockin via Change management and Methodologies
The De-Nerdicfication of Systems Administration

All of the above take the (UNIX) Systems Administrator out of the manifold
processes we used to be responsible for.  I can address each one of these 
and explain the impact.  I do not know how relevant these
are for other organisations, but this is the path I have been through 
in recent years/months.  Use any of this how you like.

1) Microsoftisation of Enterprise Services
This sounds like a rant, but it is not.  It is however quite soul destroying 
and depressing to see this in action, up close.   Originally, what I would 
call "Layer 4" services, such as FTP, Email, HTTP, LDAP 
and other similar services were traditionally hosted on UNIX systems and 
administered by a UNIX admin.
It doesn't matter whether it is Linux or a commercial UNIX platform. 
Email services were part of a UNIX
sysadmin's role and in my case, keeping the mail flowing was about 10% of 
my daily time and it was a simple process and even when there were outages
it was not fraught with too many difficulties that could not be overcome
by more resilient storage, bigger, faster processors.

Today, in my case, email is run on a cluster of MS systems, spread across
terabytes of redundant disks and admin'd full time by two dedicated email
administrators.  Where once any administrivia was managed by the helpdesk
these admins now deal with the work as it has become too complex for the 
helpdesk to resolve.   It can be argued that the new modern portable devices
make the move to MS Exchange more practical, there seemed to be a complete
ignorance to the fact that the UNIX people could supply a similar solution
much cheaper, with most of the rough edges worn off and it would still 
only be a small percentage of our time.   But this introduction to MS 
was to be the wedge of much bigger slices into our work.  Along with Exchange,
you need Calendars so that comes into the fold, although we had a Calendar 
solution, it did not work well Blackberries in the early 00's so MS got its
foot in the door via that.  So once you have Calendars and Email, the 
trick is to subsume Directories.  So along comes a Calendar/Sharepoint 
Admin, plus 2 AD Admins.   Now you have an MS infrustructre on 12+
disparate boxes, physical and virtual and 5 dedicated admins.

We can't fight this, so a big chunk of UNIX administration is offloaded
onto these "dedicated" administrators with all the costs and factors
that go with that.  A commensurate amount of knowledge of "Sendmail" internals
and deep Directory knowledge is suddenly reduced to "trivia".

2) The move to Lights Out, Colocated, Cloud and Hosted DataCentres.
This is great. I love Lights Out.  It means I do not have to visit the 
noisy cold datacentre and can do 90% of all sysadmin functions that do not
require a physical presence without leaving my desk.  I was a big fan of this
and got on board right away.  But little did I comprehend how this would 
further reduce my actual time doing "Systems Administration".  In the past,
I was appreciated for and utilised constantly, for my ability to diagnose 
a fault, raise a solution and repair it, all by myself.  But the rationale
now is, if we do not have to attend the DataCentre, why should we?  Those 
sysadmins are wasted if they have to be in the DC pulling a CPU module 
or fitting a replacement disk.  So, we get a contract with the vendor to
do "on site support".  Suddenly, another component of my job is taken away.

Skills I studied and trained for are all of a sudden redundant and an element 
of something I actually enjoyed is out the door.  The need to attend the 
DC is reduced to the point where the managers and bean counters start
thinking in terms of "why do we even have a DC?", "Why can't we virtualise
the platforms and lease the hardware from a vendor who will do all the 
support?".   50% of a UNIX admin's job description - gone!

Once something is standardised, it can be virtualised.  Once it is virtualised,
you can do that to the whole DataCentre.  UNIX admins begone!  We don't 
need your intense knowledge of SCSI reflections and how to replace the DIMM's
in a blade chassis.  We'll let the vendor deal with that.....

3)The Lockin via Change management and Methodologies
I am aware of two methodologies in use where I work.  One is something called
"ITIL" which I think stands for "IT Is Losing" or 
"Idiots Training Idiots Legitimately" but I think it actually means "Process
comes before Productivity".  The second one I am aware of is something called
PRINCE2, or "The Methodology formerly known as PRINCE, but didn't work right".

Both these methodologies are supposed "enhance the dialogue between 
IT and the stakeholders", or "standardised the processes used to conduct 
business within the IT unit".   To me, they are unmitigated Productivity 
Killers.  Having to go through a CAB for 4 months while a File System 
endlessly corrupts itself when it could have been a 15 minute fix. 
Waiting 6 months to confirm the installation of Security Patches onto a 
customer facing vulnerable system".   Having an urgent Change Request rejected 
because the "Subject Matter Expert" was not the one raising the change. 
Conducting an "Emergency Change Meeting" instead of getting the problem 
fixed...

And so on.  Spending money meanwhile, on consultants and training in these
methodologies, while missing out on improving Technical staff through 
getting them trained on actual technical stuff and not hiring technical 
people to assist when there is a surge in a particular area.

These methodologies do nothing to assist the sysadmin in their job.  While 
I understand the requirement for change managment processes, 
so you do not set fire to your databases, they have become the job itself. 
Paralysed by the inability to perform actual maintenance, these methodologies 
often lead to systems in entropy while the methodology states 
"if it is running do not change it".   A much bigger, more panicked disaster 
later is the result.


4) The de-nerdification of Systems Administration.
I have worked in an environment where nothing was documented and everything 
was mapped in the single most important person's in the building 
(the sysadmin's) head.  This was great if he relinquished that information to 
others, but he was difficult. 
So, this is not a good place to be.  But, conversely
this person was capable of so much.  The email, directories, the web, 
the network security, the provisioning of hardware/OS/apps.   This guy 
can do it all.  Other UNIX admins want to be him, some just want to have
his job.  It is on the brains and typing skills of admins like this that 
so many IT depts have been built.  The sysadmin was respected, well paid
and if not too overworked loved what they did.   Today, I see my own role
reduced to a series of mouse-clicks, filling in stuff for a Consultant 
so they can do what I was once tasked for.  Handing over work that I used
to enjoy, that I had possibly carved a niche for myself doing.  Creating 
bespoke applications to merge/demerge print queues, or creating web farms 
or compiling GNU apps where required.  All of it requires experience
and knowledge that takes years to learn properly.  All of it now useless
as individual consultants come in and do a bit of this part of the job,
hire another to do that bit....

Apparently all that is too hard for us now, as it is moved into the realms
of Buy Not Build methodologies, that ignore the in house skills of the 
admins, reduced to just watching on the sidelines where once we would 
attend a meeting to provide input and even offer a solution, we now 
only find out at the last minute that something is implemented (in some cases
we could have simply replicate the thing ourselves) and our role is to 
give the consultant an access to do the thing, or to attend "training" on
what mouse clicks need to be done in what order.

In short, depending on what organisation or business you are in, I think the 
role of systems admin is changing.  In my case, we are being corporatised
and Change Managed so that while all the KPI's look good for the Project 
Managers, we need to constantly find a way to justify ourselves, as we 
are overlooked and considered less than godly as we once were.

Meanwhile, deskilling increases as more tasks are taken away and 
"standardised" with no reference to how things are actually done.  Outsourcing
and Buy Not Build returns the sysadmin to being dumbed down.  The 
solutions I am told are something like "Suck it up" or "Get another job".

All of this means when they go through the hire process for sysadmins 
of the future, their skills will not mean as much, their role in the 
organisation becomes greatly diminished and the dumbing down and 
denerdification will eventually mean organisations that do things this 
way will ultimately become inflexible to urgent changes in technological
directions, incapable of self-innovation in the face of requirements 
and a capacity to ignore the brilliant (sometimes flawed) ability of 
their admins to provide solutions that are elegant, thrifty and useful.

I would like to be a Sysadmin who isn't just playing clicky-clicky with 
the mouse.  Cripes, even my Sun Workstation has been replaced by a Mac 
(I am not really complaining about that...).   I would love to be 
asked to be technical lead on some project, or to come up with a solution 
to a problem that involved coding or doing something useful with my brain.
But I think it is too late.  We have a PM for that, or Consultants.
I have no control over my career anymore.  I thought I could do this 
for the rest of my working life, but I think the role of the sysadmin
is not as respected or as relied upon as it once was.  Like the old 
lawn mower that used to do a great job as it was pushed around, we 
are just old smelly dinosaurs in the garage of modern IT....


rachel

-- 
Rachel Polanskis                 Kingswood, Greater Western Sydney, Australia
grove at zeta.org.au                http://www.zeta.org.au/~grove/grove.html
    "The perversity of the Universe tends towards a maximum." - Finagle's Law