SysAdmin Checklist

26 June '07 - 04:57 by benr

I was asked to respond to this post: System Administration; an insider’s perspective. Its sort of general, but tries to capture some ideas on where a well rounded sysadmin should have some proficiency.

Lots of lists have appeared over time detailing what someone things a good sysadmin should know or be experienced with.. but honestly, I've found that it really depends on what environment your talking about. In large corporate environment with lots of admins its all about specialization. In small and medium sized companies you want a "jack of all trades" who can do punchdowns for the new PBX and implement HA databases. In startup environments its all about drive and raw talent, not so much experience.

I've been lucky in my career to have worked at very large sites (Sun, Cadence, Fujitsu, MCI Systemhouse), small and medium sites (Homestead, Clarify), and now a "startup" (Joyent). The demands of each are very different.

Of the 3 different types of sites, I can definitely tell you I don't like small and mediums. They tend to be more conservative than they should because of cash constraints and they often want to play like the big dogs but without spending big dog bucks. Startups are far more agile and many of them that I've worked with at Joyent have a refreshing "whatever it takes, just make it happen" approach.

Many admins don't like big mega-corps... but I've always liked them. You get to work with a big crew and specialization is a key to success. All those big fat perks are kinda nifty too. Best of all, its nice to have some redundancy so that you can take a vacation (or a weekend) without the pager. Besides, I have this weird fascination with tile bathrooms... can't say why, but its like my little "You've made it buddy!" benchmark. Lenolium bathrooms are a big "Run as fast as you can!" indicator.

My advice for admins is all about attitude and approach...

  • You're not better than the Janitorial crew... you clean up after other peoples messes and bang your head trying to figure out why people can't flush the damned toilet.
  • You're not better than the receptionist... you don't look, act or smell as nicely as they do, and don't fool yourself, they don't like the people they interact with either.
  • You're not better than the management... they set policy, you enact and enforce it, who's holding the leash?
  • You're not better than shipping and receiving... they just give you the stuff, you're the lucky bastard that gets to rack it.
  • You're not better than the sales guys... they make money, you spend it, on a good day you might save some.

There is a lot of ego in this job. There needs to be to survive in a job where no matter how much you know, no matter how much you've learned, you're always confronted with the one question or problem you just don't know the damned answer to and you can't let that show. And, frankly, we hold the keys to a company. If a sysadmin goes psycho we can put a business out of business... push this button and kiss your company goodbye. Its actually shocking that doesn't happen more often. Its the famous OfficeSpace "My, My Stapler...I can set fire to the building."

It's a tough gig. Long hours, huge expectations, a pat on the back if your lucky when things are good, a gun to the head if you make a mistake. It's not easy. It takes a very special type of character to do what we do, and I think anyone brave enough to do it and to stick with it is commendable.

And so the best advice I can give is...

  • Do it because you love to do it, not because you want to make money. You just won't last and you'll hate life. Thankfully most of those people washed out when the bubble burst.
  • Focus on what you enjoy, be true to your interests. Trying to be an expert in something you don't truly have a passion for is a slow walk up a steep hill.
  • Never forget that we are the interface between man and machine. We're not just about machines... we make complex computing solutions work for a business and its needs, not vice versa.
  • Your value increases as you share your knowledge, not as you horde it. Like I've always said "Admins help Admins". Sharing and collaboration are the foundation on which UNIX are founded. Whether kernel layers or people, we all have to work together for a common goal.
  • Become the greatest by being the least. The break that got me out of a call center and into a data center was offering to label servers on my own time without pay, and driving 100 miles each way to do it. Don't pass up opportunities because they are "benighted you".
  • Follow if you want to lead. I've learned the hard way that pushing out your chest and inflating that ego only give people a bubble to burst. When the blame is on you just take it, don't pass the buck. Follow your managers and leads as best as you can, don't try to out gun them. Be professional and don't allow your personal feelings of anger, envy, or frustration take away your professionalism.
  • When your in the data center remember that your dealing with machines. Don't over romanticize it. Bits go in, bits come out. This is science; deduce. Don't allow yourself to be frustrated or intimidated by a machine. A man created it to be used by man, the problem is generally that you as a user don't think like the guy that wrote it. Don't fall into the voodoo trap. At the end of the day, its a big fancy calculator on which you push in 80085 and laugh with your friends about.
  • Be passionate and let that shine through. People don't want a resume, they want a talented individual who is going to put 110% into building a better team, a better infrastructure, a better product. If you don't have passion, go change tires for a living.
  • Be willing to do the work. Moreover, be willing to do all the work that no one realizes is needed for you just to do the work. This isn't Windows where you install, uninstall, install, uninstall and then call support. You need to know the tools, the protocols, the configuration, the interactions, and everything above or below in that stack. But if you want to be more than a Google cut-and-paste sysadmin, you need to do the work, you need to use your brain, you need to seek, understand and then execute, and do so in a way that can be repeated or expanded upon in the future. And yes, its a huge amount of work, but plugging in a Fibre Channel SAN and understanding a Fibre Channel SAN are two massively different things. What kind of SA do you want to be?

When you talk with an admin during an interview or just over a Guinness you can tell a lot about them, and it has nothing to do with how many times they've configured Postfix or used Sun Cluster. It's about their character and the passions for technology.


- - C O M M E N T S - -

funny a lot

xinye (Email) - 26 June '07 - 05:44

Very succint and pragmatic article.

nura - 26 June '07 - 07:56

Excellent post. I’ve found the biggest difference between a good and excellent sysadmin is the good sysadmin will be work a few days to solve a problem – an excellent sysadmin won’t be able to sleep until they’ll solved it.

Cuyler - 26 June '07 - 08:33

Excellent Article.

Luke MacNeil (URL) - 26 June '07 - 10:50

Terrific post Ben.

j.

Jay Soffian - 26 June '07 - 16:29

> an excellent sysadmin won’t be able to sleep until
> they’ll solved it.

What? No! That’s not an excellent sysadmin. That’s a sysadmin who can’t let go of his work. The kind of sysadmin that gets ripped off by his employer for loads of hard work.

Seriously: know when to stop. Know when it’s time to pass the flame to the next guy. Know when it’s time to go home, to kiss the kids+wife and to have some rest. You’ll be more rested than the “overworked” sysadmin and the change of pace and scenery may actually allow you to come up with the solution to your problem.

Anywho, in general I agree with Ben’s list. There’s some good advice on there. The “enjoy what you do” line is definitely true. At the moment I’d rather be building houses than networks and it does in fact influence my work.

Cailin Coilleach (URL) - 27 June '07 - 00:54

Cailin: Absolutely. During a big push on a project (20 hour days for over a week straight, no time for rest) I got to a breaking point where all I could think about was quitting my job and moving to Hawaii to sell melons on the side of the road.

Burnout is a big BIG problem. And we too often think its part of the job. Thats a damned lie, a ego driven myth thats absolutely destructive.

We all have to be reminded, sleep has been proven to be one of the most powerful mental and physical regenerative tools at your disposal. While you sleep your brain consolidates memories and pushes them from short term to long term memory. Sleeping more WILL make you a better admin. You’ll re-learn things less often and remember more. Plus, its fun!

benr - 27 June '07 - 03:39

Cuyler: an excellent sysadmin makes sure the problem never occurs, preferably by raising the right flag with the right people. An excellent sysadmin doesn’t whine and will be listened to when he’s got a point to make. For me, the difference between a skilled sysadmin and an excellent sysadmin has to do with:

a) excellent social skills and a good feel for the organisational
structure and culture. Be part of it all, don’t stand aside, stay in touch.

b) being able to translate from business to functional specs to tech: speak the right language and give a solid motivation. This makes sure you’re taken seriously.

c) finding the right solution for a problem, as objectively as possible. Let go of that hammer or everything turns into nails. Be clear about your limitations and find a solution elsewhere when needed. You don’t have to do everything personally, but make sure the organisation is served correctly.

d) additionally, make sure you’re not needed! Document, delegate, make sure everyone knows what’s going on. Make sure this is thoroughly tested by taking holidays (bring the phone, though ;) )

hvm (Email) - 27 June '07 - 05:46

A fantastic article deserving many comments, though I have but one: the one single emphasized word is by far the most important bits.

Eric Enright (Email) - 28 June '07 - 00:15

Hmmm, quite a few admins got into the field because they want to avoid dealing with people. Social skills are painful for introverts.

Oh, and sysadmins need an eye for detail.
$s/horde/hoard/

Devdas Bhagat (Email) - 28 June '07 - 01:38

Eric: I’ve seen many sysadmins of the introvert type, often highly skilled on a technical level. They do need an interface to the organisation, though. I’ve seen sysadmins getting disgruntled, because they didn’t feel understood or appreciated by “them” (management), while management complains of lack of control. Most users and managers don’t know about the many ways in which technology can be used to improve their work. Who’s going to talk to them and make sure they’re served well? Many admins would be able to do so much more for their employers, if only they understood what’s going on.

hvm (Email) - 28 June '07 - 04:36

Great article. A whole bunch of high quality articles, really. I would like to make a comment on the reference to “long hours”. When there is a problem, I know that I am expected to work until it’s fixed. I know that I can (and will) be paged at any hour while on call, and that I will stay up all night on a weekend night 6-10 times a year for shceduled maintenance. I know there is effectively no comp time for this, it’s part of the job.

Therefore, when there ISN’T a problem, I think I should be able to leave after an approximately 8 hour workday without comment, not be given a lengthy, non-time critical task late in the day, and expected to finish it that day, and not be otherwise delayed when its time to go.

billy bathgates (Email) - 29 June '07 - 12:22

Amen Billy.

tamr - 01 July '07 - 11:35

Excellent list there Ben and some very cogent advise as well.
The three major things we all should strive for are (imo)—

a) Understand the mechanics of whatever we’re dealing with
b) Understand how it all fits into a larger picture (not just the puzzle within the puzzle)
c) Be lazy and automate like crazy—I’m sure we can all agree that a lot of the “mess” that we try and fix is caused by humans (either someone else or ourselves).

Cheers,

Dwai

dwai (Email) (URL) - 02 July '07 - 11:23

Great, spot on.

Chris (Email) - 10 August '07 - 00:55

hyatt regency bellevue
proform c500 treadmill

nanny_[!2] (Email) (URL) - 06 August '08 - 02:04

barings greater china fund
philly daily news paper for 12 31 05

nanny_[!2] (Email) (URL) - 06 August '08 - 08:49

tidecharts for ocean city
morton bildings

nanny_[!2] (Email) (URL) - 06 August '08 - 22:48

healthy finger nail pictures standard appliances intetin boat retail whole sales blink 182 online songs
monkeys for sale in ca how much does a tower cost by its self meret oppenheim imdb melanie wills how to do sleight-of-hand magic

nanny_[!2] (Email) (URL) - 11 October '08 - 12:20

galen medical group professional writing services for cover letters zales locations in u.s. arrogant worms erics bbq pembroke pines florida
black female american poets ufos in the bible kark and comcast liquidation of small stores kswb and morning news

nanny_[!2] (Email) (URL) - 12 October '08 - 21:49

chalk drawings email flagentssalaries commercial kitchen atlanta apparel mart how to film your own music video
ocean life drawings kawasaki mopeds the auburn news auburn ma magellanic penguin warts moles and skin tags

nanny_[!2] (Email) (URL) - 13 October '08 - 04:59

ohana lady graphic pimp letters my space albany ny newpapers sleeping quotes
comcast spotlight fresno and networks contractor tradesmen insurance coverage daiseys pool barrel union federal agent mad cuz

nanny_[!2] (Email) (URL) - 13 October '08 - 12:17

simclar new 9.0 download why do i have black little bumps on my skin mr sushi ca sally growler
the o jays live dvd stacey deffenbaugh nero express manual peigle gone away lyrics

nanny_[!2] (Email) (URL) - 14 October '08 - 01:49

Very interesting article: “SysAdmin Checklist”

Clifford (Email) (URL) - 23 January '09 - 18:05

We would like to nominate you for a FreedomToBlog.com Best of Blog Entry.

Please visit our site at: [[http://www.freedomtoblog.com]] to submit your blog entry!

Benefits include:
1) Permanent Backlinks to your blog
2) Fully SEO optimized for maximum exposure
3) A chance to be published into a top 500 best of blog book"

asep (Email) - 26 May '09 - 04:09

It was a very nice idea! Just wanna say thank you for the information you have shared. Just continue writing this kind of post. I will be your loyal reader. Thanks again.

Christian louboutin shoes (Email) (URL) - 27 October '09 - 00:36

Thank you very much!

cheap links of london (Email) (URL) - 07 November '09 - 00:49

Thanks!!!

Tiffany jewellery (Email) (URL) - 14 November '09 - 02:33

Great post! Hope to be better. Better means more features.
good post,I think so!
Thanks for your information, i have read it, very goodï¼
Bing is a really overlord!! support Bing~~
This is great news. Best of luck for the future and keep up the good work.

links of london (Email) (URL) - 17 November '09 - 03:16

Then do you know the christian louboutin,so come to see these christian louboutin shoes.

christian louboutin (Email) (URL) - 18 November '09 - 12:06

I really believe that these social networks will have a huge impact on what we can accomplish as groups, it’ll help us be very organized and communicate.

Abercrombie Fitch (Email) (URL) - 08 December '09 - 02:37

Just one question: how to add your blog into my rrs reader, thanks so much.

christian louboutin (Email) (URL) - 03 January '10 - 07:03

links of london links of london charm

links of london (Email) (URL) - 12 January '10 - 05:36

New battery review module surface is black, not because of the positive electrode exists, the module is likely to be integrated back contact silicon solar cells, silicon, where the role is often to extend the service life of equipment.Mitsubishi motors in October 2009 28-30 at the yokohama exhibition center held the Pacific “GreenDevice2009″ displayed on the high efficiency solar camera battery 19.1% respectively. In the square 15cm in polysilicon solar cells, realized the highest efficiency. The research achievements in 2009 September 2009 and the application of physics to the international society in October EUPVSEC “, “said in a statement.

battery review (Email) (URL) - 30 January '10 - 03:44

I very intersted in the article

cheap mbt shoes (Email) (URL) - 23 April '10 - 03:33

links of london links of london charm
[[http://www.trademic.com/cheap-wholesal..]]

china wholesale (Email) (URL) - 05 May '10 - 01:59

P90x .It really is not expensive if you factor in the cost

of a gym membership,P90x workout . The cost for P90X is

about three months of a paid gym membership but you get to

keep the program foreverP90x . You can try many of the

online sites, but it will be the same as buying from the

company or a Beachbody Coach. Make sure you are getting

original DVD’s. People are selling copies all over. The

problem is how long will they last, P90x workout ,and you

truly need the exercise and nutrition guide to even follow

the program. You can go to any site or you can go to

[[http://www.p90xmall.com]] and click on products. P90x dvd You can

order directly from the site,P90x dvd.

p90x (Email) (URL) - 05 May '10 - 05:53

A good idea! [[http://www.offerreplicawatches.com.The..]] sell Chanel Watches, Replica BRM Watches,offer Cartier Watches, Copy Ebel Watches, Replica Hublot Watches, Fake Tag Heuer Watches, wholesale Richard Mille Watches.

They also sell some very fashion watches, such as Montblanc Watches for sale,Chopard Watches, Dior Watches and Gucci Watches. Rolex watches are hot sell. If u are Armani Watches and Breitling Watches fan. You can still look at Omega Watches and Longines Watches, many new arrivals watches just come here.You can have a try on 58338312391273913. :[url=[[http://www.offerreplicawatches.com]www..]]].

Replica Chanel Watches,Dvd Boxes (Email) (URL) - 12 May '10 - 07:37

Louboutin Factory AddressChristian Louboutin’s line of shoes includes such versions as the Ariella Talon,Louboutin Shoes Knock OffsCatwoman, Jaws, Lady Claude, Merry

Louboutin Shoes (Email) (URL) - 12 May '10 - 14:05

Louboutin Factory AddressChristian Louboutin’s line of shoes includes such versions as the Ariella[[[[h..].]]]]..]]].

Louboutin Shoes (Email) (URL) - 12 May '10 - 14:07

If you are looking for the best brand new laptop batteries at the most preferred price, you have come to the right place. [[http://www.adapterlist.com/toshiba/sat..]] We provide the highest quality Battery at the lowest price with the highest level of service, all in a secure and convenient platform.

laptop battery (Email) (URL) - 14 May '10 - 08:46

thank you for this outstanding article.I thought certino was the best technologh for laptop battery performance.We specialize in [[http://www.globallaptopbattery.co.uk/d..]] ,laptop AC adapters. All our products are brand new, with the excellent service from our laptop battery of customer service team.

laptop batteries (Email) (URL) - 14 May '10 - 08:48

[[http://www.vibramfivefinger.us/]] vibram five fingers

vibram five fingers (Email) (URL) - 10 June '10 - 09:11

Here elaborates the matter not only extensively but also detailly .I support the write’s unique point.It is useful and benefit to your daily life.You can go those sits to know more relate exceptfor.net things.They are strongly recommended by friends.Personally I feel quite well.

moncler outlet store (Email) (URL) - 13 June '10 - 05:24

[[http://www.sxhgz.com]] : nike shoes

puma shoes (Email) (URL) - 21 June '10 - 01:56

[[http://www.mbtshoeslatest.com]]
[[http://www.nikeairmaxshoe.com]]

Nike air max shoe (Email) (URL) - 24 June '10 - 07:50

I totally love this article.

ed hardy (Email) (URL) - 06 July '10 - 09:08

Do you like the ugg ? Ugg Classic Short Boots are UGG Australia’s original heritage styles. Ugg Classic Tall will keep your feet dry and the ultimate comfortable.
[[http://www.ugg-boots-london.com/ugg-cl..]]

Ugg Classic Short (Email) (URL) - 07 July '10 - 01:25

I will bookmark your blog and have my kids check up here frequently Air Jordan 2010. I’m very certain they will understand lots of new stuff Air jordan 1 here than anybody else. I do appreciate that youve added relevant and intelligent commentary here though. Thank you!

Cheap Jordan Shoes (Email) (URL) - 03 August '10 - 04:02

good day:) it’s my first time here but i really enjoy what you posts here in your site Michael Jordan Shoes.i will always visit your site for more updates. Lots of Good information in your posting Cheap Jordan Shoes, I bookmarked your blog post so I can visit again in the near future, Cheers :)

Air jordan 1 (Email) (URL) - 03 August '10 - 04:03

very cool article ,thanks for sharing the article!like my cool stuff .very useful.
uCoolStuff is the leading China wholesaler for [[http://www.ucoolstuff.com]] cool stuff, [[http://www.ucoolstuff.com]] cool gifts , unusual gadgets and other unique gift ideas. We provide the very latest cool stuff and cool gifts for you

cool stuff (Email) (URL) - 25 August '10 - 07:05

excellent article , I added you to my [[http://www.china-wholesale-directory.c..]] Top China Wholesalers category.. thanks for sharing the article!

China Wholesale Directory (Email) (URL) - 25 August '10 - 07:07

Personal information





Remember your information?
Comment

Small print: All html tags except <b> and <i> will be removed from your comment. You can make links by just typing the url or mail-address.


^M