My employer (Cisco) makes IPv6 available internally on a test basis. Once I configured the isatap hostname on my system, all I had to do to get my Ubuntu laptop on IPv6 was:
sudo apt-get install isatapd
I look forward to trying this on a few more systems:
Last week I enjoyed a great story by Rands, who, as a team lead, had a total communication disconnect with one of his team members. While he enjoyed an easy rapport with Harold and Stan, he just wasn’t clicking with Wallace. At first he accepted things as they were, but soon learned that the disconnect with Wallace was a genuine problem in need of improvement. Rands concluded that the only thing to do with Wallace was to completely unwind his normal assumptions about rapport and “clicking” with a colleague and just get down to making basic communication work. This can be pain-stakingly frustrating, but this is what you need to do when you’re not getting the easy, intuitive connection you want with someone you rely on.
One of the comments (Harry) chided Rands: “Here’s the deal: if your boss asks you to lead, he either gives you the power to sack people, or you don’t accept his job offer. In your case, Wallace is obviously not compatible with you. So you sack him.”
I thought “No–Good engineers are expensive, and it is preferable to learn to steer an existing engineer in the right direction rather than finding and training a replacement.” Other comments pointed out that Wallace wasn’t incompetent or incapable, just that he needed clear expectations with management. Someone named Dave chimed in:
“I’ve been a Wallace, and I’ve also been a Harold, and from my standpoint it has less to do with personality than with context. You can have a poor team dynamic, with no clear leadership and constantly-shifting goals, where everybody ends up isolated in their corner and becoming Wallace, at least in part. Or you can have a good team, where even the most Wallace-y engineer becomes Harold for at least a few hours each day.”
Amen. We each have in us both a Wallace and a Harold. They are Yin and Yang. The Wallace side of my personality wants to get heads down in to the work, but needs to know what to work on. The Harold side takes some time to chat up his manager and coworkers to find work and set priorities, then steps aside and lets Wallace get back to work. Tech workers tend to be more innately introverted, they tend to want a good manager to play the part of Harold, and come back and set clear work objectives and priorities. When things are not going well, unhappy people will tend to revert to their base personalities. For engineers this often means getting stuck in Wallace mode.
Sometimes employees are happy and eager, and sometimes they are curmudgeons. It depends on the context of work and life morale, mediated by an employee’s emotional intelligence. These are variables that can be influenced, allowing for change over time. Management needs to provide a positive work environment with clear goals. Employees need to do our part in building a positive home environment, with positive life aspirations, while also cultivating a greater degree of self awareness. An employee who learns to steer their own craft and deliver what management wants will create a more positive work environment for their colleagues.
As a Systems Administrator, I have spent my share of time in various data centers. So, I was keen to read Ben Rockwood’s Personal Must-Haves. He wants a Leatherman, a particular mug, a particular water bottle, an iPhone and a particular bar code scanner. Honestly, it kind of felt like reading a marketing advertisement. I like the idea of a bar code scanner that can dump ASCII as if it were a keyboard device:
Your laptop will register it as a keyboard, so when you press the button to scan the contents of the barcode are “typed in” where ever you like, which means you can use it with Excel just as easily as my prefered auditing format, CSV’s created in vi.
I explained that I keep my liquids in the break area, so I don’t need any fancy mugs. And I don’t know if an iPhone supports making calls over wifi, which is important because mobile signal is often poor to none in a datacenter, nor have I any idea if the camera is all that handy for quick, low-light macro shots.
My own list would include:
In the Cage
A proper toolbox
An inventory of cables in a variety of sizes that match your color scheme.
Velcro cable ties
Label maker
A USB DVD-ROM/CD burner
Some blank DVDs and CDs
Some USB memory sticks
Spare server parts
In the Cage – Networking and Comunications
Wifi access point to a DMZ or sandbox.
A specific port on an Ethernet switch configured for the guest VLAN, and a long, loudly-colored cable reserved for connecting to it.
Dynamic DHCP (Seriously: sometimes your NetOps people don’t grok the convenience of DHCP . . .)
A terrestrial VoIP phone with a very long cable.
In the Cage – Human Sanity
Noise-cancelling headphones
Earplugs
A pocket camera, stashed in the toolbox, which can take good close pictures in low light with minimal shutter lag.
A power brick for your IT-issue laptop. (Especially if the cage is DC power!)
A sweater or jacket.
I like wearing the earplugs, then earbuds under noise-cancelling headphones, or over-the-ear earphones. You block out most of the noise and can enjoy some tunes while doing what is often non-thinking physical labor.
You never forget your first computer.
For Christmas of 1984, Grandpa gave us a
Commodore 64. A couple years later we
got a disk drive, and eventually we even
had a printer. Before the disk drive we
had to buy programs on cartridge, or
type them in to the basic interpreter
line by line. Mostly I just played
cartridge games.
Eventually we got a modem, and I could
talk to BBSes at 300 baud in 40 glorious
columns. (Most BBSes assumed
80-columns.) I was happier when I got a
1200 baud modem for my Amiga, which
could display 80 columns of text.
In my second year of college I
discovered the joy of C programming on
Unix workstations, which led to my
present career as a Unix SysAdmin. I
spend my days juggling multiple windows
of text, generally at least 80x24. /djh
I have been concerned that as Yahoo decays, that Flickr may at some point no longer remain a good place to host my photos. I do wish someone would create a competing service which supported the API. Some kid made Zooomr a few years ago, which was to sport a feature-complete Flickr API, but as best I can tell the kid moved to Japan and lost interest in Zoomr, which remains an abandoned stepchild.
Picasa? The desktop client is kind of neat but I don’t much like the web interface. It feels like another one of those one-offs Google bought but then had no idea what to do with it. Anyway, it’s just not my thing.
So, I took a look at SmugMug, who have been trying to lure Flickr refugees, but the consensus seems to be that if you like Flickr, SmugMug can not approximate Flickr. (The biggest concern for me is the loss of the “title” attribute. I’ve got 7,500 images online acquired over a decade . . .)
This is disappointing, because I like SmugMug’s promise of customization, and I have never been afraid to roll my arms up to hack on templates, HTML, and CSS to achieve my desires. Perhaps in the next few years SmugMug will become a little more flexible such that it can easily achieve what I want:
Individual pages for my photos
Support for a “title” attribute
An ability to browse title/descriptions (Flickr “detail” view)
Every so often I have this idea that the WordPress Gallery feature should take some steroids and create a friendly, Flickr-API-compatible hosting environment, which you could then customize just as much as you can customize a self-hosted WordPress blog . . . but that is very far beyond my code abilities and free time.
I’ll give Amazon.com credit for making their mp3 downloader available to Linux users:
I clicked on Ubuntu, and Chrome downloaded amazonmp3.deb and the Ubuntu Software Center fired up and told me:
Wrong architecture ‘i386’
That was frustrating. I sent a note to Amazon.com thanking them for their Linux support and asking them to please consider rolling some x86_64 packages. Then I asked Google for advice, and got this fine post:
(Your browser should hand the package off to an installer, else you’ll just do something like sudo dpkg -i Downloads/getlibs-all.deb.)
3) Run getlibs!
Looks something like this:
0-13:08 ~$ sudo getlibs /usr/bin/amazonmp3
libglademm-2.4.so.1: libglademm-2.4-1c2a
libgtkmm-2.4.so.1: libgtkmm-2.4-1c2a
libgiomm-2.4.so.1: libglibmm-2.4-1c2a
libgdkmm-2.4.so.1: libgtkmm-2.4-1c2a
libatkmm-1.6.so.1: libgtkmm-2.4-1c2a
libpangomm-1.4.so.1: libpangomm-1.4-1
libcairomm-1.0.so.1: libcairomm-1.0-1
libglibmm-2.4.so.1: libglibmm-2.4-1c2a
No match for libboost_filesystem-gcc42-1_34_1.so.1.34.1
No match for libboost_regex-gcc42-1_34_1.so.1.34.1
No match for libboost_date_time-gcc42-1_34_1.so.1.34.1
No match for libboost_signals-gcc42-1_34_1.so.1.34.1
No match for libboost_iostreams-gcc42-1_34_1.so.1.34.1
No match for libboost_thread-gcc42-mt-1_34_1.so.1.34.1
The following i386 packages will be installed:
libcairomm-1.0-1
libglademm-2.4-1c2a
libglibmm-2.4-1c2a
libgtkmm-2.4-1c2a
libpangomm-1.4-1
Continue [Y/n]?
Downloading ...
Installing libraries ...
3.1) If, like me, you got “no match for libboost” as above, or you get “amazonmp3: error while loading shared libraries: libboost_filesystem-gcc42-1_34_1.so.1.34.1: cannot open shared object file: No such file or directory” then do this bit:
So, I recently replaced my home router with an Actiontec GT724WGR. But I neglected to tell the TiVo, so it quietly started to lose programming data. Then I tried getting the TiVo to talk to the new router, but it wasn’t connecting. Long story short, it appears the TiVo is not supporting WPA2, so on the Actiontec, I went in to Wireless Setup > Advanced Settings and kicked it down to WPA and now the TiVo is updating its information successfully.
But it took some doing to figure that out. It would be helpful if the TiVo N02 error page touched upon the limitations of TiVo’s support for wireless security.
That “Sea of Glass” building from a different perspective.
While clicking around in a WordPress install last night I discovered that under Appearance > Media I can change the size of images posted through WordPress.
The G2 is fast as heck. It has all the cool new Android apps, and T-Mobile let’s you do tethering out of the box. We moved our apartment last month and setting up a wireless access point on my phone was braindead easy and plenty fast while we waited for the DSL installation. Everything works faster, and the battery life is better to boot.
The keyboards has a generally nice feel to it. But . . .
The biggest drawback is the lack of a number row on the keyboard. Really irritating to have to press ALT to type numbers. Entering “special” characters is a bitch-and-a-half. For example, to type a < you have to type ALT-ALT-long-press-j. WTF? Also, I miss the scroll wheel. There is a button on the phone that sometimes-but-not-always works as a directional pad to surf through a text field but I have learned to stab my thumb at the screen until I manage to land the cursor where I want it. (What I really miss is the Sidekick 2 direction pad.)
It is a very very nice phone with a short list of dumb shortcomings.
Automation saves time both by getting tasks done more quickly and by ensuring consistency, thus reducing support calls.
Start with a script that outputs the commands that would do the task. The SA can review the commands for correctness, edit them for special cases, and then paste them to the command line. Writing such scripts is usually easier than automating the entire process and can be a stepping stone to further automation of the process.
A simple script that assists with the common case may be more valuable than a large system that automates every possible aspect of a task. Automate the 80 percent that is easy and save the special cases for the next version. Document which cases require manual handling, and what needs to be done.
There have been times in my career when I have felt that people look at automation as a one-off task. “Write a script to automate this task.” Other times I have been asked how I go about automating things, and my answer is that automation isn’t a task so much as an iterative process:
I try to do the task at least once, maybe a few times.
Along the way I document what I had to do to get the job done.
From there, I follow the documentation, and refine edge cases as I go.
After that I’ll write a script, and get it working. (do)
I revise the documentation to explain how to use the script. (document)
And then, I use the script to complete requests, fixing the script when it fails. (refine)
Often enough I have been called upon to help another group automate something. That is a little trickier because I may never get the chance to do the task. Hopefully the other group has written some documentation, otherwise I’ll have to tease it out of them. The whole refinement process is the most obviously collaborative. I’ll document “use the script . . . it should do this . . . if it does something else, send me details.”
There is also the question of what-is-worth-automating. I believe it is the “Practice of System and Network Administration” which breaks tasks in to four buckets: frequent-easy, frequent-difficult, infrequent-easy, infrequent-difficult. You get the most payoff by focusing your automation on the frequent tasks. Easier tasks are generally easier to automate, so go ahead and start there, then turn your focus on the frequent-yet-difficult tasks. If you regard automation as an iterative process, then infrequent tasks are that much harder to automate. This is doubly true when the task is sufficiently infrequent that the systems have a chance to evolve between task execution. Infrequent tasks tend to be adequately served by well-maintained documentation in lieu of an automated process.
A last note for infrequent tasks. Part of the difficulty for these can be a combination of remembering to do them, and finding the correct documentation. One approach to “automating” an infrequent task would be to write a script that files a request to complete the task. This request should of course include a pointer to the documentation. For example, I have a cron job which sends me an e-mail to complete a monthly off-site backup for my personal web site. The e-mail contains the list of commands I need to run. (And yes, the daily local database backups are executed automatically.)
Several leading institutions fail, leading consumers to a rush on the remaining institutions, causing them to fail. A cascade of failures brings the whole system crashing down until the central authority undertakes a massive, unprecedented intervention to bring the system back to normal. At first, the degree of central intervention required is underestimated, but in time sufficient resources are brought to bear and the complex system recovers.
Interestingly, Skype’s network is actually a peer-to-peer network. It is a complex system which normally proves highly resilient, with in-built safety mechanisms to contain failure and ensure reliability. But under the right circumstances, failure can cascade. I couldn’t help but read that as a metaphor for free-market economics, which can usually take care of itself, but will enter a fugue state often enough to require a strong authority to intervene and put it right.
As a SysAdmin, the Skype network sounds like a very interesting beast. I figure that an action item against a future failure might be to provide a “central reserve bank” that monitors the health of so-called supernodes and automatically fires up large numbers of the dedicated mega-supernodes in the event of a widespread failure. (And such a strategy could well exacerbate some other unanticipated failure mode.)
Good news! Southwest Airlines offers wifi on my flight! Only $5 introductory price! I have to try this out!
The service is “designed by Yahoo!”
It is kind of really really slow to make connections.
Wait . . . WTF is this?!!
Yup. Southwest Airlines wifi does HTTP session hijacking to inject content in to your web pages.
This is a perfect illustration of the need for net neutrality: your Internet Service Provider should not interfere with your ability to surf web pages. This would be comparable to your phone company interrupting your telephone calls with commercials. Outrageous! Wrong! Bad!!
(On Mei’s computer there are actual ads in the blue bar on top, but my AdBlock plugin filters those.)
It gets worse from there. On the “designed by Yahoo!” experience you can surf on over to Yahoo! just fine. But I’m a Google man. Here’s what Google looks like:
Work-around #1: On sites that support them, use HTTPS URLs. Those are encrypted, so they can’t be hijacked. So, where http://www.google.com/ fails, https://www.google.com/ gets through!
But my little WordPress blog lacks fancy-pants HTTPS. And the session hijacking breaks my ability to post.
Work-around #2: If you have a remote shell account, a simple ssh -D 8080 will set up a SOCKS proxy, and you can tell your web browser to use SOCKS proxy localhost:8080 . . . now you are routing through an encrypted connection: no hijacking!
Update: they charge is $5/segment, so $10 if your plane stops in Las Vegas, and you get to type your credit card number a second time. Though, on the second segment, Google loads okay, but I still had to route through the proxy because the magic header was blocking WordPress’ media interface.
The Gogo Wireless on Virgin America always worked way better than this, and Google covers the cost over the holidays. And as far as I know: no session hijacking!
I saw this float across my Google Reader yesterday, thanks to Tom Limoncelli. If you are a sysadmin in an environment fixing to do more virtualization, it is well worth a skim:
NOTE: this isn’t an anti-virtualization rant, more of a “things to watch out for” briefing.
Some of my take-aways:
Sure we’ll have fewer physical servers, but the number of deployed systems will grow more quickly. As that grows so too will our systems management burden.
As the system count grows faster, we may hit capacity chokepoints on internal infrastructure like monitoring, trending, log analysis, DHCP or DNS faster than previously assumed.
Troubleshooting becomes more complex: is your slow disk access an application, OS, or hardware issue becomes also a potential host OS, networking/SAN or filer issue as well.
Regarding troubleshooting: we may add another team to the mix (to manage virtualization) while trouble-shooting has an increased probability of requiring cooperation across multiple teams to pin down. Increased importance on our ability to cooperate across teams.
Change management impacts: small changes against a larger number of systems sharing architecture snowball even more. One can add something to the base image that increases disk use by 1% for any one system, but multiply that across all your systems and you have a big new load on your filer. (1,000 butterflies flapping their wings.)
Reduced fault isolation: as we have greater ability to inadvertently magnify increased load and swamp network and storage infrastructure, we have a greater ability to impact the performance of unrelated systems which share that infrastructure.
The article also cautions against relying on vendor-provided GUIs because they don’t scale as well as a good management and automation framework.
Ah, the other thing noteworthy there is the ACM Queue magazine is now including articles on systems administration. (I subscribed to the system administration feed.)
A wiki feature I would like to see: when a node has not been edited for a certain period of time, it sends a message to the author and contributors, to review the node for relevance, and update or delete the node, if appropriate. The period of time should be configurable, and the feature should be able to CC the message in to the local request management system.
The check should be repeated on multiples of the period of time. For example, if you want to review your nodes once per year, the first year comes, the author reviews the node, sees no need to make any changes, then another year passes, you get a reminder to take another looksee . . .
This would be useful especially for an operations environment, to ensure that the shared knowledge hasn’t gone too stale, keeping the wiki resource relevant. I sincerely hope this feature already exists in a few systems!
Also useful for tracking down dependencies: I was reminded that for FreeBSD, what I would wish to call pkg_which is pkg_info -W, which will tell you what package a file came from.