Link:
https://dannyman.toldme.com/2007/07/25/365-main-oops-wtf/
Wow! It turns out that Tuesday was a great day to not be working as a SysAdmin in San Francisco:
http://laughingsquid.com/massive-power-outages-hit-san-franciscos-soma-district/
(I like what Yelp have done with their down page.)
The short story is that an underground transformer exploded downtown, and the 365 Main data center failed to automatically start their generators, and had to start them manually, cutting power for nearly an hour for some customers, many of which are smaller, trendier web sites like Craigslist, LiveJournal, Yelp and others. (I have interviewed with half of the companies mentioned in Scott’s post.)
You do not want to lose power across a production-class network. This can cause equipment failure, servers to delay boot because they need to run disk consistency checks, servers to stall boot noting a missing keyboard, disk errors, or whatever. Some services may wedge up because when they started they couldn’t talk to the database . . . in some cases you may have had machines running for a few years, which may have last rebooted three SysAdmins ago. The running state may be subtly different from the boot state, with no documentation . . .
A few years ago I had a chance to rebuild a production network from the ground up, with a decent budget to do everything the right way: redundant network switches, serial consoles, remote power management . . . I remember talking to my manager as to whether we might want a UPS in each rack. We figured that the data center is supposed to keep the power running, or else. Also, if the data center loses power then we lose our network access anyway . . . perhaps the whole point of this post is that data centers do lose power, so a UPS can be worthwhile. If nothing else, it may leave your systems up and ready to go as soon as the network is restored.
Data centers have UPSes too. Huge ones that you may get to walk through on a tour. The purpose of the UPS is to provide battery power between the time utility power fails and on-site generators begin to provide energy. I don’t know enough to comment on this particular case, but I do recall touring a data center in Emeryville, and the guy explained that batteries become less effective over time, and a lot of data centers fail to test their batteries regularly. When wired in series, one bad battery brings down the entire UPS, and so even though you have a generator on-site, the UPS can fail before you manage to transfer to generator power. While this stuff is beyond my expertise, I’m inclined to believe that this is what happened at 365 Main yesterday: a data center should not only test its failover-to-generator procedure on a regular basis, they need to ensure sufficient battery capacity to keep systems running during the time it would reasonably take to switch to generator power.
Update–July 27: Earth2Tech points out that 365 Main uses newer, ecologically-conscientious flywheel technology to maintain current between the switch from utility to generator power, and speculate that the flywheel may have played a part in the power failure. Their writeup is very good, overall.
3 Comments
Link:
https://dannyman.toldme.com/2007/07/24/wordcamp-notes-day1/
On the weekend of July 22 and 23, I and about 400 other folks attended WordCamp 2007 in San Francisco. This is a conference about WordPress blogging software, and blogging itself. I am usually a bit wary of killing my weekend by spending the bulk of it with a bunch of nerds. Especially bloggers. But then, I am a nerd, and this is, I admit, a blog . . . that and registration was merely $25 and covered my food for the weekend. That’s a pretty compelling deal for the unemployed! Added value was found at the open bar on Saturday night at one of my favorite bars: Lucky 13.
Here are notes I compiled during the Saturday presentations. (more…)
2 Comments
Link:
https://dannyman.toldme.com/2007/07/19/yay-gift-economy/
I have worked as a waiter and I am regularly featured these days in the role of patron. My ex-wife is from Japan, and in Japan there is no tipping. She liked the simplicity and fairness of this model, and I can respect that. But I like — no, I love tipping. Why? It is that subversive little corner of our capitalist system that runs as a “gift economy.”
The act of giving, and the act of depending on the generosity of another person-these are both important activities required to build healthy people. In our society we have successfully taken the “guesswork” out of a lot of the giving-receiving relationships: you work for a specific wage, you pay a specific rent, a specific tax rate, you pay for food at a certain price determined by supply and demand, matters of government, and personal style. You subject yourself to the rule of law which is in turn mediated by your participation in Democracy.
We don’t live with the uncertainty of Kings, we don’t farm with the uncertainty of the weather. For those of us in the comfortable end of the middle class, the stressful uncertainties that mean more to people of lesser means mean a lot less to us: the price of milk, the price of gas, whether we are at war in Iraq . . .
It doesn’t take a lot of faith in the goodness of human nature to successfully live a life like mine. I’m insulated from a lot of the vagaries of the human condition. I’m not alone in this. And some of us, well, we forget about all that hassle: we are free to press our energies in to work, family, community, creative pursuits. I like this freedom, but . . . sometimes I open my eyes and see that the things that are stressing me out and challenging me are pointless, petty, or mundane. Especially in technology, victory becomes software shipped and larger numbers in the bank and retirement accounts.
We never go to bed hungry.
We will not wear body armor, carry M-16s and ride in Humvees through the garbage-filled streets of Baghdad, scared that we may not make it home from the cradle of civilization quite right.
We will eat until we are content, push the plate away half full. We will leave our lights on and run our dishwashers and our washing machines and stare into computer screens, trying to increase the zeros in those bank accounts. We will be constantly on the go, from climate controlled office to comfortable car to ski vacation to flights across the planet where we may dine on new foods, until we are content, and push the plate away again.
There have been a few times when I have been at the supermarket with not enough money. That kind of stress is memorable.
That kind of stress will be a part of anyone’s life. Your parents and grandparents will age and become infirm. Their hospital bills will eat up your savings accounts. You may in time return the favor of those diapers that were changed before you were too young to remember. You will be walking to your home one day and you will fall down and the doctor will explain that you have MS. Your wife will come home one day and tell you she needs to take a break, to go live with her new boyfriend.
The things that you take for granted today could change in an instant.
When I go out to eat, when I take a cab home, I have to make room in my capitalist business transaction for the tip. There are “rules” but the only enforcement is in your own character. When you give service, if you are honest with yourself, you accept that you are in an act of giving. Yeah, its your day job, and yes, chances are that the quality of your diet is a reflection of the quality of the tips you receive.
But, unlike the wage-earner, you are reminded every day that there is no guarantee, no law, nothing that says that your giving must be rewarded with anything beyond the minimum wage. So what do you do? There are no guarantees in life, you will do your best, and you will likely find that, despite some bad tipping and the occasional stiff, depending on the voluntary generosity of others actually works out okay.
You earn your income not merely with your Diligence, but with Trust and with Faith.
In my life, I have found that there are times when you have to depend on the voluntary generosity of other folks. I like that tipping remains a part of our capitalist economy. It is a reminder to those of us who are “comfortable” that we too, are subjects of fate. It is a reminder that our modern enterprise still relies a great deal in faith that the great majority of people can be relied upon to give. Voluntarily.
2 Comments
Link:
https://dannyman.toldme.com/2007/07/18/email-defer/
So, I am pretty good at keeping on top of my Inbox. Every so often I plow through, and I “delete, delegate, defer, do” which means that mostly I delete or archive messages that require no action, or I’ll make entries in my calendar, then delete or archive, or I’ll write a reply, perhaps a lengthier reply. Or, I’ll transcribe the notes somewhere to work into an article, or whatever. When I’m done plowing, I sometimes have an empty Inbox. E-mail is triage and when that plate has cleared you can close it and go on to other things.
Sorting the mail . . . (CC:
KRCLA)
Delete, Delegate, and Do, are all really easy. They even map to the e-mail buttons fairly well:
| Delete |
“Delete” or “Archive” buttons. |
| Delegate |
“Forward” or make an entry in a bug / ticket system. |
| Do |
“Reply” with an answer or note that things got done.. |
| Defer |
??? . . . make an entry in your calendar? Tagging? |
Some casual poking reveals that this may be doable in Outlook, which really isn’t my style. Have any of my geeky readers thoughts or recommendations along these lines?
What I have tried, using Gmail: (more…)
3 Comments
Link:
https://dannyman.toldme.com/2007/07/17/old-notes-manifesto-charter-work-life-love/
[Some notes jotted down in the Sidekick long ago. Good stuff, I think. Maybe I should tack it on the wall somewhere, study, perhaps revise . . .]
Work
The joy of understanding problems and developing the most gratifying solutions.
The joy of learning new technologies with which to solve problems.
The satisfaction of getting things done, and being a reliable and respected resource for my coworkers.
The rewarding nature of setting expectations and goals and meeting or exceeding them.
Life
The satisfaction of walking on the Earth at different time, places, and seasons throughout my life, understanding what is consistent in myself and the world and that what is variable and “in play”.
Making connections with people, from fleeting moments of acknowledging eye-contact, to soul-sharing relationships that stretch across years and decades.
To be sufficiently self-aware about my relationship with the greater world so that I don’t take more than I need to achieve happiness.
To experience with honest fidelity the joy and the pain, the happiness and the sorrow, and all the rest of feelings and experiences that are inevitably felt in life.
Love
To practice being open and vulnerable and accepting, to allow for the possibility of love and growth in the relationships in which I engage.
To be present and attentive, to listen with good heart and a sharp mind when people speak to me.
To notice and confront dishonesty.
When “in love” to explore my partner to learn what makes them feel loved, and practice “true giving” towards them.
To always be completely honest.
Feedback Welcome
Link:
https://dannyman.toldme.com/2007/07/15/donated-equipment/
One habit that I have is that if I have gear I’m not using, I tend to give it away or lend it out to friends and acquaintances who might better use it. Especially with technology, this seems like a good idea: the utility of high-tech equipment degrades rapidly, and if a piece of equipment is going to become obsolete in the coming decade, someone ought to get the value from it.
Of course, right now I am stumbling around the apartment, looking for that DVD writer I bought a few years back. Where is it? Maybe I gave it away to someone. I guess the virtue is proper, but the accounting could be better. All the same, the reason this even comes up is because I am assembling a new workstation, mostly from retired equipment donated by friends. The DVD writer is not even mission-critical: I just wish I knew if it was in the house or not. If not, I can install a CD writer, which was donated to me some years back.
By the way, if you’re reading this: thank you Brian, Andrew, Michael, Lorah, Dennis, and everyone else!
And, if I ever gave you a 5.25″ Sony DVD writer, please remind me! =D
Feedback Welcome
Link:
https://dannyman.toldme.com/2007/07/06/listening-to-users/
I recall Tom Limoncelli giving a presentation called “Time Management for System Administrators” and he explained how, as part of his routine, he would walk over by his customers–his users within the company he worked at–and check in at a regular time. Some days, they might ask questions that would reveal to him potential improvements in the systems architecture, and other times they might ask simple technical support questions. Either way, by dropping in at regular intervals, the users came to feel good about their Systems staff. This can be damned handy when, as they occasionally do, the systems go down hard, staff scramble to fight fires, and users are left out in the cold with little more to work with than their innate feelings about the Systems staff. If they like you, they will feel sympathetic in your hours of stress. If they don’t like you, they hope the present outage may be a nail in the coffin of your tenure.
I was put in mind of this by the story presented in today’s Daily WTF . . . the user, who could be described as “dim” had been following a really complicated, error-prone process. She had no idea that a trivial change to the system could be made to make her life easier. The hero of the story happened to be walking by, hear her frustration, politely inquire, and five minutes later, make betterness happen:
Still, there’s a good lesson here that’s often missed; pay attention to what users are doing with the provided system and by unblocking minor bottlenecks you can become the hero.
Amen. Amen. Amen. (more…)
Feedback Welcome
Link:
https://dannyman.toldme.com/2007/06/28/thank-you/
I received an e-mail:
As I noted on Flickr: I just received my first "payment" for carrying Amazon.com advertising on my web site. Neat! This is hardly “f_ck you” money, but it is greater than zero.
For the record, I have been with Google AdSense for over a year now. That service brings in enough to cover the costs of my home DSL and then a wee bit more. Thanks to the generosity of friends, the hosting costs of my web site are nil, but I could probably cover that with revenue.
Of course, I am operationally “cash flow positive” but no financial incentive for content, just yet. No incentive beyond rhetorical diarrhea. :)
Pie-in-the-sky, my web site receives around 10,000 unique visits per month. I could probably “monetize” that better, but as I like to say, if I had 200 times more traffic, I could cover all of my living expenses.
Feedback Welcome
Link:
https://dannyman.toldme.com/2007/06/23/my-bottled-water/
Back in 2003, I remember being a house guest to a gracious host. He had some other guests over and one asked “is the water safe to drink?” This was a local yuppie asking about the tap water in Oakland, CA. The Bay Area has some of the best tap water in the nation. Even so, I had to resist the urge to point out that when I was in France I had treated a bout of homesickness by skimming the “Lonely Planet USA” guidebook, which explained that, in fact, tap water in the United States is safe to drink in 99.9% of localities, and that anywhere that the water isn’t safe to drink there will be plenty of warning signs: it is reasonable to assume that any American tap water is “safe”.
A few years later, my then-wife and I were guests at my Boss’ house, and dinner was served. Bottled water was made available at each place setting–nothing fancy, just standard, probably-from-Costco bottled water. In Washington, DC. I rationalized that the Boss’ wife is from India, where serving a meal with bottled water could merely be customary. No need to make an ass of myself by offering my own cultural critique of the Boss’ family’s hospitality.
I read that San Francisco recently enacted a ban on spending any further money for bottled water by city departments–currently the city spends $500,000/year on bottled water. This figure seems excessive and wasteful, but then, I read that Muni was a big spender, and I figured those guys out in the street or running vehicles don’t have easy access to tap water. It needn’t be trucked in from somewhere else–just, you know, water-in-a-bottle can be convenient, when you’re not near a tap . . .
I don’t know that you can buy municipal tap water in San Francisco or if you’ll have to bottle your own. I have a feeling that the ban will result in an increase in purchases of bottled soda and other beverages, for those cases where it isn’t a question of bottled versus tap water, but beverage-in-a-bottle versus access-to-a-tap.
For me, the most valuable aspect of bottled water is the water bottles, which can be refilled and reused to deliver fresh, delicious tap water whenever you like. These bottles are rugged and last for years of normal use.
My favorite is the “New Zealand Eternal”–I snagged this during a job interview. On the one hand, bottling water in New Zealand and transporting it to North America is–frankly–immoral. On the other hand, since it is so flamboyantly wrong, they supply the nicest bottle!
The “Tropicana” bottle is pretty rugged and has a nice, wide mouth. There is also no ambiguity–if you see me drinking water from a Tropicana bottle then its pretty clear this is some home-refill job, and I’m not some asshole who has to maintain the purity of his bodily fluids by importing water from New Zealand!
12 Comments
Link:
https://dannyman.toldme.com/2007/05/29/professionalism-integrity/
From an e-mail received from Grandma:
“It isn’t professionalism really, as much as it is your own integrity.”
Perhaps we might say that professionalism is applied integrity.
Different industries, companies, and people have their own standards of professionalism, and the surface details will vary. But I don’t think that I could ever accept an environment where professionalism is not built upon integrity.
But even more important is that I should regularly review my professional conduct and ensure that it squares with personal integrity. People get carried away with all that they must juggle day-to-day that they can sometimes stray from their ideals for the sake of short-term efficacy. (Or even short-term gratification.)
So, it is good to take some time off when we can.
Feedback Welcome
Link:
https://dannyman.toldme.com/2007/05/27/passion-initiative/
Passion stimulates initiative. You have to start somewhere and since initiative must vanquish inertia, you had better have a reason to expend so much energy.
Alas, it is a lot like falling in Love: passion begets a broken heart. It took years to overcome my first heartbreak and years to heal from the pain of disillusionment the first time I was laid off. Sometimes, to avoid pain, we limit our ambitions in work and love, and we refrain from committing ourselves to opportunities to create something wonderful. But that only leads to more profoundly tragic disappointments.
I think that if you have the good fortune to find yourself passionate about something, then perhaps what you need to do is to cultivate initiative; Passion is the why and initiative is the what. When you fail in the pursuit of your passion, initiative can sustain you: when you lack the why, at least you still have the what. With a faith in your initiative and a mind open to new opportunities, you should sooner find the next thing that captures your passion, and you can fall in love anew, backed by the strengths gained from previous endeavors.
Feedback Welcome
Link:
https://dannyman.toldme.com/2007/05/18/fukn-quadr0/
Systems Administrators can be an uptight bunch. In the past few weeks I have twice spent some time amidst my fellow professionals. Most are nice people, a good many are inoffensively undersocialized, and a noisy minority are just flamingly obnoxious. (I have, at times, been flamingly obnoxious.) Two nights ago one of my fellows recommended Cory Doctorow’s mind-churning post-apocalyptic masterpiece “When SysAdmins Ruled the Earth” . . . I dare you to read it!
Not long ago I joined a professional mailing list, and today I thought I would chime in on the topic of mobile phone reimbursement. I received a polite note from the list moderator: my message had bounced, could I please re-sends the message as plain text only. These days I am using Gmail, which sends messages in the ubiquitous multipart/alternative format, which leads with text that is followed by a potentially-prettier HTML “alternative”.
I dug around in the preferences to see where I could set “text only” but couldn’t find anything, and took that as a sign that in 2007, even Google doesn’t care about supporting this antiquated preference. I have since noticed that you can just click “plain text” right in the tool bar while you are sending a message. But . . . well, I felt inclined to engage in the time-honored tradition of obnoxious computer experts and impose upon the guy my social-technological criticism of the status quo in the form of a well-crafted flame: (more…)
2 Comments
Link:
https://dannyman.toldme.com/2007/05/16/theophoric-daniel/
I ended up wandering around Wikipedia, and discovered:
- My given name, Daniel, literally means “judge-is-god” in Hebrew
- Though, Hebrew goes right-to-left, so ×“Ö¸Ö¼× Ö´×™ÖµÖ¼×ל literally reads (to me) as “el is judge”
- Daniel is thus theophoric, which means it “embeds the name of a god, both invoking and displaying the protection of that deity.”
- “El was the supreme god, the father of mankind and all creatures and the husband of the Goddess Asherah.”
- In Judaism, God has many names: “the various names of God in Judaism represent God as he is known, as well as the divine aspects which are attributed to him.”
- The historic figure of Daniel was a pious Jew who spent most of his life away from his kinfolk, working for King Nebuchadnezzar II of Babylon.
- Daniel interpreted the handwriting on the wall that had been left by Angels, which foretold the imminent assassination of King Belshazzar and the subsequent fall of Babylon to Persia.
- Under Persian rule, Daniel was given great administrative authority, and likely played a significant role in freeing the Jews from the “Babylonian Captivity”, though he himself did not return to Jerusalem with them.
- Later in his life, Daniel ministered as a prophet. In Christianity he is one of the “four great prophets” but in Judaism, he is not regarded as a prophet, because he did not speak directly with God.
And I can’t boil this paragraph down to a bullet point:
Daniel’s fidelity to God exposed him to persecution by jealous rivals within the king’s administration. The fact that he had just interpreted the emperors’ dream had resulted in his promotion and that of his companions. Being favored by the Emperor, he was untouchable. His companions were vulnerable to the accusation that had them thrown into the furnace for refusing to worship the Babylonian king as a god; but they were miraculously saved, and Daniel would years later be cast into a den of lions (for continuing to practice his faith in YHWH), but was miraculously delivered; after which Darius issued a decree enjoining reverence for “the God of Daniel” (6:26). He “prospered in the reign of Darius, and in the reign of Cyrus the Persian,” whom he probably greatly influenced in the matter of the decree which put an end to the Jewish Captivity (B.C. 536).
Anyway, if I wanted to draw anything from this namesake, it would be that spending most of your adult life far from home serving a doomed empire is perfectly reasonable, but keep pious in order that one may interpret the “writing on the wall” which can lead one to develop the power to free people. Piety will further bring the benevolence of God, which will come in damned handy on those occasions when one is tossed into the lion’s den.
And here I am sitting around in California with the moniker “dannyman” which could be read “judge is man” . . . which strikes me as awfully impious.
I guess I’ll have to try harder. And I’ll have to rely on my own good judgment. Dang.
Feedback Welcome
Link:
https://dannyman.toldme.com/2007/05/14/colorado-road-trip/
So, I am catching up with all sorts of things these days, and there is the constant threat that I may return to full-time work at any moment. Those with an appetite for autobiographical detail may like to know that I have just posted several back-dated entries from last month, covering my road trip out to Pueblo, CO, where I visited Dad and his family. URLs for the voyeuristic:
http://dannyman.toldme.com/2007/04/11/driving-to-long-beach-ca/
http://dannyman.toldme.com/2007/04/12/long-beach-ca/
http://dannyman.toldme.com/2007/04/13/vegas-baby/
http://dannyman.toldme.com/2007/04/14/forlorn-stuck-in-moab-again/
http://dannyman.toldme.com/2007/04/15/bad-starter-dont-stop/
That’s what I have written up so far . . . as a spoiler: I made it to Pueblo just fine, replaced the starter myself, I bought a new camera, we celebrated Gwen’s birthday, had some quality time, and then since I couldn’t spend enough time in Pueblo, I took Dad back to San Francisco with me, where he spent a week that we both enjoyed a great deal. I dropped him off in Emeryville and he took the train home. Last Wednesday I went with a friend to see the awesome movie “The Valet” and on the way to the movie I got a call from Dad that he had driven the car to Wal*Mart, which had been a scary ride for Gwen, but proof that he could help drive on their own road trip to Chicago later this month.
At any rate, I hope to write a bit more about these things, but I make no promises. Along the way, I am still uploading pictures, so keep an eye on Flickr, and if the remainder of the trip interests you, you can skim over the “Colorado Road Trip April, 2007” collection.
Feedback Welcome
Link:
https://dannyman.toldme.com/2007/04/26/resume-thanksgooglecom/
Well, this is kind of neat. Nowadays Google sends you an automated message telling you they’re not interested:
From: resume-thanks@google.com
Subject: Thank you
We received your resume and would like to thank you for your interest in
Google. After carefully reviewing your experience and qualifications, we
have determined that we do not have a position available which is a strong
match at this time.
Thanks again for considering Google. We wish you well in your endeavors
and hope you might consider us again in the future.
Sincerely,
Google Staffing
Just for fun, I had submitted a resume, to see what random stuff Google might come up with. The reason being that any time I had applied in the past, what always happened is I would hear nothing for a month or two, then I would be contacted by a recruiter for a completely different position, and the recruiter would have no idea about the position I had originally applied for.
Last time, though, the position the recruiter proposed was more interesting than the one I had found on my own. My best understanding–and my understanding may be out-of-date–of the Google hiring process is that the managers meet with the recruiters on a weekly basis, so it will typically take two to three weeks for a recruiter to get a candidate into the pipeline: (more…)
Feedback Welcome
« Newer Stuff . . . Older Stuff »
Site Archive
