After yesterday’s post, I figured I would have to re-synchronize the slave database from the master, but probably build a more capable machine before doing that. I figured at that point, I might as well try fiddling with MySQL config variables, just to see if a miracle might happen.
At first I twiddled several variables, and noticed only that there was less disk access on the system. This is good, but disk throughput had not proven to be the issue, and replication lag kept climbing. The scientist in me put all those variables back, leaving, for the sake of argument, only one changed.
This morning as I logged in, colleagues asked me what black magic I had done. Check out these beautiful graphs:
The default value of this variable is 1, which is the value that is required for ACID compliance. You can achieve better performance by setting the value different from 1, but then you can lose at most one second worth of transactions in a crash. If you set the value to 0, then any mysqld process crash can erase the last second of transactions. If you set the value to 2, then only an operating system crash or a power outage can erase the last second of transactions. However, InnoDB’s crash recovery is not affected and thus crash recovery does work regardless of the value. Note that many operating systems and some disk hardware fool the flush-to-disk operation. They may tell mysqld that the flush has taken place, even though it has not. Then the durability of transactions is not guaranteed even with the setting 1, and in the worst case a power outage can even corrupt the InnoDB database. Using a battery-backed disk cache in the SCSI disk controller or in the disk itself speeds up file flushes, and makes the operation safer.
The Conventional Wisdom from another colleague: You want to set innodb_flush_log_at_trx_commit=1 for a master database, but for a slave–as previously noted–is at a disadvantage for committing writes, it can be entirely worthwhile to set innodb_flush_log_at_trx_commit=0 because at the worst, the slave could become out of sync after a hard system restart. My take-away: go ahead and set this to 0 if your slave is already experiencing excessive replication lag: you’ve got nothing to lose anyway.
(Of course, syslog says the RAID controller entered a happier state at around the same time I set this variable, so take this as an anecdote.)
As of 11AM this morning, and until 11AM next Tuesday, I’m “on call” . . . which means that if something breaks, especially at 3AM, I’m the first guy responsible for fixing it.
This is actually a new form of “on call” for me–this is the first time I have been in a “rotation”. At other, smaller companies, I have spent years on-call. Now, that isn’t quite so bad in a small environment where things seldom fail, but it is something of a drag to keep your boss informed of your weekend travel plans so he can watch for pages in your stead. In a larger environment, a week spent on-call can be particularly onerous, because there are plenty of things that will break. But, come the end of the week, you pass the baton . . .
So, this week, I will get my first taste, and over time I will have a better sense as to whether “on call” is better in a smaller environment or a larger environment. I have a feeling that while this week could be rough, that the larger environment is an overall better deal: there is a secondary on-call person, there is an entire team I can call for advice on different things, and the big company provides nice things like a cellular modem card, and bonus pay for on-call time.
I was dining out with coworkers, in a group of four. These colleagues were nobody specific: just extras fabricated from spare parts in the subconscious. The topic drifted to the subject of building rapport, and how light physical touches can build a connection with someone, but you might be careful about that in the work place. I reached across the table to brush my colleague’s wrist, and he leaned back, grinning. My hand came to a stop before it would have come over his dinner plate. I smiled back, “and this is about the line where I would have invaded your personal space,” and withdrew.
I headed to the bathroom, where there was a short line waiting outside the men’s room. One or two guys turned back, not wanting to stand in line, and thus making it shorter. I was confident that the line would move quickly, and in a moment I was attending to my business at a urinal. (more…)
I recently participated in some beta test challenge thing for something called Yahoo! Insiders. They sent me some schwag, including a nice little flashlight that came without the requisite 3 AAA batteries, and a cute little USB mouse that is too tiny for my massive hand. The program consisted of 9 “challenges” which basically boiled down to “use our search engine to find the answer to this question and you might win a prize.” (The prizes were nice, one day was a nice digital camera.) The search engine had some “suggestions” of what search terms you might be better off searching, which would appear if you clicked a little widget. Kind of like the Google spell checker, but with synonyms.
I didn’t use the feature because, well, it was buried under a widget and because I’m pretty good with typing keywords into search engines. I’m guessing they think “suggested keywords” might do something for newbies, though it really isn’t clear.
They just solicited some feedback. I filled out the form, and at the end they asked “is there anything at all that you would like us to know about Yahoo!, The Yahoo! Search Insiders Program or Yahoo! Search Assist?” I thought a moment, then:
What are you trying to accomplish? Build a slightly better search engine? Google works awfully damn well 90% of the time, so the bar to get anyone to switch for “better” is extremely high. Maybe you can put your massive resources behind a more ambitious idea like combining social bookmarking with Netflix/Amazon-style “recommendations” and thereby build a more personalized “Page Rank” index using social networking . . . the sort of thing Google SUCKS at.
Cool things I have come into as a consequence of volunteering with One Brick these past few months:
$50 gift card for Williams-Sonoma at the Elks Lodge Blood Drive
My new job, after a tip from a One Brick volunteer coordinator
This past weekend, a hand-me-down laptop that I can soon re-gift
Or, as Saint Francis put it: “it is in giving that we receive.”
If you are looking for fulfilling ways to spend your free time, I heartily recommend One Brick, which is very simply an organization that organizes volunteer opportunities: just sign up for their e-mail list and every week you’ll be informed of cool opportunities to get out, do some good, and make friends.
Today I took myself out to a movie. I couldn’t find anyone else who was interested in “The Glamorous Life of Sachiko Hanai” at the Red Vic. But now I know the answer to the question of what might happen if a Tokyo call girl gets shot in the forehead, giving her super intellectual powers, and then finds in her possession the cloned trigger finger of George W Bush, and is thus chased by North Korean agents looking to control a Russo-Uzbek doomsday device. I’m not sure if it is a porno or a porn parody, but especially the early part of the movie involves excessive quantities of semen. For that reason I am glad that I was shy about asking friends to go see it. This movie is wrong in so many ways that I see why it has become a cult classic.
If you live in San Francisco and possess a sufficiently perverse sense of humor and politics, its run at the Red Vic concludes tomorrow Monday August 20, with showings at 7:15pm and 9:15pm.
I was slow in getting away from Pueblo. The Colorado side of the family isn’t a hurried bunch and especially with Dad in the hospital nobody but me felt any haste in leaving. “Only the weekend,” I demure. Dad’s second stroke arrived just as I went to my first lunch with new co-workers on Monday. After not-working for nearly five months, I had selected this fateful day to get started at a new job?
He’s doing pretty well, for a guy who can’t talk and who requires 24-hour nursing assistance, a guy who has several weeks of therapy at the hospital before he gets to return home, and years more of therapy ahead. (more…)
I use the Amazon.com Chase Visa. I get a “point” per dollar spent, and three points per dollar on purchases through Amazon.com. Every 2,500 points, I get a $25 gift certificate. Pretty neat.
Well, they hadn’t sent me gift certificates in a while, so I called and got the matter cleared up. I am currently working my way through $350 in gift certificates.
If you do the math you may surmise that I spend an awful lot of money. I will offer a tip that if you want to maximize your credit card rewards, you should manage IT for a small company that relies upon you to charge equipment to your credit card and then be reimbursed. Especially if you have earned a better credit limit than your boss’ corporate card.
“The coldest Winter I ever knew was a summer in San Francisco.”
It has been overcast, chilly and wet in my neighborhood throughout July. Monday the sun came out for about an hour in the morning, then again at sunset. I ran out of the house when that happened but it was too late in the day to get much sun. The midwesterner in me reminds myself that this is a temporary and “symbolic” Winter, without the snow. It is just weird the way the seasons work when you live adjacent to the Pacific Ocean.
(No, I’m not actually depressed. Well, this gray does make me blue, and that is why I am conscientious about getting out doors any time the clouds break. I am supposed to be starting work next week, so I should be getting more sun during the week.)
(I like what Yelp have done with their down page.)
The short story is that an underground transformer exploded downtown, and the 365 Main data center failed to automatically start their generators, and had to start them manually, cutting power for nearly an hour for some customers, many of which are smaller, trendier web sites like Craigslist, LiveJournal, Yelp and others. (I have interviewed with half of the companies mentioned in Scott’s post.)
You do not want to lose power across a production-class network. This can cause equipment failure, servers to delay boot because they need to run disk consistency checks, servers to stall boot noting a missing keyboard, disk errors, or whatever. Some services may wedge up because when they started they couldn’t talk to the database . . . in some cases you may have had machines running for a few years, which may have last rebooted three SysAdmins ago. The running state may be subtly different from the boot state, with no documentation . . .
A few years ago I had a chance to rebuild a production network from the ground up, with a decent budget to do everything the right way: redundant network switches, serial consoles, remote power management . . . I remember talking to my manager as to whether we might want a UPS in each rack. We figured that the data center is supposed to keep the power running, or else. Also, if the data center loses power then we lose our network access anyway . . . perhaps the whole point of this post is that data centers do lose power, so a UPS can be worthwhile. If nothing else, it may leave your systems up and ready to go as soon as the network is restored.
Data centers have UPSes too. Huge ones that you may get to walk through on a tour. The purpose of the UPS is to provide battery power between the time utility power fails and on-site generators begin to provide energy. I don’t know enough to comment on this particular case, but I do recall touring a data center in Emeryville, and the guy explained that batteries become less effective over time, and a lot of data centers fail to test their batteries regularly. When wired in series, one bad battery brings down the entire UPS, and so even though you have a generator on-site, the UPS can fail before you manage to transfer to generator power. While this stuff is beyond my expertise, I’m inclined to believe that this is what happened at 365 Main yesterday: a data center should not only test its failover-to-generator procedure on a regular basis, they need to ensure sufficient battery capacity to keep systems running during the time it would reasonably take to switch to generator power.
On the weekend of July 22 and 23, I and about 400 other folks attended WordCamp 2007 in San Francisco. This is a conference about WordPress blogging software, and blogging itself. I am usually a bit wary of killing my weekend by spending the bulk of it with a bunch of nerds. Especially bloggers. But then, I am a nerd, and this is, I admit, a blog . . . that and registration was merely $25 and covered my food for the weekend. That’s a pretty compelling deal for the unemployed! Added value was found at the open bar on Saturday night at one of my favorite bars: Lucky 13.
Here are notes I compiled during the Saturday presentations. (more…)
Hello, it’s me again. This Matthew Malooly feller has some interesting web site. He’s a lab-sitter like me, I think. Someday I’ll do like the old site had and have a list of web pages I like. Good, well-written and informative web sites you know, about people that introduce them to their mind, assuming they have one. If they’re dull they mightn’t bother with a web site in the first place. And if you find me dull, well feel free to go elsewhere, I’m not forcing you to read.
Which is one great thing about the web. You read what you want, you see and listen to what you want, and what is nasty you just avoid.
Actually, the dude has a link to my page. I remember now. It’s like we see eye to eye on this web page stuff.
Well, on through other URLs I have sitting in my mail to go through …
Okay, so it’s been a good haul, and I now have a new page up to deal with the fact. Yay!
So after class today I scanned pictures. Gotta start gettin’ goin’ again you know? Lotsa good ones on the way, but you won’t see ’til I’m done settin’ everythang up. That might be awhile. Sorry. Like fine wine … nothing before it’s time you know?
I wish I could think of a few interesting things to say here? Well, let’s see, I did think yesterday to maybe start and this time stick to carrying a little “idea” book around with me … what have we got?
A procmail “mail filter” CGI “control center” — ‘nuf said! Huh? Well, it’ll be a relatively complex CGI ultimately for EnterAct to implement for it’s users. The idea seemed particularly keen last night after I’d read that Tom was gonna audit IMAP so it was secure enough to run on EnterAct’s systems. Means two things to me – one is that there should be coming about a secure implementation of IMAP (for BSD) and two that people might find the most basic features of procmail to be useful: primarily of simple sorting for multiple INBOXes accessible via IMAP and perhaps SPAM filtering. I’m still working out details in my mind while idly wondering what likelihood of and where I’m gonna find the free time to engage in such a project. Yikes!
Story idea — Alright, point-of-view of a tree as it’s being chopped down … you know, what it feels about humans, what it knows about them. The history of how the humans keep going in number, the lumberjack’s perspective. The trick though is that I know nearly nothing of forestry or what it may be like to be a lumberjack.
Rhet 143 — Gotta write this gay “Narrative of Place” … so why not go to the moon eh? Cold, timeless, sterile, no air, dark and desolate. Avec ma solitude … well, dunno, we’ll see but Seshagiri wants a draft for Wednesday. Grrr!
Well, another crap journal entry muddled through. Go back and enjoy Matt’s site, and that of Brian Lee the rat boy! I go do something else now. I’m tired.
The subway broke again, so our streetcar is waiting patiently at West Portal. I wish Muni were more robust. They have always done well at providing shuttle buses when things are broken, but I’d rather depend on a system that didn’t need to be so adept with shuttle buses.
