After yesterday’s post, I figured I would have to re-synchronize the slave database from the master, but probably build a more capable machine before doing that. I figured at that point, I might as well try fiddling with MySQL config variables, just to see if a miracle might happen.
At first I twiddled several variables, and noticed only that there was less disk access on the system. This is good, but disk throughput had not proven to be the issue, and replication lag kept climbing. The scientist in me put all those variables back, leaving, for the sake of argument, only one changed.
This morning as I logged in, colleagues asked me what black magic I had done. Check out these beautiful graphs:
The default value of this variable is 1, which is the value that is required for ACID compliance. You can achieve better performance by setting the value different from 1, but then you can lose at most one second worth of transactions in a crash. If you set the value to 0, then any mysqld process crash can erase the last second of transactions. If you set the value to 2, then only an operating system crash or a power outage can erase the last second of transactions. However, InnoDB’s crash recovery is not affected and thus crash recovery does work regardless of the value. Note that many operating systems and some disk hardware fool the flush-to-disk operation. They may tell mysqld that the flush has taken place, even though it has not. Then the durability of transactions is not guaranteed even with the setting 1, and in the worst case a power outage can even corrupt the InnoDB database. Using a battery-backed disk cache in the SCSI disk controller or in the disk itself speeds up file flushes, and makes the operation safer.
The Conventional Wisdom from another colleague: You want to set innodb_flush_log_at_trx_commit=1 for a master database, but for a slave–as previously noted–is at a disadvantage for committing writes, it can be entirely worthwhile to set innodb_flush_log_at_trx_commit=0 because at the worst, the slave could become out of sync after a hard system restart. My take-away: go ahead and set this to 0 if your slave is already experiencing excessive replication lag: you’ve got nothing to lose anyway.
(Of course, syslog says the RAID controller entered a happier state at around the same time I set this variable, so take this as an anecdote.)
I’ve got a MySQL slave server, and Seconds_Behind_Master keeps climbing. I repaired some disk issues on the server, but the replication lag keeps increasing and increasing. A colleague explained that several times now he has seen a slave get so far behind that it is completely incapable of catching up, at which point the only solution is to reload the data from the master and re-start sync from there. This isn’t so bad if you have access to the innobackup tool.
The server is only lightly loaded. I like to think I could hit some turbo button and tell the slave to pull out all the stops and just churn through the replication log and catch up. So far, I have some advice:
2) MySQL suggest that you can improve performance by using MyISAM tables on the slave, which doesn’t need transactional capability. But I don’t think that will serve you well if the slave is intended as a failover service.
Your options are fairly limited. You can monitor how far behind the slave is . . . and assign less work to it when it starts to lag the master a lot . . . You can make the slave’s hardware more powerful . . . If you have the coding kung-fu, you might also try to “pipeline the relay log.”
A troop of lone-tein (riot police comprised of paid thugs) protected by the military trucks, raided the monastery with 200 studying monks. They systematically ordered all the monks to line up and banged and crushed each one’s head against the brick wall of the monastery. One by one, the peaceful, non resisting monks, fell to the ground, screaming in pain. Then, they tore off the red robes and threw them all in the military trucks (like rice bags) and took the bodies away.
The head monk of the monastery, was tied up in the middle of the monastery, tortured , bludgeoned, and later died the same day, today. Tens of thousands of people gathered outside the monastery, warded off by troops with bayoneted rifles, unable to help their helpless monks being slaughtered inside the monastery. Their every try to forge ahead was met with the bayonets.
When all is done, only 10 out of 200 remained alive, hiding in the monastery. Blood stained everywhere on the walls and floors of the monastery.
There are some pretty nasty photographs on that page, and video of civilians falling to gunfire.
Images of saffron-robed monks leading throngs of people along the streets of Rangoon have been seeping out of a country famed for its totalitarian regime and repressive control of information.
The pictures are sometimes grainy and the video footage shaky – captured at great personal risk on mobile phones – but each represents a powerful statement of political dissent.
[. . .]
Burmese-born blogger Ko Htike, based in London, has transformed his once-literary blog into a virtual news agency and watched page views rise almost tenfold.
He publishes pictures, video and information sent to him by a network of underground contacts within the country.
“I have about 10 people inside, in different locations. They send me their material from internet cafes, via free hosting pages or sometimes by e-mail,” he told the BBC News website.
“All my people are among the Buddhists, they are walking along with the march and as soon as they get any images or news they pop into internet cafes and send it to me,” he said.
[. . .]
Reporters without Borders describe how a guide for cyber-dissidents provided to young Burmese was seized upon, copied and feverishly disseminated among a growing group of the young, politically active and computer-literate.
Bloggers are teaching others to use foreign-hosted proxy sites – such as your-freedom.net and glite.sayni.net – to view blocked sites and tip-toe virtually unseen through cyberspace, swapping tricks and links on their pages.
I really like to see people taking power for themselves, and I find it all the more gratifying to see the Internet used as a tool in this process.
I’m sure the Chinese government is watching this process very carefully. The BBC article indicates that the government used to be more effective in its Internet censorship efforts: (more…)
If you have a couple of hours free, I recommend sitting back and watching this video of Randy Pausch’s “final” lecture at CMU. He is a smart, talented, ambitious, and accomplished professor who seems to know how to give a lecture, and on this occasion he delivers a lecture some months before he is expected to die of cancer.
He isn’t talking about cancer or dying. He is talking about his life and his advice on how to live life well. I have no commentary; I enjoyed this special moment a great deal and I believe that it is worth sharing.
As of 11AM this morning, and until 11AM next Tuesday, I’m “on call” . . . which means that if something breaks, especially at 3AM, I’m the first guy responsible for fixing it.
This is actually a new form of “on call” for me–this is the first time I have been in a “rotation”. At other, smaller companies, I have spent years on-call. Now, that isn’t quite so bad in a small environment where things seldom fail, but it is something of a drag to keep your boss informed of your weekend travel plans so he can watch for pages in your stead. In a larger environment, a week spent on-call can be particularly onerous, because there are plenty of things that will break. But, come the end of the week, you pass the baton . . .
So, this week, I will get my first taste, and over time I will have a better sense as to whether “on call” is better in a smaller environment or a larger environment. I have a feeling that while this week could be rough, that the larger environment is an overall better deal: there is a secondary on-call person, there is an entire team I can call for advice on different things, and the big company provides nice things like a cellular modem card, and bonus pay for on-call time.
Let’s welcome every foreign student we can get our hands on. Let’s make sure that foreigners come to the Mayo Clinic here, and not the Mayo facility in Dubai or somewhere else. Let’s make sure people come to Disney World and not throw them up against the wall in Orlando simply because they have a Muslim name. Let’s also remember that this country was created by immigrants and thrives as a result of immigration, and we need a sound immigration policy.
Let’s show the world a face of openness and what a democratic system can do. That’s why I want to see Guantánamo closed. It’s so harmful to what we stand for. We literally bang ourselves in the head by having that place. What are we doing this to ourselves for? Because we’re worried about the 380 guys there? Bring them here! Give them lawyers and habeas corpus. We can deal with them. We are paying a price when the rest of the world sees an America that seems to be afraid and is not the America they remember.
Amen! Let’s stop hiding behind an Iron Curtain of Fear.
Are there any terrorists in the world who can change the American way of life or our political system? No. Can they knock down a building? Yes. Can they kill somebody? Yes. But can they change us? No. Only we can change ourselves.
I was dining out with coworkers, in a group of four. These colleagues were nobody specific: just extras fabricated from spare parts in the subconscious. The topic drifted to the subject of building rapport, and how light physical touches can build a connection with someone, but you might be careful about that in the work place. I reached across the table to brush my colleague’s wrist, and he leaned back, grinning. My hand came to a stop before it would have come over his dinner plate. I smiled back, “and this is about the line where I would have invaded your personal space,” and withdrew.
I headed to the bathroom, where there was a short line waiting outside the men’s room. One or two guys turned back, not wanting to stand in line, and thus making it shorter. I was confident that the line would move quickly, and in a moment I was attending to my business at a urinal. (more…)
Yahoo! is a California company. A few years back they complied with Chinese law to reveal the identity of Chinese dissidents who then became Chinese political prisoners. Now the dissidents are suing Yahoo! for violating a California civil rights law.
“This is a lawsuit by citizens of China imprisoned for using the internet in China to express political views in violation of China law. It is a political case challenging the laws and actions of the Chinese government. It has no place in the American courts.”
Which, on the face of it, sounds fair, but Yahoo! made the choice to engage the Chinese government and Chinese law and thereby send Chinese citizens to prison. I don’t approve. And as a California resident, I figure Yahoo! is reaping what it sowed.
I recently participated in some beta test challenge thing for something called Yahoo! Insiders. They sent me some schwag, including a nice little flashlight that came without the requisite 3 AAA batteries, and a cute little USB mouse that is too tiny for my massive hand. The program consisted of 9 “challenges” which basically boiled down to “use our search engine to find the answer to this question and you might win a prize.” (The prizes were nice, one day was a nice digital camera.) The search engine had some “suggestions” of what search terms you might be better off searching, which would appear if you clicked a little widget. Kind of like the Google spell checker, but with synonyms.
I didn’t use the feature because, well, it was buried under a widget and because I’m pretty good with typing keywords into search engines. I’m guessing they think “suggested keywords” might do something for newbies, though it really isn’t clear.
They just solicited some feedback. I filled out the form, and at the end they asked “is there anything at all that you would like us to know about Yahoo!, The Yahoo! Search Insiders Program or Yahoo! Search Assist?” I thought a moment, then:
What are you trying to accomplish? Build a slightly better search engine? Google works awfully damn well 90% of the time, so the bar to get anyone to switch for “better” is extremely high. Maybe you can put your massive resources behind a more ambitious idea like combining social bookmarking with Netflix/Amazon-style “recommendations” and thereby build a more personalized “Page Rank” index using social networking . . . the sort of thing Google SUCKS at.
Cool things I have come into as a consequence of volunteering with One Brick these past few months:
$50 gift card for Williams-Sonoma at the Elks Lodge Blood Drive
My new job, after a tip from a One Brick volunteer coordinator
This past weekend, a hand-me-down laptop that I can soon re-gift
Or, as Saint Francis put it: “it is in giving that we receive.”
If you are looking for fulfilling ways to spend your free time, I heartily recommend One Brick, which is very simply an organization that organizes volunteer opportunities: just sign up for their e-mail list and every week you’ll be informed of cool opportunities to get out, do some good, and make friends.
A: This isn’t for everyone, but here is the deal I am on right now:
$14/mo for Netflix
$12/mo for DSL------------------
$26/mo Video+Internet
This lets me watch a few movies each week, and when I really want to watch TV I can download the “Daily Show” with the commercials already edited out from BitTorrent. Since adopting this plan I have gotten more into the “shopping for my own food and cooking it myself” channel, the “tidying up and arranging my own apartment” channel, the low-key reality show “can dannyman take care of these flowers” and some call-in shows featuring friends and family. I’m considering some plus packages like “my new pet” and maybe “learn a musical instrument” but I haven’t even gotten in to the last one I tried: “Mandarin Chinese”
One of the big drawbacks to this approach is that there’s not much of a channel guide to help me keep track of all the possibilities, and good luck finding a universal remote! On the other hand, the commercials are pretty rare and innocuous, so you don’t need a DVR.
