Wikipedia:Wikipedia Signpost/Single/2005-02-28
Power outage halts editing for a day, but not donations
Wikipedia suffered a major interruption last Monday when the Wikimedia servers lost power due to circuit breakers tripping at their colocation facility. Read-only service was restored after several hours, although it took more than a day for editing to resume, but in the end no significant data was lost.
The servers crashed at about 22:14 (UTC) on Monday, 21 February, due to a loss of power at Neutelligent, the Tampa, Florida company that provides hosting services for the primary set of Wikimedia servers. After some scrambling to determine why network access had been lost, Jimbo Wales contacted Neutelligent and learned that the cause was the tripping of circuit breakers.
In spite of the fact that some of the servers had a second power supply on a separate circuit, this circuit breaker tripped as well, leaving all of the servers entirely without power. Speculative theories suggested that because the tripping mechanism is magnetic, one circuit breaker tripping could also have caused others around it to trip, or that the servers all simultaneously switching over to the new circuit might have overloaded it as well. However, the actual reason for the circuit breakers tripping remains unknown.
The recovery
Once power was restored, the developers then faced the challenge of bringing the servers back online from an unplanned shutdown. This was complicated by the fact that all of the databases actively replicating at the time of the crash ended up being corrupted. Fortunately, one slave database server had been halted earlier and was storing updates in a log rather than actively applying them to its copy of the database.
This intact copy was used to restore read-only service initially, while the logs were being applied to bring the database back to its current state before the power went out (at most, a few edits in process at the moment of the crash may have been lost). Additional technical details can be found in the report on Meta. Wiki editing was restored at 22:26 (UTC) on Tuesday, 22 February, just over 24 hours after the initial crash.
In order to get back to a full level of service, the database needed to be copied again to several servers, and features such as watchlists and contribution histories remained mostly unavailable over the next day. Performance also stayed rather slow for several days.
LiveJournal comparisons
The incident had several points in common with the power outage last month that took down another community website with a sizable database,
LiveJournal, which uses a MySQL database like Wikimedia, needed a similar amount of time to restore service to users after its power outage. They also had to deal with corruption in the databases that had been shut down. Based on LiveJournal's experience, some developers suggested that the data corruption in the active copies of the database might have been due to the RAID hardware giving the database incorrect reports that information had been written to the hard disk when it was actually being stored in cache first.
Brad Fitzpatrick, founder of LiveJournal, expressed his sympathies over the situation as the developers were working to restore service.
Fundraising progress during downtime
Pages related to the ongoing Wikimedia fundraiser (see archived story) were naturally disabled along with everything else. As an emergency measure, a static version of the main fundraising page was temporarily hosted on Angela's website, but the traffic there, with a Slashdot story about the crash also contributing to the load, soon exceeded her bandwidth quota. About eight hours after power was lost, Wikimedia was able to make another backup of the fundraising page available on its own servers.
Wikimedia CFO
The Report On Lengthy Litigation
The Arbitration Committee devised a way to handle violations of the No legal threats policy last week, a type of conduct for which it had not previously needed to construct a specific remedy. New cases also came up as problems over circumcision-related articles continued, this time involving a different editor, and a privately negotiated arrangement with Anthony DiPierro collapsed and went into arbitration.
No using the wiki while maintaining legal threats
While threats of legal action had been cited in the original decision against RK, no specific remedy was applied on that point beyond the general four-month ban RK received. In the case of WikiUser, however, the arbitrators had to consider evidence that he had directly stated that he planned on suing Wikipedia.
In June 2004, only a few days after first beginning to edit Wikipedia and getting into a dispute with several users over the
Besides all of this, the arbitrators found evidence of personal attacks and determined that WikiUser had significantly disrupted Wikipedia in a variety of ways, including vandalism and edit warring. These problems, rather than the legal threats themselves, led first to a request for comment and then finally the arbitration case. The personal attacks and disruption each earned a one-year ban, with the bans to run concurrently.
After the year was concluded, the ruling provided that the ban for making legal threats would also end, assuming that WikiUser did not make further threats or take other legal action. The ban would have to remain in place if legal action was initiated; arbitrator Sannse made the point that she considered this a protective measure rather than a punitive one. Meanwhile, if WikiUser returns at the end of one year, the decision includes a six-month probation period during which he can be blocked for disruptive behavior.
New cases
The ban of Robert the Bruce (arbitration) apparently was not sufficient encouragement for some of the activists on the circumcision articles to improve their behavior, and this time one of the editors on the opposite side of the controversy, Robert Blair, became the subject of an arbitration case requested by Alteripse. The arbitrators voted for a temporary injunction prohibiting Blair from editing articles in the subject area, as they had previously with Robert the Bruce, which was scheduled to go into effect on Monday.
Meanwhile, a dispute that had narrowly avoided arbitration on previous occasions came to a head over Anthony DiPierro's efforts to preserve articles that have gone through the
The arbitrators voted to accept the case on Saturday, and DiPierro withdrew from the agreement to the standing order after the case was accepted. In addition to Raul654, four other arbitrators decided to recuse themselves due to various levels of personal involvement in the matter, leaving only six arbitrators to hear the case with
Critiques of Wikipedia remain hot topic, both outside and in
The volume level in the ongoing public debate over Wikipedia picked up again considerably last week. Arguments and rebuttals continued to circulate while Wikipedia editors reacted to these discussions by making changes in the encyclopedia as appropriate.
The second issue of the fledgling Free Software Magazine published an article last week entitled "The FUD-based Encyclopedia", written as a response to last year's article from former Britannica editor Robert McHenry, "The Faith-Based Encyclopedia". Author Aaron Krowne, one of the founders of PlanetMath, gave the glowing review, "I cannot remember the last time I accessed a Wikipedia article that was not of apparent professional quality." Krowne set out to rebut McHenry's commentary point by point, and the arguments were further aired in a Slashdot discussion about the article.
Digging up conspiracy theories
An article that was "not of apparent professional quality" was not nearly so hard to find for a blogger who posts under the name Callimachus, however. On a blog called Done With Mirrors, he told of finding the
Not only was the article sketchy, Callimachus commented, "It all looks like a lot of conspiracy theory hoo-ha." The article briefly mentioned the company's employee losses in the
Instalanche season in effect
Instapundit stirred up additional traffic on Friday by linking to Callimachus' post, and later updating it with a link to older criticism from Judith Weiss of Kesher Talk, who has been collecting examples of complaints about Wikipedia. The conspiracy theory material was then removed, but later readers coming from Instapundit still left their commentaries on the talk page.
This followed an earlier posting 20 February on Instapundit calling attention to the
How to handle criticism?
In the middle of these arguments and counterarguments over critiques of Wikipedia, the article
The idea of merging the "Criticisms" content back into the main article was suggested by several people in the original vote, and
Wikipedia seen as possible competitor in vertical search
The race for a better search engine was cast in a different light last week, as analyst David Coursey explored the possibilities of "vertical search", and Wikipedia's position in this area could lead to new competition.
In an opinion column last Thursday on eWeek, "Search Engines Succeed at Stoking Frustration", Coursey comments on his increasing difficulties finding useful answers through Google search results. Taking a cue from another reporter's article about vertical search, he suggests, "Sometimes I also think that for common searches I might be better off with a service that actually involves humans in finding answers online, if any of those still exist."
Vertical search is a term used in a recently-released study from JupiterResearch, referring generally to narrower search engines that are focused on specific categories. It relies less on the web crawler approach used by the broad search engines ("horizontal search"), and could offer an alternative to advertisers dealing with the prices of popular keywords at the big search companies. While the concept is not clearly defined, it seems that hyperlinking is one feature that could easily contribute to the "vertical" aspect.
Coursey specifically mentions Wikipedia twice in his analysis, saying that this kind of service "is not too far off the Wikipedia track." Although he doesn't mention the discussions of Google hosting Wikipedia — he might not know about it, or may consider it too speculative to comment on — the negotiations are interesting to consider as a possible alliance between horizontal and vertical search.
In discussing the concept, Coursey considers Wikipedia in tandem with
Entry point for mainstream media
Another interesting consideration is the fact that were reportedly among the other bidders.
Some reports saw this as the New York Times' strategy to get into the blogging world, characterizing About.com as analogous to a network of 500 bloggers. Doing the math this way, reviews were mixed as to whether the price paid ($820,000 per "blog") was possibly too steep.
The cost of doing business in this area was one issue debated at the recent Harvard conference on blogging and journalism (see archived story), as more traditional media consider how to react to the blogging phenomenon. The conference included a heated discussion involving Jill Abramson, managing editor of the Times, and Dave Winer over the importance of having a media organization with the resources to fund a bureau in Baghdad, as compared to a blog network.
At one point, Jimmy Wales chimed in to mention how Wikipedia had come up with a product that could compete with the Encyclopædia Britannica, with its multimillion-dollar budget, while spending a fraction of the amount. While a simple analogy might tie this to Wikinews as a developing project that can compete with other news organizations, the possibility also exists of news companies expanding in ways that compete with the Wikimedia Foundation's other projects. With the Times' purchase of About.com, Wikipedia may also be moving closer toward head-to-head competition with media organizations.