read all about it discover all the news

Over the last few weeks, a few high-profile accounts on Identi.ca, like @methoddan and @jezra, were inexplicably locked out of the system. I'm currently analyzing the possible causes, but I wanted to give a general idea of what's going on.

Here are my current suspected causes:

1. Incorrect lockout by a community moderator. We have about 40 community members who have the ability to "silence" another user -- preventing them from logging in or posting. These users have shown themselves to be responsible by doing effective spam reporting in the past, and granting them these limited privileges has been a very effective anti-spam measure. It is possible, but not likely, that one or more has either accidentally or mischievously silenced a "good" identi.ca member.

   To identify if this is a problem, I'm working on audit tools to figure out by whom, and for what reason, an account was silenced. These should go live soon.
    

2. Mass-silencing by script. In late December 2011, we had a concerted and wide-spread attack on Identi.ca that brought down the system for hours at a time. The attack consisted of posts by somewhere between 5000 and 7000 registered accounts, posting randomly-generated text several times a second. The accounts were posting from thousands of different IP addresses.

   The only commonality was a similarity in registration period and posted content -- the posts were based on templates like "I just <verb>ed with <name> at <place>." I was able to isolate some unique patterns, and any user who registered in the particular date range and who had posted a notice with a particular text pattern was silenced automatically. About 5000 users were silenced using this method. It's possible that some "innocent" users would be blocked here, but more "senior" users would not fit in the date range.
    

3. Automated silencing. During this same period, to catch newly-registered users and users that had not yet posted, I added a posting filter that would look for posting patterns that matched the same text production. The new notices were blocked and the posting account was silenced. These automated silencing tools did not check for user registration date range, however.

   The text patterns I used were pretty fine-tuned and, in particular, were clumsy and grammatically incorrect English. But it's possible that they were what triggered the silencing of some users. I think that this is the most likely reason that experienced Identi.ca users were silenced.

   I've disabled these filters for now, although I may in the future re-enable them if we have a similar attack.

I personally apologize to the users who've had this problem; we've erred on the side of spam prevention a little too harshly this time around, and it caught people who didn't deserve it. It's not pleasant to be told you're not welcome as part of a community, and since there's no indication of what to do about being banned in our ban messages, it can feel brusque and unfair.

Compounding this was a change in mail server during the same period, which meant that some requests for help weren't delivered. Not helpful, either!

Again, my apologies to Dan and Jezra and anyone else who saw this problem.

 Our upgrade to StatusNet 1.0.0 in September 2011 had some unfortunate side-effects. Some of the public services we ran were taken off-line after the upgrade... and never returned.

One of the most beloved, freelish.us, the Open Source, Open Content bookmark-sharing service, has been requested (emphatically) by many users. Most of the functionality has been subsumed by the rest of StatusNet, but we wanted to have it back up anyway. As of this past weekend, the server is back on-line.

Thanks to everyone for your patience and confidence. I'm looking forward to see where the freelish.us community goes.

Identi.ca will join Reddit, the Cheezburger Network, and dozens of other web sites in protest of the Stop Online Privacy Act proposed in the US Congress and its corresponding Senate bill, Protect IP. We will be blacking out the service on Wednesday, January 18th 2012 from 8AM to 8PM EST. I want to explain why.

First, and most importantly, Identi.ca users have asked for it. A poll on Identi.ca had a 90% favorable response rate for a blackout.

Second, because our company is based on an open Web and an open Internet. We want to make the social web look more like the Internet -- distributed, hierarchical, and participative. It's a journey, and we want the Internet to still be there when we reach our goals.

And third, because SOPA/PIPA will directly affect community-generated sites like Identi.ca.

If you're looking for more information on the SOPA/PIPA bills, and how they affect you whether or not you live in the USA, I recommend reviewing americancensorship.org. And I hope that if you run a Web service or publishing platform, you'll join us in this blackout.

We'll have an off-site discussion at identicablackout.status.net during the blackout.

There is a new proposed community group for PubSubHubbub at the W3C. 

http://www.w3.org/community/groups/proposed/#pubsub

PubSubHubbub ("PuSH") is one of the key technologies for an open, real-time Web.

It uses the Webhooks pattern to let subscribers register interest in an Atom feed, and to let publishers distribute updates at time of publication, rather than at a later poll time.

(If you think that's not a big deal, I highly recommend Evan Henshaw-Plath and Kellan Elliott-McCrea's great talk at OSCON 2008. They discuss the problems with polling many feeds at scale, and propose a pub-sub solution, albeit XMPP-based.)

The specification for PubSubHubbub was originally developed by Google engineers Brad Fitzpatrick and Brett Slatkin (cc'd). They have a good video that goes over what the protocol does and why they developed it. There's also a demo from the 2009 Real-time Crunchup with similar info.

In the years since its first release, the PubSubHubbub has been remarkably widely-implemented -- on blogs and in feed-reader applications. New usage has turned up some new requirements. Among other things, defining how the protocol works for non-Atom data types (like Activity Streams JSON or even binary types), and defining how to limit distribution to particular individuals.

Brad and Brett have, I understand, given their support to a new community group. The original IP is licensed under the OWFa agreement, which I believe is compatible with W3C community groups.

I am interested because PubSubHubbub is the key component to OStatus. status.net and Identi.ca together host almost a million PubSubHubbub-enabled feeds. I hope other people and organizations interested in the federated social web can support this new group, too.

 Identi.ca is currently down; I just got off an airplane. We had a flood of new users and posters over the last 24 hours, and the extra traffic seems to have sunk our database server.

I'm trying to revive the server (it's currently re-starting), but I might take the opportunity to move to our new data centre. If we're down, maybe it's time to do the move. Watch this space for details.

Update 14:50 PM PST: I can't seem to revive the main database server, so I'm copying data to another server in the same data centre to see if it wants to wake up.

Update 23:40 PM PST: The main and backup database servers are down, so I've started the process of moving to the new data centre. It should take about 10-12 more hours.

Update 01:30 AM PST: The import from backup is going nicely. It looks like we're going to lose a few hours of posts from Monday night, but I might be able to recover them from the old servers. I've consequently changed the DNS for identi.ca to point to the new data centre; that's going to take a few hours to propagate, maybe up to 24 hours.

Update 03:40 AM PST: The import stopped because I did something dumb (had the database dump uncompressed on the same drive as /var/lib/mysql); I need to clean up the problem, and import the rest of the data. Good news is that it's not too difficult to do; bad news is that it will take some time to get everything moved around again.

Update 09:50 AM PST: Looks like I'm getting the last of the data imported (needed to move hundreds of gigs of data from one server to another, munge it with Perl, then move part of it back again). If things go well, we should see identi.ca back up within a few hours. DNS is still propagating for the new data centre, though; some people might see the old "server down" message for a while.

Update 17:00 PST: We've been back up for a few hours. DNS is still propagating; it may take a while for things to get out. Those of you who are comfy editing /etc/hosts may want to map "identi.ca" to 50.57.205.237 there. Otherwise, things will gradually come online soon.

 Identi.ca is currently down; I just got off an airplane. We had a flood of new users and posters over the last 24 hours, and the extra traffic seems to have sunk our database server.

I'm trying to revive the server (it's currently re-starting), but I might take the opportunity to move to our new data centre. If we're down, maybe it's time to do the move. Watch this space for details.

Update: The DNS TTL on identi.ca is far too high for this to work tonight. I'm going to dial it down and hopefully move next week.

 I'll be moving the main servers for Identi.ca to our new data centre in the Rackspace Cloud. This is going to require a lot of data copying, so there's going to be some downtime while it moves. I expect somewhere between 4 and 12 hours.

There'll be a new blog post up with status updates on Thursday.
 

 I'll be moving the main servers for Identi.ca to our new data centre in the Rackspace Cloud. This is going to require a lot of data copying, so there's going to be some downtime while it moves. I expect somewhere between 4 and 12 hours.

There'll be a new blog post up with status updates on Thursday.
 

We had about 6 hours of unscheduled downtime tonight due to a bad slave server.

The site is back up now. Sorry for the disruption.