Premise

Yes, I'm going to be advising the Opposition on IT

As the Guardian has just reported, I've been asked to advise the Tories on IT policy, and I've accepted.

I'm posting here, on this mostly disused blog away from mySociety.org to make a couple of things very clear:

1. It is me that is doing the advising, not mySociety.  And mySociety is strictly non-partisan. This partisanship is guaranteed by a range of trustees, staff and volunteers of an almost ungovernable independence who would rather see me burned at a stake than change one line of code to benefit a political party. 

NB. This is not a paid position, and it isn't full time. I'll be continuing the business of running mySociety almost full time. 

2. I am not a political partisan - party politics bores me rather. I'm not a member of any political party, nor have I ever been. I've worked for the Institute of Economic Affairs, and I've worked for the Blair era Strategy Unit, as a civil servant. And yes, clearly everyone has beliefs about the world, and I'm no exception. But if you've met me or read anything I've written, you'll know my passions tend to get raised around things like Freedom of Information, open data, open source and open standards. A bit sad, maybe, but they're things that matter to me a great deal because I believe they can improve lives immeasurably, here and abroad. I also don't believe that they are an issue owned by any one political party: they're something we should all be striving towards.

I've advised and occasionally harassed the current  government on these things, including co-author the Power of Information review for the Cabinet Office.  I do my best to give advice to anyone who asks for it, and I have spent much of the last few years in Whitehall talking with anyone who'll ask, including plenty of ministers and senior civil servants. More recently I've been holding conversations with some wonderful new technology people in the new US Executive Branch: the government of my other parent country.

I have enjoyed advising this government, and I look forward to advising the next. And to harassing and prodding when advice isn't enough. And when there's another new government after that  I hope very much  to do the same with them, too. And so on, until they carry me out in a box.

Update 00.33 5th October 2009

Tom Watson has posted a criticism of my decision on his blog. I've left a comment that I expect him to moderate into visibility soon, but here it is for ease of access.

Hi Tom,

I’m genuinely sorry that the news probably isn’t what you wanted to hear. In particular it makes me sad to receive criticism from someone I’ve had such a good working and informal relationship with.

I would like to stress to your readers that mySociety will continue to be rigorously impartial, just as it was when I was advising ministers like you in the current government. Apart from anything else, I don’t have the power to make mySociety be anything else other than impartial – the staff and volunteers are way too independent minded to allow a dubious change in direction.

Lastly, on the timing of the announcement: I’m afraid I’ll have to go with your ‘at best naive’ analysis. I do not want to damage your party, and I hope you can see I have no interest in doing so.

all the best,

Tom

Government responds to the Power of Information Review

I'm happy to see that the Government has accepted most of the recommendations in the Power of Information review. You can see Sam Smith's annotatable version here.

I'll try to keep a close eye on what acceptance means in practice, and will hopefully be able to answer questions from the implementation teams. I may even start referring all the people who phone and email asking if they can get postcodes & boundaries & maps for their non-profit purposes towards the civil servants responsible for getting them more information ;)

The Power of Information Review launches

I'm delighted to announce that the review I've been working on with Ed Mayo and the Cabinet Office has launched today. You can get the official PDF version here or my friend Sam Smith's  annotatable version that he just threw together.

I'm immensely grateful to Fran, Steve, both James and Amalie who threw their work life balance out the window to get this done in so little time.

I'm really looking forward to what people have to say about it, and to seeing the government response which should be out in a few weeks.

Searching for examples of civil servants taking part in other people's sites

So, we've been busy interviewing everyone and anything that won't run away from us.  As usual, asking questions has produced even more questions.  And that's where I'm hoping some of my many (6) readers might come in.  Can you perhaps point us to:

1. Examples of places where civil servants or other public officials successfully engage in other people's user-created sites? (I'm loath to say in other people's discussions, because if they were making useful additions to wikis, that'd be fine too).
2. Examples of public servants who have been stopped from engaging for what seem like either good or bad reasons.
3. Examples, good or bad,  where governments gave either monetary or other support to non-government user-created websites.  What did they give? What did they get out of it?
4. Stand out, clear-as-daylight examples of Uncontroversially Good Things that happened when government officials did deign to contribute to other people's sites. And vice versa.

The feedback from people here and on the UK & Ireland DoWire list and on eMint has already been brilliant and perhaps more useful than they know. Please keep rack  your brains and/your inboxes/Google/your friends' brains to help with these questions - the better answers we get, the more we have the chance to shape the way that Government evolves to react to the user-generated era.

So, who should we talk to?

So here we are, week 2 and this review is just getting started, gantt charts and project plans all sorted.  The next 3-4 weeks present a great opportunity to go and talk to pretty much anyone who has an important role to play in the creation or re-use of public sector data of the non-private/personal kind.

So, the question is "Who?", and what should we be asking? This isn't a mega review that can fix an interview with every department and agency and company and charity in the whole sector and talk to them all in alphabetical order. Instead we need to hunt down examples of good practice and missed opportunities and ask people how they came about, and what's going to happen next. Remember, in particular we're looking for people who've built online communities which help people help each other, as well as public or private data providers and re-users who're mashing up everything that isn't nailed down.

It lives!

Those of you who have long since forgotten you ever had this blog in your RSS feed might be shocked to see it jerk back into life, zombie-like.

These days I tend to post over at the mySociety developers blog , but I have a good reason to come back here for a bit.

I have been asked by the Cabinet Office to help run a really interesting review on two topics close to my heart.

First, the way that the public sector produces and publishes public information, stuff like statistics, data, reports, league-tables etc and what it should be doing or not doing to make this more suitable for the Internet age.

Second, I'll also be looking at the sort of public information that people and organisations also create online, from Wikipedia to the annotations people leave on TheyWorkForYou. Again, the top level question will be whether there are things the government should or shouldn't be doing to help these fields thrive.

I'm posting on this rather dusty blog because this work isn't  being done in a mySociety capacity, and I want a home to talk about this project whilst it's under way. I don't expect many people will spontaniously stumble across this dicussion in the midst of many billions of more interesting pages on the Internet, so I'm planning to do quite a lot of my question asking out on the blogs and on the mailing lists and in the communities where people actually hang out.  But I thought it was important to have a home on the web where people can come and talk about aspects of the review if they feel like it.

I'll be taking about 2 days a week off mySociety to work on this, and it looks like I'll have some research help from the Strategy Unit.  I'm sure I'll work out how to fit in the other 6 days of mySociety work some how...

Not blinkered just yet

“Bone-chilling reading” wasn’t a term that got applied to many books about the internet at the time of the first dot com boom. But law professor Cass Sunstein’s 2001 book Republic.com was a work quite outside new economy boosterism and made for worrying reading. Dressed as a study of changing patterns of news media consumption, Republic.com drew a vivid picture of a future in which the cultural effects of digital technologies turned around and savaged the rational, open minded scientific tradition that had created them.

In Sunstein’s vision net users of the future are offered ever more customised media – endless football commentary for football fans, endless flyfishing news for fly-fishing fans. The immediate personal benefit to individuals is enormous, they never again have to spend time consuming news or entertainment of a sort that doesn’t interest them (no more Antiques Roadshow before the news!). But what seems like an unmitigated good , warned Sunstein, could permanently damage the fabric of democratic societies.

What happens when someone of moderately extreme views ceases to be exposed in any way to mainstream media, asks Sunstein? In fact, what happens when relatively moderate people, or those who are simple uninterested in politics stop consuming anything but their preferred sort of news? Once you start thinking about it, the dismal scenario more or less writes-itself: a world in which parental prejudices, instead of being challenged by exposure to new knowledge, are reinforced 100% by all news and entertainment that a child receives. Where people simply aren’t aware that other reasonable opinions on topics of all sorts even exist. The pattern is relentless – people indicate through their viewing or browsing habits that they are more interested in one sort of media coverage, their media adapts and never gives them anything else. They slowly become unaware that there is anything else, and when they finally meet people of differing opinions they can only assume that they are deranged, evil, or somehow non-human. And we all know what starts to happen when groups of people decide that other people aren’t people after all.

So goes the theory, and a persuasive one it is too. One of the most discussed flash animations on the web over the last year was EPIC 2014 a ‘future history’ describing the way in which personalised media combined with micro-publishing leads to a world where Google and Amazon more or less destroy the existing news media market. By 2014 it envisions a world where for the vast majority of users mass personalisation means a news media intake which is “narrow, shallow and sensational” and where much of what is reported is simply untrue.

In 2004 a pair of academics at the University of Michigan decided that it was time to test the theory of internet media consumption leading to a more narrowed and blinkered view of politics. What Paul Resnick and Kelly Garrett found was not expected.

By comparing a weighted sample of the general public against a similarly weighted sample of internet users, they tested the hypothesis that voters at the 2004 US General election would choose to learn more positive facts about the candidate they supported than they would learn about their opponent.  The conclusions were robust:

“At a time when political deliberation seems extremely partisan and when people may be tempted to ignore arguments at odds with their views, internet users are not insulating themselves in information echo chambers. Instead, they are exposed to more political arguments than non-users.”

It is this final observation that seems the most surprising – can it really be that more internet users who can select from any news media they want prefer not to consume media that supports their points of view? Apparently so: 28% of non internet users expressed a preference for news sources that explicitly shared their political point of view, only 18% of broadband internet users said the same thing. Furthermore in his dissertation Kelly Garrett found that whilst news consumers were slightly less likely to read stories that appeared to contradict their own views, once they started reading them they would read them for longer the more they disagreed with the opinions in the articles. This is the Howard Stern phenomena online – famously people who hated Stern listened to him more hours per week than people who didn’t mind him.

This may seem surprising or even impossible to anyone who has seen analysis of political blogs over the last couple of years, where maps have been drawn of the incestuousness of the US political blogging community. They show two huge clusters of dots, one red, one blue. The blue cluster is made up of hundreds of blue Democratic sites linking primarily to other Democratic sites and the red cluster of similar activity amongst Republican blogs. 

One study by US researchers Lada Adamic and Natalie Glance  discovered that 91% of links on political blogs they studied contained links to other blogs within the same political community. Looking exclusively at the top 20 most read political blogs from each side of the partisan aisle, they also found that only 15% of posts were ‘cross citations’ to blogs of different political persuasions.

But it is precisely in these small numbers of cross-citations that we can understand the flaw in Sunstein’s vision, and the root of the unexpected poll findings by the Michigan academics. The 15% of cross citations represent precisely the open-ness, and the dependence on opposition opinions that fuel the political blogging community, and news media online generally. It is very hard to argue into silence, and political rants are always easier to build on the rejection of an opponent’s statements or actions. Political blogs may link primarily to their own kind, but they still provide thousands of instantaneous links to opposing views – this is the crucial difference between the internet and books or newspapers. A newspaper, magazine or television channel may attempt to be neutral and contain diverse opinions, but ultimately it cannot provide the forbidden thrill of infuriating, genuine opposition, unedited and unfiltered, just a click away.

In a  study of the British political landscape I co-authored with Chris Lightfoot  earlier this year, one of our observations was that Guardian readers were well over 98% were to the left of center on our axes. There is clearly strong political self selection even in the relatively highly educated readers of this newspaper – even amongst an audience that knows value of balance and neutrality. Self selection of news consumption will continue online and offline, presumably so long as people have political opinions. Ultimately it is perhaps that oldest of sentiments, the car-crash mentality, the desire to see dangerous, ghastly things that secretly thrill us that will save us from the nightmare vision of Cass Sunstein’s Republic.com.

 

Trial of the Century

I have pulled myself out of blog purdah to tell the world about the most most important court case of the 21st century. It hasn't happened yet, and the the docket won't be filed for a few years. But when it goes to the Supreme Court, as it inevitibly will, the Justices will hold the very nature of politics in their hands.

It all starts with the humble video recorder. In the early 1980s the film and television industry lined up against home video recording, claiming that home taping would destroy great swathes of their professions.  Sony, who had an interest in selling these new wonder devices ended up fighting these industries for the right to manufacture videos with 'record' buttons. It went all the way to the supreme court, and Sony won, paving the way for videos to become as common as sinks in homes across the developed world.  Cory Doctorow's version of this story  is much better than mine.

Two decades later, issues of recording and sharing are back on the agenda, with P2P the new tattooed poster child. But this court case isn't going to be about P2P. It's going to be about Tivo, and the genre of new device it spawned: the Personal Video Recorder, or PVR.

PVRs are devices which sit on top of your TV and do lots of wonderful things, all based on their ability to record TV and play it back. They let you pause live TV, and then unpause it when you get back from the loo. They make it easy to record  programmes you might be interested in, and some even guess what you want to watch.  All have one notable effect - people who own them gradually watch less and less live TV. Why bother when you've a personal archive of things you'll probably enjoy watching  just sitting there waiting for you at the end of a long day? And best of all, just as with a video recorder, you can skip the adverts!

And it is these ads that will be at the heart of the court case. When you watch all your TV pre-recorded, why would you sit through the adverts, when you can skip them? The answer is, "you don't" - people overwhelmingly skip through.  For advertisers, this spells disaster. And everytime someone buys a new PVR, the number of adverts being watched falls just that little bit more.

The advertising industry isn't taking this lying down, of course.  It has insinuated that people who skip adverts are actually stealing the shows  they help pay for. And when accusations like that are being made, you can be pretty sure the court cases are soon to follow.

There is big money on either side, and sometime in the next few years, it'll end up in the supreme court, as it did with Sony in the 80s. The issue will be simple - should it be permissable to build devices that let users skip adverts, or should they be designed so as to force users to watch at least a few. Either option is possible - it's a pure question of who wins.

So what, you ask? This is at best a consumer protection issue isn't it? This isn't case of the century, surely?

Well, yes, until you remember one thing - the US presidential canditates spent about half a billion dollars on TV advertising at the last election.  TV is the default medium and motivator of US politics. It protects incumbents, raises high barriers to entry, destroys weak candidates, reaches the most people, and is the reason US politics is such a big money game. And this court case will decide whether a significant proportion of the electorate ever watches another political advert.

Nuff said.

Institute of Community Studies

The Institute of Community Studies is a venerable organisation, quietly responsible for a plethora of British institutions like the Open University and the Consumers Association. A couple of months ago it was given a huge injection of new life with the appointment of Geoff Mulgan as it's new head.

I suggested immediately after his appointment that a powerful tool to help it along might be a public WiKi. By acting like a communal notebook, the many exciting new projects in genesis can be discussed and evolved, all in a transparent, easy to access fashion. So, with the help of some free hosting from Mythic Beasts, Geoff's been busy writing the core material to get the site started.

The upshot is this - today I'm very pleased to present you with the ICS WiKi. Please dive in, read and get involved - it all starts today.

Still learning from Dean

As the election draws near more and more people are asking me for my advice on using the internet to campaign. My standard answer 'Spend it on TV and print' doesn't seem to be going down too well, so I thought I'd use this as an opportunity to reflect on the further lessons from Howard Dean's experience of the first political dot.bomb.

This has all been inspired by W.B Yeats. Recently, Anno Mitchell reminded me of his famous words about the awkward relationship between ideological fervour and talent, "The best lack all conviction, while the worst/Are full of passionate intensity". The problem now is that the net gives the 'passionate intensity' crowd the ability to organise and evangelise remarkably successfully, wheras previously the down side of that package (being 'the worst') would have denied them access to the mainstream. What we owe Dean is that fact that this is a phenomenon that we'll be better attuned to detecting next time round.

I do have three bits of advice for campaigners:

1) It's great for organising your troops - work really hard on using it to get the information they need out to them. Let them at your rebuttal machinery. Spend most of your e-budget on this.

2) For the mainstream parties and their outward facing personas, the net still offers depressingly little beyond the chance to sell you message in a highly customised fashion to those people who show up on your site in the first place, most of whom will have already made up their minds. But these systems are worth building anyway, just to catch the odd dilligent floating voter who's sick of soundbites and who wants to know what you actually stand for.

3) Here's my optimisitic bit. Any contest that can be won by a constituency base which is thin but spread over a very large geographic area can be won with the help of the net. This means you can win a funding race, or a non-controversial legal campaign, or any race where it's you vs the status quo (as opposed to between you vs the rest of the country in polling booths). The problem for the next non-proportionally-representative general election is that geograhical density of votes matter, and the net can't deliver those, at least not at the moment.

So, what are you waiting for? Start raising money and start buying newsprint ;)