Details of every phone call and text message, email traffic and websites visited online are to be stored in a series of vast databases under new Government anti-terror plans.
Landline and mobile phone companies and broadband providers will be ordered to store the data for a year and make it available to the security services under the scheme.
The databases would not record the contents of calls, texts or emails but the numbers or email addresses of who they are sent and received by.
For the first time, the security services will have widespread access to information about who has been communicating with each other on social networking sites such as Facebook.
Direct messages between subscribers to websites such as Twitter would also be stored, as well as communications between players in online video games.
The Home Office is understood to have begun negotiations with internet companies in the last two months over the plan, which could be officially announced as early as May.
It is certain to cause controversy over civil liberties – but also raise concerns over the security of the records.
Access to such information would be highly prized by hackers and could be exploited to send spam email and texts. Details of which websites people visit could also be exploited for commercial gain.
The plan has been drawn up on the advice of MI5, the home security service, MI6, which operates abroad, and GCHQ, the Government’s “listening post” responsible for monitoring communications.
Rather than the Government holding the information centrally, companies including BT, Sky, Virgin Media, Vodafone and O2 would have to keep the records themselves.
Under the scheme the security services would be granted “real time” access to phone and internet records of people they want to put under surveillance, as well as the ability to reconstruct their movements through the information stored in the databases.
The system would track “who, when and where” of each message, allowing extremely close surveillance.
Mobile phone records of calls and texts show within yards where a call was made or a message was sent, while emails and internet browsing histories can be matched to a computer’s “IP address”, which can be used to locate where it was sent.
The scheme is a revised version of a plan drawn up by the Labour government which would have created a central database of all the information.
The idea of a central database was later dropped in favour of a scheme requiring communications providers to store the details at the taxpayers’ expense.
But the whole idea was cancelled amid severe criticisms of the number of public bodies which could access the data, which as well as the security services, included local councils and quangos, totalling 653 public sector organisations.
Labour shelved the project – known as the Intercept Modernisation Programme – in November 2009 after a consultation showed it had little public support.
Only one third of respondents backed the plan and half said they feared the scheme lacked safeguards and technical rigour to protect highly sensitive information.
At the same time the Conservatives criticised Labour’s “reckless” record on privacy.
A called Reversing the Rise of the Surveillance State by Dominic Grieve, then shadow home secretary and now Attorney General, published in 2009, said a Tory government would collect fewer personal details which would be held by “specific authorities on a need-to-know basis only”.
But the security services have now won a battle to have the scheme revived because of their concern over the ability of terrorists to avoid conventional surveillance through modern technology.
They can make use of phone tapping but their ability to monitor email traffic and text messages is limited.
They are known to have lobbied Theresa May, the Home Secretary, strongly for the scheme. Their move comes ahead of the London Olympics, which they fear will be a major target for terror attacks, and amid a climate of concern about terrorists’ use of the internet.
It has been highlighted by a number of attacks carried out after radicalisation took place through websites, including the stabbing by a young Muslim woman of an MP at his constituency surgery.
Sources said ministers are planning to allocate legislative time to the new spy programme, called the Communications Capabilities Development Programme (CCDP), in the Queen’s Speech in May.
But last night privacy campaigners warned the scheme was too open to abuse and could be used for “fishing trips” by spies.
Jim Killock, executive director of the Open Rights Group, a civil liberties campaign organisation, said: “This would be a systematic effort to spy on all of our digital communications.
“The Conservatives and Liberal Democrats started their government with a big pledge to roll back the surveillance state.
“No state in history has been able to gather the level of information proposed – it’s a way of collecting everything about who we talk to just in case something turns up.”
There were also concerns about the ability of phone and internet companies to keep the information secure.
And the huge databases could also be used by internet service providers, particularly to work out which advertising to target at users.
Broadband firms including BT came up with a scheme almost three years ago to target advertising, but it did not get off the ground.
However, if companies were able to exploit the information they will be compelled to keep for the CCDP, they would be much more capable of delivering advertising to computers and even mobile phones based on users’ past behaviour.
Gus Hosein, of Privacy International, said: “This will be ripe for hacking. Every hacker, every malicious threat, every foreign government is going to want access to this.
“And if communications providers have a government mandate to start collecting this information they will be incredibly tempted to start monitoring this data themselves so they can compete with Google and Facebook.”
He added: “The internet companies will be told to store who you are friends with and interact with. While this may appear innocuous it requires the active interception of every single communication you make, and this has never been done in a democratic society.”
A Home Office spokesman said: “It is vital that police and security services are able to obtain communications data in certain circumstances to investigate serious crime and terrorism and to protect the public.
“We meet regularly with the communications industry to ensure that capability is maintained without interfering with the public’s right to privacy.
“As set out in the Strategic Defence and Security Review we will legislate as soon as Parliamentary time allows to ensure that the use of communications data is compatible with the Government’s approach to civil liberties.”
Andrew Kernahan of the Internet Service Providers’ Association said: “It is important that proposals to update Government’s capabilities to intercept and retain communications data in the new communications environment are proportionate, respect freedom of expression and the privacy of users, and are widely consulted upon in an open and transparent manner.”