Public procurement conditions for trustworthy AI and algorithmic systems
Governments increasingly use AI and algorithmic systems. The City of Amsterdam for example uses algorithmic systems in some of their primary city tasks, like parking control and acting on notifications from citizens. In the last couple of years, many guidelines and frameworks have been published on algorithmic accountability. A notable example is the Ethics guidelines […]
by Linda van de Fliert
Governments increasingly use AI and algorithmic systems. The City of Amsterdam for example uses algorithmic systems in some of their primary city tasks, like parking control and acting on notifications from citizens.
In the last couple of years, many guidelines and frameworks have been published on algorithmic accountability. A notable example is the Ethics guidelines for trustworthy AI by the High-Level Expert Group on AI advising the European Commission, but there are many others. What all these frameworks have in common is that they name transparency as a key principle of trustworthy AI and algorithmic systems. But what does that mean, in practice? We can talk about the concept of transparency, but how can it actually be operationalized?
That’s why the City of Amsterdam took the initiative to translate the frameworks and guidelines into a practical instrument: contractual clauses for the procurement of algorithmic systems. In this post, you can read all about these procurement conditions and how you can be involved in taking them to the next level: a European standard for public procurement of AI and algorithmic systems.
Terms & Conditions
Starting late 2019, the City of Amsterdam joined forces with several Dutch and international experts, ranging from legal and procurement experts to suppliers and developers, resulting in a version 1.0 of a new set of procurement conditions, and an accompanying explanatory guide. This first iteration is already free for all to use and adapt.
By choosing procurement conditions as a means to operationalize the ethics and accountability frameworks, we hit two birds with one stone. First of all, it provides clear guidance to suppliers, who according to the World Economic Forum, “understand the challenges of algorithmic accountability for governments, but look to governments to create clarity and predictability about how to manage risks of AI, starting in the procurement process.” Secondly, and maybe more importantly, procurement conditions demand clear definitions, both of key concepts like ‘algorithmic system’ and ‘transparency’, as of the conditions themselves.
Transparency in practice
Although the procurement conditions aim to tackle several issues related to the procurement of algorithmic systems, like vendor lock-in, the main novelty is that they provide a separation between information needed for algorithmic accountability on the one hand and company-sensitive information on the other. The conditions distinguish between three main types of transparency that the supplier should provide:
Technical transparency provides information about the technical inner workings of the algorithmic system; for instance the underpinning source code. For many companies, this type of information is proprietary and often considered a trade secret, it’s their ‘secret sauce’. Therefore, unless it is the procurement of open source software, technical transparency will only be demanded in case of an audit or if needed for explainability (see below).
Procedural transparency provides information about the purpose of the algorithmic system, the process followed in the development and application and the data used in that context; for instance, what measures were taken to mitigate data biases. Procedural transparency provides a government with information that enables them to objectively establish the quality and risks of the algorithms used and perform other controls; to provide explainability (see below); and to inform the general public about algorithmic usage and the manifold ways on how it affects society. Procedural transparency is mandatory in every procurement.
Explainability means that a government should be able to explain to individual citizens how an algorithm arrives at a certain decision or outcome that affects that citizen. The information provided should offer the citizen the opportunity to object to the decision, and if necessary follow legal proceedings. This should in any event include a clear indication of the leading factors (including data inputs) that have led the algorithmic system to this particular result and the changes to the input that must be made in order to arrive at a different conclusion. Providing this information becomes mandatory for any relevant product or service procured by the city under the new rules.
The procurement conditions and their explanatory guide give a detailed account of the situations in which each of these types of transparency applies.
Towards a European standard for public procurement of AI and algorithmic systems
The ambition for this project has always been to show that it is possible to operationalize general guidelines for AI ethics and to encourage others to do so as well. That’s why we hope these conditions will become the inspiration for a European standard for public procurement of AI and algorithmic systems. We took some steps towards that ambition already:
From February 2020 to June 2020, the European Commission held a public consultation on their AI white paper. The City of Amsterdam and Nesta, together with the Mozilla Foundation, AI Now Institute and the City of Helsinki, published a position paper as a response to that consultation, asking the EC to facilitate the development of common European standards and requirements for the public procurement of algorithmic systems.
Within the Netherlands, the conditions are now being implemented by several municipalities, regional governments and government agencies, collecting feedback from suppliers and working towards a version 2.0.
On June 25th, DG GROW hosts a webinar, titled Public Procurement of AI: building trust for citizens and business. At this webinar we will launch a more generalized version of the procurement conditions, that can be easily adapted to fit your organization. In the meantime, click the links to download the procurement conditions and their explanatory guide in pdf, to use within your organization right now.
Can’t wait? Want to help? Or just want to stay informed? Please let us know through this form how you want to be involved!
How video games are becoming the next frontier in the ‘Tech Cold War’
The 'platformisation' of the games industry is posing some serious challenges for Europe and the internet at large.
What the researchers identified are two mutually reinforcing trends that blur the lines between certain online games and traditional platforms: by curating in-game events, adding social-media-like features and enabling increasingly sophisticated player interaction, games have the potential to become platforms in all but name, giving developers and third parties an engaging, new channel for the delivery of paid content and services, which can range from pop music concerts and movie trailer premieres to political campaigns.
What makes these trends more concerning is that the global gaming industry is exhibiting the tell-tale signs of ‘platformisation’ even at the macro level. Having experienced a period of democratisation and significant growth on the production side in the late 2000s and early 2010s – consider, for example, the advent of app stores and the renaissance of indie games – we are today seeing a period of heavy consolidation and centralisation of market power. And just as in other segments of the tech and creative industries, the new gatekeepers of gaming are engaged in winner-takes-all battles for attention, data, monetisation and intellectual property.
Why Europe is losing out
Widely recognised as one of the world’s fastest-growing industries, some estimates see the gaming sector turning over as much as $300 billion by 2025. Already today, games significantly outpace the global film and music industries. While the EU is a major consumer market for games, with revenues in excess of €21 billion in 2019 alone, it lacks the corporate heavyweights that dominate the industry in Asia and North America. As in other segments of the technology sector and creative industries, Europe boasts a rich tapestry of world-class developers and innovators but is home to few of the major studios or publishers and, at best, plays a supporting role in the development of gaming hardware, services and infrastructure. With the loss of the UK’s exceptionally strong gaming sector – which gave birth to Tomb Raider and Grand Theft Auto – to Brexit, it’s fair to say that Europe risks once again falling behind the big and, in China’s case, emerging players – a familiar refrain in the Tech Cold War.
Making Europe competitive in gaming will require greater support and smarter, forward-thinking regulation at the transnational level. Until relatively recently, the politics and regulation of video games were largely under the purview of national governments. Like many other areas of cultural and media policy, EU Member States tend to treat video games as a national competence. Often that means that countries have to go it alone when they feel the need to regulate, as Belgium did with its recent ban on loot boxes in games. But as online gaming and digital distribution are becoming the norm, it’s no longer possible to ignore the medium’s borderless nature and geopolitical relevance. Brussels needs to be prepared to deal with the looming challenges of the industry.
Through the technology glass
One solution is to look at gaming through the prism of platforms, technology and data policy, rather than just media and creative industries policy. This makes sense for several reasons. Firstly, on topics like Europe’s ‘digital sovereignty’ or the future of AI, the institutions in Brussels have finally come to terms with the idea that digital, competition and foreign policy are inextricably intertwined. As with data governance or social media regulation, it makes sense to view video games in the same context of Europe’s systemic competition with the Chinese and U.S. digital economies.
Secondly, large swathes of today’s gaming industry are owned, controlled or gate-kept by a small number of dominant and data-hungry technology companies, many of which are U.S. or China-based. That is a notable change from the early days of gaming when the industry was shaken up by garage start-ups, medium-sized toymakers, slot machine operators and manufacturers of HiFi equipment.
Lastly, gaming is plagued by many of the same transnational issues that we’re dealing with in technology and data policy. The gaming sector, too, struggles to contain the power of platforms, ensure fair competition, curtail the amplification of harmful content and champion data protection. Its concerns, too, include the manipulation of online marketplaces, foreign takeovers and the security and safety of products and services.
A ‘platformer’ as a platform is a platform
As the University of Amsterdam paper shows, a small sub-segment of games can – and probably should – be considered content delivery platforms. Sticking with their example, Fortnite is not so much a game in the traditional sense as it is an adaptable infrastructure that allows its developer Epic Games to deliver content and services, including advertising and product placements, to players in a highly engaging and immersive way.
Trying to target Fortnite with ex-post regulation in 2021 would be missing the point. The game has been around for over three years, a lifetime in a fast-moving industry. It’s also just one highly-visible example of symptoms that affect an increasingly ‘platformised’ and politicised industry. Take PlayerUnknown’s Battlegrounds (PUBG), a popular South Korean eSports title that goes heavy on microtransactions and has been downloaded a respectable 800 million times.
Whether or not they would accept their classification as platform providers, it’s fair to say that the better-resourced publishers and gaming service providers have become more mindful of their responsibilities when it comes to ‘traditional’ online harms, particularly safeguarding minors. The rallying cry of “protect the children” – whether that’s from gratuitous violence, too much screen time or online grooming – has been a depressing constant in the politics of video games for decades, even if the evidence base often remains shaky.
But as gamers get older – the average age of video game players in the EU is 31 years – and the industry finds itself at the centre of geopolitical competition, other ‘online harms’ are likely to come into focus. In 2019, Reporters without Borders released the Uncensored Library, essentially a Minecraft server granting in-game access to banned journalistic articles in an attempt to evade internet censorship in countries where Western social media channels were banned. Although laudable on its own terms, the project highlights how video games can become vectors and catalysts for political speech and even propaganda, a complex phenomenon that deserves a differentiated policy response.
Concerns over radicalisation loom especially large. At least since the Gamergate controversy of 2014, there is an implicit assumption that gaming subcultures skew towards digitally-native, hyper-engaged adolescent males with extreme views, a combination of characteristics often targeted by Russia’s Internet Research Agency and other state-sponsored troll farms. On the whole, that characterisation doesn’t hold true. Gamers are a more diverse and representative crowd than we give them credit for, and the stigmatisation of players as violent, at-risk individuals or misogynist shut-ins is more counterproductive than helpful when trying to identify or address the issue.
Data flows and foreign takeovers present another contentious issue worth examining in this regard. Online games, and mobile games, in particular, are becoming an increasingly important source and beneficiary of data harvesting. As state or state-owned actors are beginning to invest in video games on a large scale, their ties to the industry are inevitably going to raise questions about the downstream use and potential abuse of gaming data. It’s easy to see how an increasingly state-sponsored gaming landscape could have a similarly destabilising effect on public trust as the arrival of Russian TV and Chinese tabloids had on the Western media ecosystem in the 2010s.
Indeed, the biggest area of concern seems to be China’s meteoric rise in the games industry, which makes as much sense economically as it does in terms of strategic data access. With investments in over 300 gaming companies, Tencent has rapidly become the world’s biggest video game publisher. Allegations of data-sharing between the tech giant and the Chinese government have already been the subject of occasionalcriticism, but its stakes in gaming companies with significant data assets, including Fortnite developer Epic Games and eSports giant Riot Games, are likely to receive more scrutiny going forward.
Whether data is genuinely at risk in these cases may almost be beside the point. If Europe wants to rekindle the public’s trust in data-sharing and the digital economy, its regulators and policymakers will have to become much better at anticipating, understanding and addressing data and takeovers issues in the games industry.
Playing to win
These problems extend beyond games that function like platforms themselves. Even ‘offline’ titles or online games that don’t quite fit the description of ‘quasi-platform’ tend to be inextricably linked to services that do. Plug-and-play is a thing of the past. In today’s video game economy, players have to interact with external platform providers that distribute games, enable access to additional content, track and broadcast their achievements, connect them to other players across the world and allow eSports enthusiasts to cheer for their favourite pro gamers.
Fortnite’s success, for example, is enabled by a platform-powered ecosystem that includes, but is not limited to, the developer’s own Epic Games Store, Twitch, Steam, YouTube, Playstation Network, Microsoft’s Xbox Live and Store, and many others. Pending a European antitrust complaint as well as several lawsuits, the iOS App Store and Google Play Store may or may not be added back to that list eventually. Last summer, both Apple and Google pulled Fortnite for breaching store policies when Epic tried to circumvent their in-app purchasing systems, which funnel 30 cents on every dollar made to Cupertino and Mountain View respectively.
Zooming out to the macroeconomic level, the Epic feud becomes just one of the many battles over platformisation, centralisation and anti-competitive practices that are set to define the next decade in gaming.
The effect of platform economics on games is equally obvious in the context of more open systems like the PC. Digital distribution is well-established and largely driven by bonafide platforms like Valve’s Steam store. It has cut out most of the middlemen and almost completely collapsed the second-hand economy. With packaging, discs, transportation, logistics and brick-and-mortar retailers out of the equation, publishers are seeing more money for their product and consumers get instant access to software from the comfort of their own home. Controversially, however, Steam – operated by a company that only employs around 360 people – takes a 30 per cent cut on every game sold through its platform. Much like Apple and Google, it has become a gatekeeper and quasi-essential infrastructure for PC gamers.
The list of grievances associated with Steam, and digital distribution more generally, reads eerily familiar to platform critics everywhere: asymmetrical contractual agreements with developers and publishers, unfair trading practices, data mining, targeted advertising, fake reviews and intransparent search algorithms that often dictate whether small-time developers get any consumer exposure at all. But 17 years into its existence, the Steam model is unlikely to change. Policymakers should focus on what’s next.
The next big thing
Among the handful of remaining players in digital distribution on PC, a familiar winner-takes-all mentality has taken hold. Would-be competitors need serious financial heft. Perhaps it’s therefore not surprising that Steam’s most serious challengers are backed by some of the world’s most valuable companies: Tencent is going head-to-head, while Microsoft, Apple, Google and Facebook are all looking to disrupt the digital distribution model in their own ways.
Europe, as in most other areas of the tech industry, sees itself relegated to the roles of consumer and supporting act. GOG, part of Poland’s CDProjekt Group, provides gamers with a relatively traditional store experience and boasts some laudable principles, such as integration of competitor platforms, DRM-free ownership of software and fairer treatment of developers, but it has so far struggled financially.
Tencent’s bid to corner the market comes courtesy of the Epic Games Store which, boosted by a cash injection from the tech giant and soaring Fortnite revenues, launched in late 2018. Intent on carving out a significant piece of the market before it’s too late, the service adopted an aggressive strategy: to lure in potential customers, it has given out at least one free game every week since launch – totalling more than 749 million giveaways in 2020 alone. In addition, Epic has signed a host of expensive exclusivity deals that prevent other distribution platforms from selling popular titles.
Across the Atlantic, perhaps the most serious attempt at shaking up the gaming market comes from Microsoft. Redwood pursues a more ambitious and novel business model than Epic, but at its core, it employs a similarly predatory pricing strategy. By moving its own game catalogue and dozens of licensed titles to the Xbox Game Pass, Microsoft combines a heavily subsidised, monthly subscription model with an opaquely curated selection of games. It also integrates the offering with its Microsoft Store, Xbox Live network and xCloud on-demand gaming service. Not content with limiting its ambitions to just one hardware base, Microsoft provides the service to Xbox consoles, PC and mobile devices, all of which can be covered with a single subscription. If Fortnite is a quasi-platform, Xbox Game Pass is designed to become a hyper-platform, and its strategy raises questions for consumer choice, competition and privacy.
Whoever emerges victorious from the war over digital distribution, both consumers and innovators will likely suffer in the long term. Players may at first rejoice at the idea of a weekly giveaway or a ‘Netflix for games’, but will eventually find themselves trapped in yet another walled garden. Developers and creatives, in turn, may hope to strike gold through greater and more targeted exposure on a highly centralised platform, but they too will find themselves at the whim of largely unaccountable and self-interested gatekeepers. Smaller competitors will struggle to gain traction or survive, as aggressive pricing strategies will always favour the giants, whose access to consumer data and endless lines of credit enables them to take and hedge long-term risks.
What’s left to play for?
After more than a decade of platform economics, the dynamics shaping today’s gaming industry are easy enough to spot. Their consequences may not always be predictable, but on balance they are likely to perpetuate the the same inequalities that we observe in the digital economy at large, further centralising power and profits in the hands of fewer market actors.
The stakes in this new theatre of the ‘Tech Cold War’ are high and, as in other sectors of the digital economy, Europe is at risk of not just losing out economically. In gaming, it could lose in a race for soft power at home and abroad. An overly passive Europe risks becoming a rule-taker, rather than a standard-setter; a captive consumer, rather than an innovator and market-shaper; and, in the parlance of privacy, a data subject, rather than a data controller. Not every excess of the industry will require disruptive, top-down regulation from Brussels. But policymakers across Europe would do well to spend more time reflecting on games and where the medium is headed.
The NGI Policy-in-Practice Fund – announcing the grantees
We are very excited to announce the four projects receiving funding from the Next Generation Internet Policy-in-Practice Fund.
Policymakers and public institutions have more levers at their disposal to spur innovation in the internet space than often thought, and can play a powerful role in shaping new markets for ethical tools. We particularly believe that local experimentation and ecosystem building are vital if we want to make alternative models for the internet actually tangible and gain traction. But finding the funding and space to undertake this type of trial is not always easy – especially if outcomes are uncertain. Through the NGI Policy-in-Practice fund, it has been our aim not only to provide the means to organisations to undertake a number of these trials but also make the case for local trials more generally.
Over the past summer and autumn, we went through a highly competitive applications process, ultimately selecting four ambitious initiatives that embody this vision behind the NGI Policy-in-Practice fund. Each of the projects will receive funding of up to €25,000 to test out their idea on a local level and generate important insights that could help us build a more trustworthy, inclusive and democratic future internet.
In conjunction with this announcement, we have released an interview with each of our grantees, explaining their projects and the important issues they are seeking to address in more detail. You can also find a short summary of each project below. Make sure you register for our newsletter to stay up to date on the progress of each of our grantees, and our other work on the future of the internet.
Interoperability to challenge Big Tech power
This project is run by a partnership of three organisations: Commons Network and Open Future, based in Amsterdam, Berlin and Warsaw.
This project explores whether the principle of interoperability, the idea that services should be able to work together, and data portability, which would allow users to carry their data with them to new services, can help decentralise power in the digital economy. Currently, we are, as users, often locked into a small number of large platforms. Smaller alternative solutions, particularly those that want to maximise public good rather than optimise for profit, find it hard to compete in this winner-takes-all economy. Can we use interoperability strategically and seize the clout of trusted institutions such as public broadcasters and civil society, to create an ecosystem of fully interoperable and responsible innovation in Europe and beyond?
Through a series of co-creation workshops, the project will explore how this idea could work in practice, and the role trusted public institutions can play in bringing it to fruition.
During the pandemic, where homeschooling and remote working have become the norm overnight, bridging the digital divide has become more important than ever. This project is investigating how we can make it easier for public bodies and also the private sector to donate old digital devices, such as laptops and smartphones, to low-income families currently unable to access the internet.
By extending the lifetime of a device in this way, we are also reducing the environmental footprint of our internet use. Laptops and phones now often end up being recycled, or, worse, binned, long before their actual “useful lifespan” is over, putting further strain on the system. Donating devices could be a simple but effective mechanism for ensuring the circular economy of devices is lengthened.
The project sets out to do two things: first, it wants to try out this mechanism on a local level and measure its impact through tracking the refurbished devices over time. Second, it wants to make it easier to replicate this model in other places, by creating legal templates that can be inserted in public and private procurement procedures, making it easier for device purchasers to participate in this kind of scheme. The partnership also seeks to solidify the network of refurbishers and recyclers across Europe. The lessons learned from this project can serve as an incredibly useful example for other cities, regions and countries to follow.
Many of the digital services we use today, from our favourite news outlet to social media networks, rely on maximising “engagement” as a profit model. A successful service or piece of content is one that generates many clicks, drives further traffic, or generates new paying users. But what if we optimised for human well-being and values instead?
This project, led by the BBC, seeks to try out a more human-centric focused approach to measuring audience engagement by putting human values at its core. It will do so by putting into practice longer-standing research work on mapping the kinds of values and needs their users care about the most, and developing new design frameworks that would make it easier to actually track these kinds of alternative metrics in a transparent way.
The project will run a number of design workshops and share its findings through a dedicated website and other outlets to involve the wider community. The learnings and design methodology that will emerge from this work will not just be trialled within the contexts of the project partners, but will also be easily replicable by others interested in taking a more value-led approach.
In a data economy that is growing ever more complex, giving meaningful consent about what happens to our personal data remains one of the biggest unsolved puzzles. But new online identity models have shown to be a potentially very promising solution, empowering users to share only that information that they want to share with third parties, and sharing that data on their own terms. One way that would allow such a new approach to identity and data sharing to scale would be to bring in government and other trusted institutions to build their own services using these principles. That is exactly what this project seeks to do.
The project has already laid out all the building blocks of their Data Trust Infrastructure but wants to take it one step further by actually putting this new framework into practice. The project brings together a consortium of Dutch institutional partners to experiment with one first use case, namely the sharing of vital personal data with emergency services in the case of, for example, a fire. The project will not just generate learnings about this specific trial, but will also contribute to the further finetuning of the design of the wider Data Trust Infrastructure, scope further use cases (of which there are many!), and bring on board more interested parties.
Policy-in-Practice Fund: Interoperability with a Purpose
Sophie Bloemen and Thomas de Groot from Commons Network, and Paul Keller and Alek Tarkowski from Open Future, answer a few of our questions to give us some insight into their project and what it will achieve.
by Sophie Bloemen, Alek Tarkowski, Paul Keller, Thomas de Groot
We’re introducing each of our four Policy-in-Practice Fund projects with an introductory blog post. Below, Sophie Bloemen and Thomas de Groot from Commons Network, and Paul Keller and Alek Tarkowski from Open Future, answer a few of our burning questions to give us some insight into their project and what it will achieve. We’re really excited to be working with four groups of incredible innovators and you’ll be hearing a lot more about the projects as they progress.
Can you explain to us what you mean by interoperability, and why you think interoperability could be an effective tool to counter centralisation of power in the digital economy?
Simply put, interoperability is the ability of systems to work together and to exchange data easily. In policy circles today, the term is mostly used as a solution to the structurally unhealthy dominance of a small number of extremely large platform intermediaries that is increasingly being understood to be unhealthy, both socially and politically speaking.
In the case of Facebook, for instance, introducing interoperability would mean that a user could connect with Facebook users, interact with them and with the content that they share or produce, without using Facebook. This would be possible because, thanks to interoperability, other services could connect with the Facebook infrastructure.
This scenario is often presented as a way of creating a more level playing field for competing services (who would have access to Facebook’s user base). In the same vein, proponents of interoperability argue for an ecosystem of messaging services where messages can be exchanged across services. This is also the basis of the most recent antitrust case by the US government against Facebook. At the very core, these are arguments in favour of individual freedom of choice and to empower competitors in the market. We call this approach competitive interoperability.
While this would clearly be a step in the right direction (the Digital Markets Act proposed by the European Commission at the end of last year contains a first baby step, that would require “gatekeeper platforms” to allow interoperability for ancillary services that they offer but not for their main services) it is equally clear that competitive interoperability will not substantially change the nature of the online environment. Increasing choice between different services designed to extract value from our online activities may feel better than being forced to use a single service, but it does not guarantee that the exploitative relationship between service providers and their users will change. We cannot predict the effects of increased market competition on the control of individual users over their data. In fact, allowing users to take their data from one service to another comes with a whole raft of largely unresolved personal data protection issues.
So, even though there are limits on this particular idea of interoperability, this does not mean that the concept itself has no use. Instead, we say that it needs to be envisaged with a different purpose in mind: building a different kind of online environment that answers to the needs of public institutions and civic communities, instead of ‘improving markets’. We see interoperability as a design principle that has the potential to build a more decentralised infrastructure that enables individual self-determination in the online environment. We propose to call this type of interoperability generative interoperability.
What question are you specifically trying to answer with this research? And why is it important?
In our view, the purpose of generative interoperability must be to enable what we call an Interoperable Public Civic Ecosystem. And we are exploring what such an ecosystem would encompass, together with our partners. We know that a public civic ecosystem has the potential to provide an alternative digital public space that is supported by public institutions (public broadcasters, universities and other educational institutions, libraries and other cultural heritage institutions) and civic- and commons-based initiatives. Can such an ecosystem allow public institutions and civic initiatives to route around the gatekeepers of the commercial internet, without becoming disconnected from their audiences and communities? Can it facilitate meaningful interaction outside the sphere of digital platform capitalism?
This experiment is part of the Shared Digital Europe agenda. Could you tell us a bit more about that agenda?
Shared Digital Europe is a broad coalition of thinkers, activists, policymakers and civil society. Two years ago we launched a new framework to guide policymakers and civil society organisations involved with digital policymaking in the direction of a more equitable and democratic digital environment, where basic liberties and rights are protected, where strong public institutions function in the public interest, and where people have a say in how their digital environment functions.
We established four guiding principles that can be applied to all layers of the digital space, from the physical networking infrastructure to the applications and services running on top of the infrastructure and networking stack. And they can also be applied to the social, economic or political aspects of society undergoing digital transformation. They are:
Cultivate the Commons,
Decentralise Infrastructure and
Empower Public Institutions.
Now with this new project, we are saying that interoperability should primarily be understood to enable interactions between public institutions, civic initiatives and their audiences, without the intermediation of the now dominant platforms. Seen in this light the purpose of interoperability is not to increase competition among platform intermediaries, but to contribute to building public infrastructures that lessen the dependence of our societies on these intermediaries. Instead of relying on private companies to provide infrastructures in the digital space that they can shape according to their business model needs, we must finally start to build public infrastructures that are designed to respond to civic and public values underpinning democratic societies. In building these infrastructures a strong commitment to universal interoperability based on open standards and protocols can serve as an insurance policy against re-centralisation and the emergence of dominant intermediaries.
These public infrastructures do not emerge by themselves; they are the product of political and societal will. In the European Union, the political climate seems ripe for creating such a commitment. As evidenced by the current flurry of regulation aimed at the digital space, the European Commission has clearly embraced the view that Europe needs to set its own rules for the digital space. But if we want to see real systemic change we must not limit ourselves in regulating private companies (via baby steps towards competitive interoperability and other types of regulation) but we must also invest in interoperable digital public infrastructures that empower public institutions and civil institutions. If European policymakers are serious about building the next generation internet they will need to see themselves as ecosystem-builders instead of market regulators. Understanding interoperability as a generative principle will be an important step towards this objective.
Policy in Practice Fund: Data sharing in emergencies
Manon den Dunnen from the Dutch National Police answers a few of our burning questions to give us some insight into the project and what it hopes to achieve.
by Manon den Dunnen
We’re introducing each of our four Policy-in-Practice Fund projects with an introductory blog post. Below, Manon den Dunnen from the Dutch National Police answers a few of our burning questions to give us some insight into the project and what it hopes to achieve. We’re really excited to be working with four groups of incredible innovators and you’ll be hearing a lot more about the projects as they progress.
Your project is exploring how we share information with public bodies. What is the problem with the way we do it now?
Europe has made significant progress on protecting our privacy, including through the General Data Protection Regulation (GDPR). However, there remain several problems with the collection and use of personal data. Information about us is often collected without our consent or with disguised consent, causing citizens and other data owners to lose control over their personal data and how it is used. This contributes to profiling (discrimination, exclusion) and cybercrime. At the same time, it is laborious and complex for citizens and public institutions to obtain personal data in a transparent and responsible way.
That is why a collective of (semi-) public institutions has been working towards an independent, public trust infrastructure that allows the conditional sharing of data. The Digital Trust Infrastructure or DTI aims to safeguard our public values by unravelling the process of data sharing into small steps and empowering data subjects to take control of each step at all times.
Rather than focusing on information sharing with public bodies, our project will explore public bodies taking responsibility for creating processes that help to safeguard constitutional values in a digitised society.
What kinds of risks are there for handling personal information?
Preventing these problems requires organisations to work according to several principles, most of which can be found in the GDPR. Let’s take cybercrime as a potential threat. The risk of your data being exposed to cybercrime increases as your data is stored in more places. Organisations must therefore practice data minimisation, but there are other approaches available. For example, structures that allow citizens to give conditional access to information about them, rather than having to store the data themselves. This ‘consent arrangement’ is exactly what this project will set out to test.
This idea will be new for many people, so expert support and protection should be provided when setting up a consent arrangement for data sharing. But the potential impact is huge. Take for example someone with an oxygen cylinder at home for medical use. This is not something a citizen would be expected to declare as part of their daily life. But when there is a fire, both that citizen and the fire brigade would like that information to be shared.
That piece of data about the citizen’s use of an oxygen cylinder is the only information needed by the fire brigade. But current systems often share far more information automatically, including the person’s identity. Citizens should be able to give the fire brigade conditional and temporary access to only the information that an oxygen bottle is present, such as in emergencies like a fire.
How can public bodies do this safely and with the trust of citizens? What kind of role do you think public bodies can play in increasing trust in data-sharing in the digital economy more broadly?
A data-driven approach to social and safety issues can truly improve the quality of life, facilitate safety and innovation. But we must set an example in the way we approach this. At each moment, we must carefully consider which specific piece of information is needed by a specific person, in what specific context and moment of time, for what specific purpose and for how long.
By investing in this early on and involving citizens in a democratic and inclusive way, we can not only increase trust but also use the results to demand partners to do the same and create the new normal.
You are working on a specific case study with the Dutch Police and other partners. Can you tell us a bit more about that experiment, and how you think this model could be used in other contexts too?
During the next few months, we will create a first scalable demonstrator of the Digital Trust Infrastructure. It will be based on the use-case of sharing home-related data in the context of an incident such as a fire. The generic building blocks that we create will be fundamental to all forms of data sharing, like identification, authentication, attribution, permissions, logging etc. They will also be open-source and usable in all contexts. We have a big list of things to work on!
But that is only part of the story. Complementary to the infrastructure, an important part of the project focuses on drawing up a consent arrangement. This will allow residents to conditionally share information about their specific circumstances and characteristics with specific emergency services in a safe, trusted way. To put people in control of every small step and ensure the consent arrangement will be based on equality, we will organise the necessary expertise to understand every aspect. The consequences of our actions have to be transparent, taking into account groups with various abilities, ages and backgrounds.
We will also explore how, and to what extent, the conditions and safeguards can be implemented in a consent arrangement and embedded in the underlying trust infrastructure in a democratic and resilient way. We will also look at how a sustainable and trustworthy governance structure can be set up for such a consent arrangement. We will share all our findings on these areas.
How can people get involved and find out more?
We are currently collecting experiences from other projects on how to engage residents in an inclusive, democratic and empowered (conscious) way. All the knowledge that we are building up in this area will be shared on the website of our partner Amsterdam University of Applied Sciences (HvA). Naturally, we would value hearing about the experiences of others in this area, so please do get in touch.
Workshop report: Follow us OFF Facebook – decent alternatives for interacting with citizens
The NGI Policy Summit hosted a series of policy-in-practice workshops, and below is a report of the session held by Redecentralize.org.
Despite the incessant outcry over social media giants’ disrespect of privacy and unaccountable influence on society, any public sector organisation wanting to reach citizens feels forced to be present on their enormous platforms. But through its presence, an organisation legitimises these platforms’ practices, treats them like public utilities, subjects its content to their opaque filters and ranking, and compels citizens to be on them too — thus further strengthening their dominance. How could we avoid the dilemma of either reaching or respecting citizens?
Redecentralize organised a workshop to address this question. The workshop explored the alternative of decentralised social media, in particular Mastodon, which lets users choose whichever providers and apps they prefer because these can all interoperate via standardised protocols like ActivityPub; the result is a diverse, vendor-neutral, open network (dubbed the Fediverse), analogous to e-mail and the world wide web.
Leading by example in this field is the state ministry of Baden-Württemberg, possibly the first government with an official Mastodon presence. Their head of online communications Jana Höffner told the audience about their motivation and experience. Subsequently, the topic was put in a broader perspective by Marcel Kolaja, Member and Vice-President of the European Parliament (and also on Mastodon). He explained how legislation could require the dominant ‘gatekeeper’ platforms to be interoperable too and emphasised the role of political institutions in ensuring that citizens are not forced to agree to particular terms of service in order to participate in public discussion.
Workshop report: Futurotheque – a trip to the future
Sander Veenhof, Augmented reality artist and Leonieke Verhoog, Program Manager at PublicSpaces took their session attendees on a trip to the future.
by Louis Stupple-Harris
The NGI Policy Summit hosted a series of policy-in-practice workshops, and below is a report of the session held by Sander Veenhof and Leonieke Verhoog, creators of Futurotheque.
Sander Veenhof, Augmented reality artist and Leonieke Verhoog, Program Manager at PublicSpaces took their session attendees on a trip to the future. They did this ‘wearing’ the interactive face-filters they created for their speculative fiction and research project the ‘Futurotheque’. The AR effects transformed them into citizens from the years 2021 right up to 2030, wearing the technical equipment we can expect to be wearing during those years. But besides the hardware, the filters foremostly intended to visualise the way we’ll experience the world in the near future. Which is through the HUD (Head Up Display) of our augmented reality wearables.
As users, we tend to think of the future of AR as more of the same in a hands-free way, but this session aimed to look beyond the well-known use-cases for these devices. Of course, they will provide us with all our information and entertainment needs and they can guide us wherever we are. But will that be our navigation through the physical world, or will these devices try to guide us through life? In what way will cloud intelligence enhance us, making use of the built-in camera that monitors our activities 24/7? What agency do we want to keep? And in what way should citizens be supported with handling these new devices, and the new dilemmas arising from their use?
These are abstract issues, but the face-filter visualisations applied on Sander and Leonieke helped to visualise the day-to-day impact of these technological developments on us as individuals, and have an interesting discussion with the session participants. After a dazzling peek into the next decade, the conclusion was that there’s a lot to think about when these devices are going to be part of our society. But fortunately, that’s not the case yet. We still have time to think of ways to integrate these devices into our society beforehand, instead of doing that afterwards.
Workshop report: People, not experiments – why cities must end biometric surveillance
We debated the use of facial recognition in cities with the policymakers and law enforcement officials who actually use it.
by Louis Stupple-Harris
The NGI Policy Summit hosted a series of policy-in-practice workshops, and below is a report of the session held by European Digital Rights (EDRi), which was originally published on the EDRi website.
We debated the use of facial recognition in cities with the policymakers and law enforcement officials who actually use it. The discussion got to the heart of EDRi’s warnings that biometric surveillance puts limits on everyone’s rights and freedoms, amplifies discrimination, and treats all of us as experimental test subjects. This techno-driven democratic vacuum must be stopped.
From seriously flawed live trials of facial recognition by London’s Metropolitan police force, to unlawful biometric surveillance in French schools, to secretive roll outs of facial recognition which have been used against protesters in Serbia: creepy mass surveillance by governments and private companies, using people’s sensitive face and body data, is on the rise across Europe. Yet according to a 2020 survey by the EU’s Fundamental Rights Agency, 80% of Europeans are against sharing their face data with authorities.
On 28 September, EDRi participated in a debate at the NGI Policy Summit on “Biometrics and facial recognition in cities” alongside policymakers and police officers who have authorised the use of the tech in their cities. EDRi explained that public facial recognition, and similar systems which use other parts of our bodies like our eyes or the way we walk, are so intrusive as to be inherently disproportionate under European human rights law. The ensuing discussion revealed many of the reasons why public biometric surveillance poses such a threat to our societies:
• Cities are not adequately considering risks of discrimination: according to research by WebRoots Democracy, black, brown and Muslim communities in the UK are disproportionately over-policed. With the introduction of facial recognition in multiple UK cities, minoritised communities are now having their biometric data surveilled at much higher rates. In one example from the research, the London Metropolitan Police failed to carry out an equality impact assessment before using facial recognition at the Notting Hill carnival – an event which famously celebrates black and Afro-Carribean culture – despite knowing the sensitivity of the tech and the foreseeable risks of discrimination. The research also showed that whilst marginalised communities are the most likely to have police tech deployed against them, they are also the ones that are the least consulted about it.
• Legal checks and safeguards are being ignored: according to the Chief Technology Officer (CTO) of London, the London Metropolitan Police has been on “a journey” of learning, and understand that some of their past deployments of facial recognition did not have proper safeguards. Yet under data protection law, authorities must conduct an analysis of fundamental rights impacts before they deploy a technology. And it’s not just London that has treated fundamental rights safeguards as an afterthought when deploying biometric surveillance. Courts and data protection authorities have had to step in to stop unlawful deployments of biometric surveillance in Sweden, Poland, France, and Wales (UK) due to a lack of checks and safeguards.
• Failure to put fundamental rights first: the London CTO and the Dutch police explained that facial recognition in cities is necessary for catching serious criminals and keeping people safe. In London, the police have focused on ethics, transparency and “user voice”. In Amsterdam, the police have focused on “supporting the safety of people and the security of their goods” and have justified the use of facial recognition by the fact that it is already prevalent in society. Crime prevention and public safety are legitimate public policy goals: but the level of the threat to everyone’s fundamental rights posed by biometric mass surveillance in public spaces means that vague and general justifications are just not sufficient. Having fundamental rights means that those rights cannot be reduced unless there is a really strong justification for doing so.
• The public are being treated as experimental test subjects: across these examples, it is clear that members of the public are being used as subjects in high-stakes experiments which can have real-life impacts on their freedom, access to public services, and sense of security. Police forces and authorities are using biometric systems as a way to learn and to develop their capabilities. In doing so, they are not only failing their human rights obligations, but are also violating people’s dignity by treating them as learning opportunities rather than as individual humans deserving of respect and dignity.
The debate highlighted the worrying patterns of a lack of transparency and consideration for fundamental rights in current deployments of facial recognition, and other public biometric surveillance, happening all across Europe. The European Commission has recently started to consider how technology can reinforce structural racism, and to think about whether biometric mass surveillance is compatible with democratic societies. But at the same time, they are bankrolling projects like horrifyingly dystopian iBorderCTRL. EDRi’s position is clear: if we care about fundamental rights, our only option is to stop the regulatory whack-a-mole, and permanently ban biometric mass surveillance.
Workshop report: What your face reveals – the story of HowNormalAmI.eu
At the Next Generation Internet Summit, Dutch media artist Tijmen Schep revealed his latest work - an online interactive documentary called 'How Normal Am I?'.
by Louis Stupple-Harris
The NGI Policy Summit hosted a series of policy-in-practice workshops, and below is a report of the session held by Tijmen Schep.
At the Next Generation Internet Summit, Dutch media artist Tijmen Schep revealed his latest work – an online interactive documentary called ‘How Normal Am I?‘. It explains how face recognition technology is increasingly used in the world around us, for example when dating website tinder gives all its users a beauty score to match people who are about equally attractive. Besides just telling us about it, the project also allows people to experience this for themselves. Through your webcam, you will be judged on your beauty, age, gender, body mass index (BMI), and your facial expressions. You’ll even be given a life expectancy score, so you’ll know how long you have left to live.
The project has sparked the imagination – and perhaps a little feeling of dread – in many people, as not even two weeks later the documentary has been ‘watched’ over 100.000 times.
At the Summit, Tijmen offered a unique insight into the ‘making of’ of this project. In his presentation, he talked about the ethical conundrums of building a BMI prediction algorithm that is based on photos from arrest records, and that uses science that has been debunked. The presentation generated a lot of questions and was positively received by those who visited the summit.
The NGI Policy Summit hosted a series of policy-in-practice workshops, and below is a report of the session held by the Coalition of Cities for Digital Rights, written by Beatriz Benitez and Malcolm Bain.
Data sharing platforms are playing an important role in the cities by integrating data collected throughout or related to the city and its citizens from a wide variety of sources (central administration, associated entities, utilities, private sector) to enable local authorities, businesses and even occasionally the public to access this data produced within the City and use it for limited or unlimited purposes (open data).
Malcolm introduced the session, highlighting that while Cities are keen to share data and use shared data in city digital services, they are (or should be) also aware of the digital rights issues arising in these projects related to citizens’ privacy, transparency and openness of the data used, accessibility and inclusion of citizens as well as the existence of bias in the data set used and the privatization of the use of city-related data. Luckily, cities are also in the best position to introduce the concept of ‘digital rights by design’ in these projects, and correct issues such as bias, privacy intrusions, fairness, profiling and data misuse. He briefly show-cased the Coalition work in this area in the Data Sharing Working Group, focusing on the ‘building blocks’ for rights-compliant data sharing projects to extract value from urban big data while respecting residents and visitors’ rights, including policies, processes, infrastructures, and specific actions and technologies.
Daniel highlighted the work of Eurocities on their Citizens Data Principles, which aim is to offer guidance to European local governments on more socially responsible use of data, and recognise, protect and uphold the citizens’ rights on the data they produce. The principles support how to use data-generated knowledge to improve urban life and preserve European values through scientific, civic, economic and democratic progress. Daniel presented one of his own city’s data-sharing project, Periscopio, a framework for sharing information contained in urban data (public and private) in such a way that it allows social agents and citizens to be involved to create social, scientific, economic and democratic value, as well as enabling the creation of better urban services.
Then, the cities of San Antonio, Long Beach, Portland, Toronto, Rennes, Helsinki, Amsterdam and Barcelona each presented some case studies from their cities, highlighting different issues raised by their data-sharing platforms and projects.
For the City of San Antonio, USA, Emily B. Royall addressed the issue of data bias and the need to listen to the community under the theme ‘Leveraging Data for Equity’.
Johanna Pasilkar of Helsinki shared with us the work of ‘MyData’ operator initiative and how to ease the daily life of the residents by consolidating data collected by the city’s departments and organisations and enabling sharing across several municipalities (data portability).
On behalf of the City of Amsterdam, Ron Van der Lans told us about the collaboration with the city by sharing traffic data with navigation companies such as Google, Waze and BeMobile to improve the mobility and quality of life of citizens.
Hamish Goodwin from the City of Toronto, Canada explained how they are attempting to integrate digital rights principles into the city digital infrastructure and the municipalities’ decision-making and how to put a policy framework into practice – the results of this are just coming out.
From the city of Rennes, Ben Lister introduced us to the RUDI, a local, multipartner, data sharing platform which goes beyond open data and connects users and producers to create new or/and better services.
Héctor Domínguez from the city of Portland, USA told us about the importance of ‘Racial Justice’ as a core value to regulating emergent technology, based on the respect for privacy, trusted surveillance and digital inclusion.
Ryan Kurtzman on behalf of the City of Longbeach, USA spoke about positive and negative associations of smart cities, and how participatory design of citizens in digital services can leverage the positive aspects: personal convenience, engagement and solving social challenges.
To conclude the round, Marc Pérez-Battle from Barcelona presented several data sharing and open data projects led by the City Council.
The City participants highlighted the need for embedding digital rights at design time (privacy, transparency, security, accessibility, etc.), citizen participation, and having the flexibility to adapt and correct any issues that may arise – something that may be more difficult when the technologies are embedded in the city infrastructure, and thus all the more important for correct design. Common themes among the projects include the importance of citizen involvement in projects, the respect for privacy and security, and the need for transparency and avoiding data bias. In addition, listeners to the session in the online chat also raised the issue of data ‘ownership’, and if this is a useful concept or rather misleading – cities are more stewards of data for the public, rather than an owner of data that they gather and use.
The session concluded stating that much work was still to be done, but just by raising Cities’ awareness of digital rights issues in data-sharing projects, we are making a big first step. The Coalition will shortly be releasing the Data Sharing Concept Note, and associated case studies that were briefly presented during the round table.