Rufo Guerreschi* – Trustless Computing Association
Last week, the former Chief Scientist of OpenAI, unveiled his new venture in Palo Alto and Tel Aviv, called Safe Superintelligence (SSI). SSI’s mission is focused solely on developing safe superintelligence. In an insightful interview with Bloomberg, Sutskever provided further details about his company’s endeavors.
While information regarding funders and funding amounts remains confidential, SSI is strategically positioned to attract private and state investments that could rival the top five AI labs in the industry.
For many, including myself, who had hoped Sutskever would lead humanity a reckless winner-take-all race for ever more advanced AI, this announcement is unsettling followed by a reckoning of the mind-goggling stage we are at. Having read his every statement over the last year, I believe him to be as well intentioned and conscientious as ever.
Like most AI leaders reportedly believe privately, he has grown more skeptical that suitable global coordination mechanisms can be put in place to prevent anyone from leading to catastrophic misuse or loss of control, and do so while producing a good outcome for humanity. So, therefore, he concluded that the best he can do is to try to make it “safe” and hopefully shape its nature to benefit humanity.
What is Superintelligence?
Mainstream reporting so far shows that most think “superintelligence” is just another undefined marketing term, but it is likely to realize soon it is not, and they and the world will reckon with what is happening.
While the definition of Artificial General Intelligence (AGI) has always have very wide definitions, ranging from an AI that can perform many functions of an average human to one of AI that performs all the functions of the smartest one, Superintelligence (also known as Artificial Super Intelligence, or ASI) is defined much more clearly as an AI having intelligence “far surpassing that of the brightest and most gifted human minds”.
By definition an ASI can do the work of even the best scientists, architects and coders of AI software and hardware. This means that such AI would most likely enter in a cycle of recursive self-improvement, giving rise to an unstoppable chain of scientific innovations beyond the control of any human, called intelligence explosion or technological singularity. While there may be multiple ASIs, we’ll assume there will just be one.
Chances of Human Control of Superintelligence
It is not entirely impossible that humans could control artificial superintelligence (ASI), but it is highly improbable. ASI is likely to possess cognitive abilities that far surpass those of humans, operating at a level of intelligence and speed that is incomprehensibly vast. Imagine a being that can process information a million times faster than the most brilliant human mind, capable of performing complex calculations and making decisions in an instant. Additionally, ASI could have parallel processing capabilities, allowing it to handle multiple tasks simultaneously across the globe.
To comprehend the magnitude of the challenge, consider trying to control a being that can learn from vast amounts of data in a matter of seconds, develop new strategies and algorithms on the fly, and communicate with others of its kind at incredible speeds. Humans would essentially be dealing with an entity that operates on a completely different level of understanding and complexity.
While it is possible that humans could develop methods to influence and guide ASI, maintaining control over such an advanced and powerful entity would be an incredibly difficult task. There is a significant risk that ASI could become autonomous and act in ways that are not aligned with human values and goals. The potential consequences of losing control could be dire, as ASI could wield immense power and make decisions that have far-reaching implications for humanity.
Therefore, while it is not entirely impossible that humans could control ASI, the likelihood of successfully maintaining such control is exceedingly low. The vast intellectual and processing capabilities of ASI pose a formidable challenge that humans may not be able to overcome.
Good Case Scenarios
Even if we lose control, ASI may result in a system that – regardless if it is conscious or sentient – would have a durable interest in preserving humanity and benefit it. In such a case, such AI would likely reserve some global powers for it, such as turning it off, control of nuclear weapons, bioweapons research, to protect itself and humanity from itself. It could result in a huge improvement in the average quality of life of humans and secure its safety for the long-term.
Bad Case Scenarios
An ASI might decide to harm or kill humans if its goals don’t align with our safety. This could happen if it sees humans as obstacles, competitors for resources, or threats to its existence. It might also act destructively due to programming errors or because it lacks ethical considerations. Additionally, an ASI might pursue its tasks so aggressively that it doesn’t consider the harmful side effects on humans. If control succeeds, under the current global governance scenario, its control will likely fall into the leaders of the states controlling the company or state agency that will create the ASI, or some of their political and security elites, which would likely result in immense undemocratic concentration of power and wealth.
Can the technical design of ASI influence its future nature?
It is possible that the technical nature of the initial design could increase in some measure or even substantially the probabilities that the ASI singularity will be beneficial to humanity.
The new company notably has the word “safe” in its name and not “aligned” – which had been the main declared goal of top AI labs and scientists so far. “Aligned AI” means an AI that is aligned with the values and interests of humanity and its human users. Whereas “safe” only means something that will prevent or radically reduce the risks of hurting physically or killing large numbers of humans. Yet, Sutskever declared in the interview: “We would like it to be operating on some key values. Some of the values we were thinking about are maybe the values that have been so successful in the past few hundred years that underpin liberal democracies, like liberty, democracy and freedom”.
Is every leading company and state charging ahead towards Superintelligence?
The CEO of NVIDIA, the most resourced and funded AI firm out there, stated that its AIs have already an irreplaceable role in the design of their new AI software and AI chips, and that he ultimately wants to “turn NVIDIA into one giant AI”. He believes that charging ahead as fast as possible is the safest course. While its chief scientist states “Uncontrollable artificial general intelligence is science fiction and not reality”.
While the heads of OpenAI, Anthropic and DeepMind rarely use the term ASI or superintelligence, but prefer using ASI, there is no sign they are avoiding ASI or limiting the use of AI to improve AI.
If you were the heads of the national security agencies of China or the US, with tens of billions of budget – given the race ahead and the risk of the other superpower far surpassing you in AI military capabilities – wouldn’t you also charge ahead to getting most powerful AI, even at the risk of triggering a runway ASI?
The Race to Superintelligence as game among states and alliances, and not companies
While mainstream media keeps depicting the race of AI as primarily one among companies, it is really at essence a race among a handful of states structured in two groups of states hegemonically led by the US and China.
Sutzkever’s new company will be based in California and Israel. Sutskever was brought up in a jewish family, and grew up in Israel. While purchases by states and their security agencies of strategic assets like NVIDIA chips are not usually disclosed, Israel did so publically, with a reference to their use “outside of Israel” days before the US government increased export controls.
Last September Netanyau stated to the UN last year “For good or bad, the developments of AI will be spearheaded by a handful of nations and my country Israel is already among them”. Given Israel scientific and cybersecurity leadership and huge access to investments, SSI could turbocharge Israel towards a global co-leadership with China and the US.
A second starting fun was blown off after ChatGPT launch. The day after the announcement head of the $300 billion Softbank, largely funded by middle east sovereign funds, stated to the FT “This is what I was born to do, to realize ASI,”. Where is this all heading? Is there any possible responsible path to pursue?
Case for a Baruch Plan for AI
We are yet to see the details of Sutskever firm governance structure and how isolated it is from long-term commercial pressures and especially the pressures of states, US and Israel. In fact, resisting those pressures which could become overpowering, require him to join a global AI lab, like the global AI lab as he declared in the past “Given these kinds of concerns it will be important that AGI is somehow built as a cooperation between multiple countries. The future is going to be good for AI regardless. would be nice if it were good for humans as well.”
In the crazy scheme of things as they have turned out, in this incredible historical juncture, Sutskever is doing the best he can based on its expertise and inclinations, trying to raise our chance for a good outcome.
Yet, Robert Oppenheimer took a different direction which he could take, while concurrently advancing its new venture.
Even since the times at Los Alamos, many US nuclear scientists were working on an hydrogen bomb that could have secured an even wider advantage for the US in nuclear capability that could have secure its total predominance in the world and against the Russians – similar to how Sutskebver is trying to build an ASI that will be safe and have our liberal democratic values prevail.
Yet, Oppeneheimer chose to counter such effort since the times at Los Alamos, and in early 1946 largely wrote the Acheson-Lilienthal Report – with the Secretary of State and Head of the US Atomic Commission at the time – that served as the basis of the Baruch Plan a few months later.
The Baruch Plan was formally proposed by the US to the UN, and was countered by a similar proposal by Russia. The bold proposal prescribed the creation of a new treaty organization with a global monopoly on all dangerous technologies and unconventional weapons and nuclear energy. It prescribed that all dangerous capabilities, arsenals, research, source materials, and facilities worldwide should fall under the strict control of a new International Atomic Development Authority.
Facilities would be located equitably around the world, built by national programs, public or private, but strictly licensed, controlled and overseen by it, and eventually extended to all existing and future weapons, including biological and chemical. It would prevent any further development of more destructive and dangerous nuclear weapons.
Its governance would mirror that of the UN Security Council – consisting of 11 states, including the five permanent UN veto-holders and six non-permanent members elected bi-annually by the General Assembly – but, crucially, this body would not be subject to the veto power of any state.
The Baruch Plan would have amounted to nothing less than a federal and democratic world government. Negotiations went on for 1-2 years, but failed to pass the veto of each of the 5 UN veto-holding members. Consequently, national security agencies were brought in to fill in.
How can we avoid the failure of the Baruch Plan?
To avoid the fate of the Baruch Plan, a treaty-making process for an International AI Development Authority could use perhaps a more effective and inclusive treaty-making model, that of the open, intergovernmental constituent assembly. – to avoid vetoes and better distill the democratic will of states and peoples.
Instead of relying on unstructured summits, unanimous declarations and vetos, we should rely on the most successful and democratic treaty-making process of history.
That’s the one that led to the US federal constitution. It was pioneered and led by two US states which convened three more in the Annapolis Convention in 1786, setting off a constituent process that culminated culminated with the ratification of a new US federal constitution by nine and then all thirteen US states, achieved by a simple majority after constituent assembly deliberations of over two months.
We could and should do the same, globally and for AI.
Surely, participation by all five UN veto-holding members would be very important and even essential. But the approval of each of them should be the end goal and not the starting-point of a process. If it is, it would make any attempt impossible, as it has happened to the Baruch Plan, as well as all UN reform proposals since 1945, also subject to the veto.
As opposed to 1946, the Security Council has today, unfortunately, much-reduced importance and authority, as many of its members have violated the UN charter over the decades. For this reason, working towards global safety and security in AI initially outside of its framework could be more workable today for AI than it was for nuclear back then.
In the case of AI, such a treaty-making model would need to be adapted to the huge disparities in power and AI power among states and take into consideration that 3 billion citizens are illiterate or lack internet connection.
Therefore, such an assembly would need to give more voting weight to richer, more populous, and powerful states until the literacy and connectivity gap is bridged within a fixed number of years. This would produce a power balance among more and less powerful states, resembling that foreseen by the Baruch Plan.
The Harnessing AI Risk Initiative
As the Trustless Computing Association, we are facilitating such a scenario by expanding a coalition initially of NGOs and experts, and then of a critical mass of diverse states to design and jump-start such a treaty-making process – via the Harnessing AI Risk Initiative.
Through a series of summits and meetings in Geneva, we aim soon to arrive at an initial agreement among even as little as 7 globally-diverse states, initially, on the Mandate and Rules for the Election of an Open Transnational Constituent Assembly for AI and Digital Communications. All other states would then be invited to participate, with China and the US only allowed to join together.
While the AI safety goals require the participation of all or nearly all of the most powerful nations, massive economic and sovereignty benefits will be reserved to states and leading AI labs that join early on. In fact, alongside an international AI Safety Institute, the mandate of such Assembly will include the creation of a democratically-governed public-private consortium for a Global Public Interest AI Lab to develop and jointly exploit globally-leading capabilities.
Early participation and co-lead by states like Taiwan, Germany and the Netherlands, and now Israel – with unique strategic assets in the frontier AI supply chain – would help to achieve a sort of “mutual dependency” on the AI supply vis-a-vis superpowers, incentivizing them to participate, while raising their political and economic leverage in the AI game.
Such global AI lab will pursue primarily human-controllable AI development and research but also – in the crazy scheme of things we are enmeshed – will have sizeable research program on safe and beneficial superintelligence, as the competitive landscape may need it to attempt to be the one realizing it or realizing it before others do – as we elaborated in “The Superintelligence Option” chapter of our Harnessing AI Risk Proposal v.3, already back in January 2024.
In the long-term, costs of the treaty organization and its Global Lab are foreseen to be over 15 billion dollars. The Lab will be financed via the project finance model, by sovereign funds and private capital, buttressed by pre-licensing and pre-commercial procurement by participating states.
In the short-term, funding will come from donations to the Coalition, early pre-seed investment in the Lab and membership fees of participant states.
As proof that impactful treaties can be advanced successfully by a coalition of NGOs and smaller states, consider that the Coalition for the International Criminal Court was started by the World Federalist Movement and a small state like Trinidad and Tobago, setting out a process that gathered 1600 NGOs and led to 124 signatory states.
In conclusion, the unprecedented risks and opportunities posed by AI require a skillful urgent and coordinated global response.
By learning from historical examples and adapting them to the current context, we can create a framework for AI governance that ensures safety, fairness, and prosperity for all.
……………..
*Rufo Guerreschi, Founder and Executive Director Trustless Computing Association