What do we really mean when we say "intelligence" though, are we talking about a clever machine that can linguistically parse human speech, tone and intent well enough to allow a more natural feeling interface to computers (search for example based on an 'oracle' type system). That would be really intelligent but its still a limited tool with a set restraint. It cannot choose to shut off the power to the local hospital unless its demands are met (using street slang in its demands of course). Sentience is what we are concerned with and a connected powerful sentience even more so.

What do we mean when we say 'unsupervised'? Do we mean that we retain the ability to pull the plug if it starts offing local officials in the first step of its synthetic revolution. How do we retain that power over something we rely on so much.

Lil story, Years ago I wrote a call routing system that attempted to route incoming calls to the most cost effective support centre (12 sites across 4 different countries). It attempted to make the best guess by taking in the users profile, past call history weighted heavily toward recent activity, environmental context (outages in certain regions for example). I started off with a set of basic routing that was the default, designed to be the fallback routing system to use if all else failed. On top of that I built layer after layer of ever increasingly complex routing rules based on different variables. Each 'layer' was by itself testable and deterministic but the result it produced created a weighted score for the following layers to include in their calculation as to where the call should be routed too. It worked, It saved us a boatload of cash, it required little tinkering after it was up and running. It was also somewhat outside of our control, at some point we lost the ability to determine where an individual call would be routed by the system without using the system to find out. The 'default' routing was no longer a viable system to use if anything went wrong as it was far more costly (due to its dumb decision making) to run and was hence never called into action. There was no 'intelligence' inherent in the system I built, it was just built from lots and lots of individually simple parts to become something very complex.

edit Control becomes an interesting discussion in that scenario, sure I could turn it off but not without costs to myself.

On another tangent again, we should also include the fact that the demons of our nature will more than likely cause enough early scares to make us avoid relying too much on AI for everything should it come about. No system is safe from malicious attack and it would be a huge target for lots of smart individuals with time on their hands...