re: Let's talk singularity.....

quote:

---

That depends on your philosophical views. Is consciousness emergent? Or does it require something else.

---

I don't think it depends on my views so much as it depends on the true nature of consciousness, which we don't yet understand and may never understand.

quote:

---

Is consciousness emergent? Or does it require something else.

---

Getting religious here: does it require a "soul"?

quote:

---

If it's emergent then it's merely the result of a powerul processor (brain).

---

We have clusters that we believe are more powerful than a human brain, but nothing is going to emerge from them that hasn't been programmed.

quote:

---

If not, then we will do our best to simulate intelligence, which could be even worse as it would lack anything that would qualify as a conscience.

---

This raises new questions:

Can a non-conscious intelligence be taught the difference between good and evil, right and wrong?

Could it produce original thoughts? Would it ever even think about improving itself?

quote:

---

Getting religious here: does it require a "soul"?

---

Yes the "Something else" I was referring too would be a soul. Traditionally the arguement for the nature of consiousness involved Emergent or Embodied consciousness.

quote:

---

We have clusters that we believe are more powerful than a human brain, but nothing is going to emerge from them that hasn't been programmed.

---

Yes, more powerful at its designed task - data processing. Humans are particulary poor at data processing. A simple calculator is superior to us. This again goes back to our poor understanding of the brain and consciousness, it obviously involves more than just processing power.

quote:

---

This raises new questions:
Can a non-conscious intelligence be taught the difference between good and evil, right and wrong?
Could it produce original thoughts? Would it ever even think about improving itself?

---

These I cannot answer definitively. We just don't know.

quote:

---

Let's say it determines that humans are a threat, how exactly does it stop that threat? Is it going to turn light-switches into trick electrocution switches so that every human that comes into its server room dies? (Okay, I know that's ridiculous, but seriously what?) Is it going to build a gun? Set off nuclear weapons?

---

Well clearly you haven't seen the potential for mayhem that an AI gone amok could create in a computer-controlled "Smart Home" - I mean AI can easily figure out how to do more than turn lights on and off, set thermostats, start ovens, turn on music or TV and unlock doors. For instance it can overload circuits and start wiring fires in the middle of the night for whole plugged in neighborhoods (once we humans get to think ourselves dependent on it). Plus it could disable smoke/fire detectors so no help would come. We're suckers for convenience. How about scalding-hot water pouring on you for your morning shower (it might even start normally so you'd step in, then...)? Just a couple of possibles - think of the chaos and destruction just this couple of things could create in whole communities, cities. Go larger scale - office towers, hi-rise apartments, government buildings, airports, submarines. And that's without firing off a single nuke. If AI wanted to for its own inscrutable reasons, it could plunge us all into a barbaric new stone age, and then let nature take its course.
ETA: I think you're right that AI could disrupt whole economies as we more and more depend on forms of computerized banking, stock trading, purchasing, delivery of goods, etc. Imagine all this simultaneous.