The Assumptions of Searle’s Chinese Room Thought Experiment

For the serious discussion of weighty matters and worldly issues. No off-topic posts allowed.

Moderators: Azrael, Moderators General, Prelates

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby morriswalters » Mon Nov 14, 2011 11:57 am UTC

TheGrammarBolshevik wrote:(If the ridiculous interpretive shenanigans here hold, then we should be able to say that, since a computer only runs programs, running programs is sufficient for anything that any computer can do. And since every computer can run programs, every computer has the sufficient conditions for being able to do all of those things.)

This is true and false, true for the computer, false for the user. Which is the problem with the example above. It's about point of view.

As a thought experiment the Turing Machine illuminates the workings of the CPU. Since by definition, all aspects of the Turing Machine are finite, there is no ambiguity in the CPU. The CPU doesn't play Crysis, the CPU does Boolean operations, and that's all it does.

This is the Problem with Searle's experiment. Where he proposes to look, the all he can see are Boolean operations, that is the Rule Book. But this is a finite system. It is absolutely deterministic. The CPU, the operations it does, everything that happens in a digital computer, is absolutely explicit. The implication of this is that inside this process there is no error. All conditions are true or false. This is acknowledged in the truism of GIGO. Garbage in garbage out. A digital computer will do exactly what it is told to do. No more no less. So Searle can never see the program because the program is not finite. The program is a representation of the programmer. It can't be finite because the programmer isn't finite. The only time the program can return a reflection of the programmers understanding and therefore the programs understanding is in the output. Outside the CPU. In the Chinese Room experiment the only person who can judge the meaning happening inside the Chinese Room is someone not in it.QED :lol:

As a practical example of the above there is a area of expertise in mathematics called Numerical Analysis. One of the things it does is to examine the way this transition between [finite] and [not finite] introduces errors. A CPU can only perform integer operations, for instance. All other operations involve differing degrees of error. It also demonstrates why errors can be so difficult to find, all errors are errors of logic. Imprecise understanding of of what you are telling the computer to do. Feel free to hammer on this.
As a disclaimer anything I say is my opinion and should not to be confused with fact.
morriswalters
 
Posts: 2416
Joined: Thu Jun 03, 2010 12:21 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby HungryHobo » Mon Nov 14, 2011 2:51 pm UTC

morriswalters wrote:The problem as I see it is that the only way to judge understanding is to look at the outputs. This was Turing's point. You can't judge understanding based on the operations, no matter what they are, or how they are done. You can only determine if the output shows understanding. The Turing Test avoids the lack of a definition of understanding.

Exactly .


The finite table tells the machine how to read the tape, that is how many places represent a word or byte, the program is on the tape. The state register contains data on how to treat the next byte read, depending on if it is a data or an instruction. Start State ready to read, the first read is an instruction which is stored in the state register. The instructions are Boolean operations performed on the data. The state table is updated between reads.

If you like the state register can be one of the pages of the book or it can be the man after being trained. it doesn't matter all that much as long as something is acting as the state register.

Griffin wrote:There are problems some computers can solve that some other turing machine CAN NOT SOLVE. Period.


so are you arguing that consciousness is just a matter of speed? do enough operations fast enough and you have a conscious being, fall bellow some threshold and it goes unconscious again?

Of course there are constraints, some problems take X amount of memory to solve, some problems have time constraints but those have nothing to do with computability.

The Chinese room itself uses such assumptions: that time is not an issue since the person in the room would have to write on a lot of cards and do a lot of paper shuffling in the time it takes someone to answer a single question. It also gives both the human and machine the same output devices. That's the point. You really can run crysis on a turing machine, at highest quality. you could even map some section of the tape to a monitor. It would be slow but we aren't concerned with speed for a turing machine. speed it irrelevant. The turing machine is merely a demonstration that the problem can be handled at all.

You could hook up a mechanical leg to one computer and claim that it now has special abilities which make it special but ultimately you could hook up another turing machine in it's place and it could kick your ass too.

Some computers can do things faster but even quantum computers can only handle computable problems. They can handle some types of problems faster but any turing machine with sufficient memory and time can do the same calculations.

morriswalters wrote:Since by definition, all aspects of the Turing Machine are finite


actually the tape is assumed to be infinite but any program which halts in finite time would only need a finite tape so arbitrarily big would probably be a better word. It's to take quantities of memory out of the equation for proofs. Something may only run on something with 10 gigs of ram but it's still computable in finite time.

gmalivuk wrote:For example, I suspect that metastable states are not typically programmed, and yet they can happen because of hardware.


this is where it get's complicated and depends on whether you believe the universe is non-deterministic or deterministic but unpredictable due to lack of information.
Either way you could simulate almost identical behavior by virtualizing a metastable circuit and using a good pseudo-random number generator.
Give a man a fish, he owes you one fish. Teach a man to fish, you give up your monopoly on fisheries.
HungryHobo
 
Posts: 1371
Joined: Wed Oct 20, 2010 9:01 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby gmalivuk » Mon Nov 14, 2011 3:01 pm UTC

HungryHobo wrote:
Griffin wrote:There are problems some computers can solve that some other turing machine CAN NOT SOLVE. Period.
so are you arguing that consciousness is just a matter of speed?
No. First, the issue Griffin raised was more about timing than speed in general. Second, that was just one example to illustrate the point that actual computers are physical things that have physical characteristics and which can therefore be different from each other in their capabilities.

Some computers can do things faster but even quantum computers can only handle computable problems. They can handle some types of problems faster but any turing machine with sufficient memory and time can do the same calculations.
Again, this is only true if we pretend that computers aren't physical things subject to physical laws and constraints. But as I said before: if you're seriously arguing that the substrate doesn't matter at all when it comes to consciousness, then you believe a form of dualism.

actually the tape is assumed to be infinite but any program which halts in finite time would only need a finite tape so arbitrarily big would probably be a better word. It's to take quantities of memory out of the equation for proofs. Something may only run on something with 10 gigs of ram but it's still computable in finite time.
There's still the fact that TMs can only deal with inputs and outputs from a countable set, which is a constraint the universe doesn't appear to have, what with the fact that three gravitationally interacting bodies isn't difficult for it to handle.
In the future, there will be a global network of billions of adding machines.... One of the primary uses of this network will be to transport moving pictures of lesbian sex by pretending they are made out of numbers.
Spoiler:
gmss1 gmss2
User avatar
gmalivuk
Archduke Vendredi of Skellington the Third, Esquire
 
Posts: 19301
Joined: Wed Feb 28, 2007 6:02 pm UTC
Location: Here, There, Everywhere (near Boston, anyway)

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby TheGrammarBolshevik » Mon Nov 14, 2011 3:19 pm UTC

gmalivuk wrote:But as I said before: if you're seriously arguing that the substrate doesn't matter at all when it comes to consciousness, then you believe a form of dualism.

Which, incidentally, is the explicit motivation for Searle's paper.
#xkcd-q — a pretty neat LGBTQIQ channel on Foonetic

"Grant me chastity and continence, but not yet." —St. Augustine

Ceterum autem censeo, Yalensem esse delendam.
User avatar
TheGrammarBolshevik
waldo waldorf waldron waldron's wale waler wales waley walfish walford walgreen walhalla
 
Posts: 4262
Joined: Mon Jun 30, 2008 2:12 am UTC
Location: Where.

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby HungryHobo » Mon Nov 14, 2011 3:37 pm UTC

gmalivuk wrote:Again, this is only true if we pretend that computers aren't physical things subject to physical laws and constraints. But as I said before: if you're seriously arguing that the substrate doesn't matter at all when it comes to consciousness, then you believe a form of dualism.


Not at all. if you believe that 1 finite state machine can hold an intelligent conscious being then any turing machine with sufficient memory will as well.

Take your magical conscious finite state machine, run it, have a chat with it, discuss the problems of life. Let it know you're going to turn off it's eyes and ears etc for a while so it's going to be on it's own for a while, a brain it a jar as it were. Then suspend the finite state machine, note every state and copy that state into another finite state machine, say a virtualized one on a turing machine implemented with rocks and a man dragging them around.

run the finite state machine for a number of steps ("the rows blur past as I compute a single step" etc) and then after a certain amount of steps pause the turing machine, note the state, every 1 every zero and transfer this state back to the original finite state machine.

Your AI greets you as you "turn it's cameras back on" and states that it had a nice think about the meaning of life or whatever. As far as it's concerned it just sat and thought for a while without input from outside.

if it doesn't believe it was in it's normal finite state machine the whole time then your finite state machines were not equivalent. if you could not transfer the original finite state machine into the Turing machine then it wasn't a finite state machine.

something moved back and forth: information, nothing more.

If you don't think this would work or don't believe the AI would really be conscious while in the rocks but would be conscious while in the special finite state machine then you are the one proposing some kind of dualism, soul or magical conscious matter.
Last edited by HungryHobo on Mon Nov 14, 2011 4:36 pm UTC, edited 1 time in total.
Give a man a fish, he owes you one fish. Teach a man to fish, you give up your monopoly on fisheries.
HungryHobo
 
Posts: 1371
Joined: Wed Oct 20, 2010 9:01 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby gmalivuk » Mon Nov 14, 2011 4:00 pm UTC

HungryHobo wrote:If you don't think this would work or don't believe the AI would really be conscious while in the rocks but would be conscious while in the special finite state machine then you are the one proposing some kind of dualism, soul or magical conscious matter.
Yep, I sure would. If that's what I believed. My point was that actual physical computers (and brains) are not the same as idealized finite state machines, and so are not necessarily all the same as each other.

Also, note that by saying computers and brains work differently, I am *not* saying that an electronic computer *can't* be conscious. I'm just pointing out that our current understanding of brains isn't sufficient for concluding that minds can run on computers.
In the future, there will be a global network of billions of adding machines.... One of the primary uses of this network will be to transport moving pictures of lesbian sex by pretending they are made out of numbers.
Spoiler:
gmss1 gmss2
User avatar
gmalivuk
Archduke Vendredi of Skellington the Third, Esquire
 
Posts: 19301
Joined: Wed Feb 28, 2007 6:02 pm UTC
Location: Here, There, Everywhere (near Boston, anyway)

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby morriswalters » Mon Nov 14, 2011 4:27 pm UTC

The program as represented by the code in the machine(binary) is absolutely defined at any given operation, you could do the operation with a pencil. Speed only matters to the the person outside the machine receiving outputs. There is no lapse in consciousness on the part of the suspended AI. He/she can't experience it. Only the observer can. It's not like being unconscious. The state of the machine is stored when it stops. This includes any clocks. When you are unconscious and awaken you don't return to the state when you went unconscious you return to a changed state.

Computers function in exactly that fashion. If they weren't finite state machines they wouldn't work. Searle makes the claim that you can stand on the inside and if the machine had understanding then he could see it there. Modern digital computers run in two separate but connected worlds. The Boolean world(finite) and program(non finite)world. Modern Neuroscience can only study the human brain in two ways, by looking at the outputs, or by looking at scans of the brains operating. The scans can tell them where things happen and identify pathways between them but can not tell what is happening. They can only interrogate the outputs to determine the what. For example cutting the corpus callosum reduces the connections between the left and right hemisphere of the brain. But to look at what it does you have to test the outputs.

edit paragraph separation for clarity
As a disclaimer anything I say is my opinion and should not to be confused with fact.
morriswalters
 
Posts: 2416
Joined: Thu Jun 03, 2010 12:21 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby TheGrammarBolshevik » Mon Nov 14, 2011 4:43 pm UTC

HungryHobo wrote:Your AI greets you as you "turn it's cameras back on" and states that it had a nice think about the meaning of life or whatever. As far as it's concerned it just sat and thought for a while without input from outside.

Sure. It will also say this if you perform the calculation with the rocks ahead of time (perhaps the state of the machine on shutdown is predictable), or if you run the same experiment with the machine ten times but only use the rocks once, and just load in the same information from the first calculation. If I write down the appropriate state in a book and then run the experiment, is my inert book understanding anything during the downtime?

Many people have memories of experiences that did not happen, and that could not possibly have happened (for example, a distinct memory of a radio broadcast of a baseball game being interrupted by news of the Pearl Harbor bombing, which took place well after the end of the baseball season). So, for something to simply say, or even to think, that it has had some experience or another in the past is not for it to actually have had that experience.

HungryHobo wrote:If you don't think this would work or don't believe the AI would really be conscious while in the rocks but would be conscious while in the special finite state machine then you are the one proposing some kind of dualism, soul or magical conscious matter.

It's not dualism to say that some physical objects have properties that others do not. My Turing-complete computer is a piece of shit, and runs much hotter and noisier than most other Turing-complete computers. Is this "heat dualism"?
#xkcd-q — a pretty neat LGBTQIQ channel on Foonetic

"Grant me chastity and continence, but not yet." —St. Augustine

Ceterum autem censeo, Yalensem esse delendam.
User avatar
TheGrammarBolshevik
waldo waldorf waldron waldron's wale waler wales waley walfish walford walgreen walhalla
 
Posts: 4262
Joined: Mon Jun 30, 2008 2:12 am UTC
Location: Where.

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby morriswalters » Mon Nov 14, 2011 6:43 pm UTC

TheGrammarBolshevik wrote:Sure. It will also say this if you perform the calculation with the rocks ahead of time (perhaps the state of the machine on shutdown is predictable), or if you run the same experiment with the machine ten times but only use the rocks once, and just load in the same information from the first calculation. If I write down the appropriate state in a book and then run the experiment, is my inert book understanding anything during the downtime?

There is no understanding anywhere inside the machine while it's operating, none. All the machine does is run Boolean operations, twiddles bits into other bits. The understanding comes when you transition between states, and move back into the (non finite) world. Because a computers hardware is a finite state machine, you can capture the state at any point and restore it to that state. The only state which is close to this, in humans, is death. The analogous operation would be suspended animation, and people who freeze their bodies believe that the state of their bodies at death is able to be restored. A dubious belief I might add. What a program actually is, seems to fuck with peoples minds. Only a representation of what the programmer intended runs. It's only as good as the programmers understanding of what he is trying to do. When you debug software you will run data that you have massaged to test the action of what you programed. The example that comes to mind is the internal registers which hold intermediate operations. You examine those registers to check the value and see if it is what you expected. If the value is wrong, you don't look at the hardware. You look at your program. Assuming no hardware failures(ideal case) that is were the failure is. Like I said the hardware only twiddles bits. The hardware doesn't care and it doesn't need to. Think of the program as a model of the programmers understanding. The data is only as good as the model, and the only way to know that the model is wrong is by looking at the output, so if there were meaning or understanding to be found it would only be found in the output. Meaning exists only when tested against the lack of it. I think I might publish this, take a year to learn enough formal logic to state it, and then send it to whatever Philosophical Journal that will have it. It might be kind of amusing. :lol:
As a disclaimer anything I say is my opinion and should not to be confused with fact.
morriswalters
 
Posts: 2416
Joined: Thu Jun 03, 2010 12:21 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby HungryHobo » Mon Nov 14, 2011 7:23 pm UTC

TheGrammarBolshevik wrote:
Many people have memories of experiences that did not happen, and that could not possibly have happened (for example, a distinct memory of a radio broadcast of a baseball game being interrupted by news of the Pearl Harbor bombing, which took place well after the end of the baseball season). So, for something to simply say, or even to think, that it has had some experience or another in the past is not for it to actually have had that experience.



If you wanted you could include input and output as well for the rock turing machine. Ignoring the speed of the person dragging the rocks around you could *continue* your conversation while it's in the computer made of rocks. You could even feed it the same input as it would get were it still in the *special* computer, bit for bit. And it shouldn't even be able to tell the difference itself. Or course you get into silliness where you start claiming that conversations you had and/or are having don't count as real.

it would be as real as any of it's other memories. exactly as real.

this is what people seem to find hard to understand. Software is information, it's not some special piece of matter. software is software is software. always and everywhere no matter what it's running on.
Give a man a fish, he owes you one fish. Teach a man to fish, you give up your monopoly on fisheries.
HungryHobo
 
Posts: 1371
Joined: Wed Oct 20, 2010 9:01 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby gmalivuk » Mon Nov 14, 2011 7:39 pm UTC

HungryHobo wrote:this is what people seem to find hard to understand. Software is information, it's not some special piece of matter. software is software is software. always and everywhere no matter what it's running on.
And what *you* don't seem to understand is that evolved minds don't necessarily translate nicely into your magical idealized world where all hardware runs exactly the way it needs to for every computer to be equivalent to a theoretical Turing machine.
In the future, there will be a global network of billions of adding machines.... One of the primary uses of this network will be to transport moving pictures of lesbian sex by pretending they are made out of numbers.
Spoiler:
gmss1 gmss2
User avatar
gmalivuk
Archduke Vendredi of Skellington the Third, Esquire
 
Posts: 19301
Joined: Wed Feb 28, 2007 6:02 pm UTC
Location: Here, There, Everywhere (near Boston, anyway)

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby HungryHobo » Mon Nov 14, 2011 7:46 pm UTC

gmalivuk wrote:
And what *you* don't seem to understand is that evolved minds don't necessarily translate nicely into your magical idealized world where all hardware runs exactly the way it needs to for every computer to be equivalent to a theoretical Turing machine.



Please show me a finite state machine which can't be translated into a virtualised version in a turing machine. keeping in mind that any IO can be accounted for. output to screens can be mapped to areas of the tape, input similarly and any output device or peripheral at all can be mapped to an area of the tape.

some things have time/memory constraints but those are utterly irrelevant when it comes to the question of whether a problem is considered to be computable.

Nobody has put forward anything to support the idea that there's anything which the matter in evolved brains is doing which can't be emulated.
Give a man a fish, he owes you one fish. Teach a man to fish, you give up your monopoly on fisheries.
HungryHobo
 
Posts: 1371
Joined: Wed Oct 20, 2010 9:01 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby morriswalters » Mon Nov 14, 2011 8:08 pm UTC

HooRah HungryHobo, my estimation of you was faulty, I apologize, you do get it.
Spoiler:
I'll offer a practical example as a question. Does Chinese software need Chinese hardware? Narrow it down further. Does Intel sell different processors for different languages? I think not! Does the processor need to understand the languages. I think not. The software needs to know enough to translate the programmers instructions precisely to binary code, that's all, and that is written into a different layer of software. I believe it is called localization.

edited missing spoiler tag
As a disclaimer anything I say is my opinion and should not to be confused with fact.
morriswalters
 
Posts: 2416
Joined: Thu Jun 03, 2010 12:21 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby gmalivuk » Mon Nov 14, 2011 8:12 pm UTC

HungryHobo wrote:Please show me a finite state machine which can't be translated into a virtualised version in a turing machine.
First you need to reread what I said, and then show me a physical computer that *exactly* mimics the theoretical ideal. In other words, perfectly working hardware with zero glitches or bugs or other unintended consequences.

And once you find this magical machine, I'll agree that it can be translated into a virtual TM. But as long as we're talking about physical computers in the real world, such as the one I'm typing on and the one in my head, you can't just presuppose that absolutely *everything* it does is the same as a TM would do.
In the future, there will be a global network of billions of adding machines.... One of the primary uses of this network will be to transport moving pictures of lesbian sex by pretending they are made out of numbers.
Spoiler:
gmss1 gmss2
User avatar
gmalivuk
Archduke Vendredi of Skellington the Third, Esquire
 
Posts: 19301
Joined: Wed Feb 28, 2007 6:02 pm UTC
Location: Here, There, Everywhere (near Boston, anyway)

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby aoeu » Mon Nov 14, 2011 10:07 pm UTC

Doesn't assuming that computers in general can be conscious lead to the conclusion that it's essentially information which is conscious? What kind of physics would lead to that?
aoeu
 
Posts: 233
Joined: Fri Dec 31, 2010 4:58 pm UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby morriswalters » Mon Nov 14, 2011 10:19 pm UTC

John Searle wrote:http://www.bbsonline.org/documents/a/00/00/04/84/bbs00000484-00/bbs.searle2.html
Minds, Brains, and Programs

"But could something think, understand, and so on solely in virtue of being a computer with the right sort of
program? Could instantiating a program, the right program of course, by itself be a sufficient condition of
understanding?"

This I think is the right question to ask, though it is usually confused with one or more of the earlier questions,
and the answer to it is no.

"Why not?"

Because the formal symbol manipulations by themselves don't have any intentionality; they are quite
meaningless; they aren't even symbol manipulations, since the symbols don't symbolize anything. In the linguistic
jargon, they have only a syntax but no semantics. Such intentionality as computers appear to have is solely in
the minds of those who program them and those who use them, those who send in the input and those who
interpret the output.


The aim of the Chinese room example was to try to show this by showing that as soon as we put something
into the system that really does have intentionality (a man), and we program him with the formal program, you
can see that the formal program carries no additional intentionality. It adds nothing, for example, to a man's
ability to understand Chinese.

Duh.
Run this to run this.
As a disclaimer anything I say is my opinion and should not to be confused with fact.
morriswalters
 
Posts: 2416
Joined: Thu Jun 03, 2010 12:21 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby Anaphase » Tue Nov 15, 2011 12:13 am UTC

gmalivuk wrote:
HungryHobo wrote:If you don't think this would work or don't believe the AI would really be conscious while in the rocks but would be conscious while in the special finite state machine then you are the one proposing some kind of dualism, soul or magical conscious matter.
Yep, I sure would. If that's what I believed. My point was that actual physical computers (and brains) are not the same as idealized finite state machines, and so are not necessarily all the same as each other.

Also, note that by saying computers and brains work differently, I am *not* saying that an electronic computer *can't* be conscious. I'm just pointing out that our current understanding of brains isn't sufficient for concluding that minds can run on computers.

True, however it's rather hard to imagine just how such glitches could actually be a necessary condition for instantiating a mind.
Anaphase
 
Posts: 33
Joined: Fri Oct 21, 2011 7:20 pm UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby Griffin » Tue Nov 15, 2011 4:54 am UTC

Well, think about Neural Nets. Physical neural nets can, very seriously, do things digital neural nets cannot. Have identical inputs, identical gates, and the physical machine will produce different output - and not necessarily as a "bug", either - evolved neural nets will occasionally use this difference in ability to produce the correct result.

Regardless, consciousness is not a property of a thing, by itself. A person that is unconscious is, thus, no longer conscious. Consciousness is a product of interaction with other components of a larger system in an adaptive way. I'd argue that timing issues can very easily become quite relevant to that.

some things have time/memory constraints but those are utterly irrelevant when it comes to the question of whether a problem is considered to be computable.

So is my apple problem computable, or not?
Bdthemag: "I don't always GM, but when I do I prefer to put my player's in situations that include pain and torture. Stay creative my friends."

Bayobeasts - the Pokemon: Orthoclase project.
User avatar
Griffin
 
Posts: 1364
Joined: Sun Apr 08, 2007 7:46 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby HungryHobo » Tue Nov 15, 2011 1:46 pm UTC

Griffin wrote:Well, think about Neural Nets. Physical neural nets can, very seriously, do things digital neural nets cannot. Have identical inputs, identical gates, and the physical machine will produce different output - and not necessarily as a "bug", either - evolved neural nets will occasionally use this difference in ability to produce the correct result.
So are you claiming that being deterministic disqualifies computers?


Griffin wrote: Consciousness is a product of interaction with other components of a larger system in an adaptive way. I'd argue that timing issues can very easily become quite relevant to that.


Out of interest would you consider someone who is currently deaf, blind and paralysed(ie lacking any input) but awake and aware of it to be unconscious?


Griffin wrote:So is my apple problem computable, or not?


If you gave a problem which requires a minimum amount of memory to achieve then said "oh but you can't use a computer with that much memory" it would be no different. the problem would be computable but you'd still need a computer with some minimum amount of memory.

Those constraints have nothing to do with computability.

A turing machine which can do X steps per second where X is some finite number could do your task. hence it is perfectly computable.


In other words, perfectly working hardware with zero glitches or bugs or other unintended consequences.
And once you find this magical machine, I'll agree that it can be translated into a virtual TM. But as long as we're talking about physical computers in the real world, such as the one I'm typing on and the one in my head, you can't just presuppose that absolutely *everything* it does is the same as a TM would do.


So you're claiming that bugs and failures to compute accurately are somehow essential?

Thing is we can apply the same approach. why couldn't we just simulate your shoddy physical computer in our turing machine atom for atom, electron for electron? it would be vastly less efficient but you'd include any bugs or novel physical behavior.
Give a man a fish, he owes you one fish. Teach a man to fish, you give up your monopoly on fisheries.
HungryHobo
 
Posts: 1371
Joined: Wed Oct 20, 2010 9:01 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby gmalivuk » Tue Nov 15, 2011 3:21 pm UTC

HungryHobo wrote:So are you claiming that being deterministic disqualifies computers?
Not at all. I don't think anyone here is saying that brains aren't deterministic, for one thing.

In other words, perfectly working hardware with zero glitches or bugs or other unintended consequences.
And once you find this magical machine, I'll agree that it can be translated into a virtual TM. But as long as we're talking about physical computers in the real world, such as the one I'm typing on and the one in my head, you can't just presuppose that absolutely *everything* it does is the same as a TM would do.
So you're claiming that bugs and failures to compute accurately are somehow essential?
No. I am claiming that you can't just ignore them, because they might turn out to be a part of how consciousness works.

why couldn't we just simulate your shoddy physical computer in our turing machine atom for atom, electron for electron? it would be vastly less efficient but you'd include any bugs or novel physical behavior.
If it turns out to be possible to simulate reality at the quantum level on a digital computer. But as far as I know, that's still an open question.
In the future, there will be a global network of billions of adding machines.... One of the primary uses of this network will be to transport moving pictures of lesbian sex by pretending they are made out of numbers.
Spoiler:
gmss1 gmss2
User avatar
gmalivuk
Archduke Vendredi of Skellington the Third, Esquire
 
Posts: 19301
Joined: Wed Feb 28, 2007 6:02 pm UTC
Location: Here, There, Everywhere (near Boston, anyway)

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby Griffin » Tue Nov 15, 2011 4:44 pm UTC

So are you claiming that being deterministic disqualifies computers?

Disqualifies them for what?

If you gave a problem which requires a minimum amount of memory to achieve then said "oh but you can't use a computer with that much memory" it would be no different.


Obviously. But the definition of a Turing machine includes infinite memory - it does not include infinitely quick processing speed. Thus, there are problems that some hypothetical turing machines could solve, but others could not, based on properties beyond being or not being a Turing Complete. True or false?

Out of interest would you consider someone who is currently deaf, blind and paralysed(ie lacking any input) but awake and aware of it to be unconscious?


How do they know they are awake? How do /we/ know they are awake? And to be honest, there's quite a bit of communication and sensing going on even if you remove all of those. If you do truly remove someone from sensory experience? Including internal sensory experience? I'm not sure if you would have consciousness at that point. Imagine you had the same person, but they lacked any and all sensory experience FROM BIRTH. Never had it. Ignore the fact they are conscious or unconscious - tell me, do you think such a person could have understanding? Which is really what the original question was about?

For what we consider consciousness and understanding, I think time matters. I think sensory ability matters. I don't think understanding can exist in a vacuum. I don't think communication matters in the slightest (at least not external communication). Consciousness is based around the ability to react to ones environment in intelligent ways - something software alone will never, ever be capable of. This doesn't mean we can't build an AI, and I've never argued against it, because we are not, actually, limited to software.
Bdthemag: "I don't always GM, but when I do I prefer to put my player's in situations that include pain and torture. Stay creative my friends."

Bayobeasts - the Pokemon: Orthoclase project.
User avatar
Griffin
 
Posts: 1364
Joined: Sun Apr 08, 2007 7:46 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby morriswalters » Tue Nov 15, 2011 5:09 pm UTC

Searle commits a fallacy. He posits that as a man that he could not sit in the locus of a machine and gain understanding by doing the the steps that allow a machine to reply to written speech. When in point of fact the conscious part of Searle doesn't know how to understand written speech at all. Searle's brain does. He experiences understanding consciously but has no knowledge of how he did it. All conscious part of Searle experiences is the understanding that some inner process reveals to him. The output of his process.

He cannot demonstrate that the process he see's in the room is a process that he could understand even if it did generate whatever it is that is understanding, since he cannot compare it to his process. He cannot exclude the possibility that the process occurs in a way which is opaque to him simply because it finds understanding by different means.

In other words he himself(the conscious part) see's nothing but output and gets nothing but input. It's processed in a way he doesn't understand, because the only way he knows about it is by a process he says is not sufficient, that is by examining the output.

edited for clarification, maybe.

The brain itself never see's a tree, it receives signals which it interprets as trees. It doesn't hear sounds, it gets signals, and so on. Searle at the locus sees the process but he doesn't see the understanding. The understanding is the output. The analogous situation is demonstrated by a computer. The data in a computer never exists as anything more than ones and zeros to the computer. It only exists as meaningful data when output from the computer. Computers work in an unambiguous domain where all knowledge is explicit and all operations are predictable. The output itself is in a close but separate domain. The data is never accessible to the computer in the domain we exist in and in the fashion that we use it. The computer is Searle's brain and the world is Searle's consciousness.

To ask the question of what is consciousness, is meaningless. Unless it incorporates awareness of our separation from the world and from the internal process that generates consciousness, that is, that consciousness is a product of a domain not directly accessible to us, much as the world is not in a domain that the brain can access directly. Consciousness itself is a process by which we interact with the world with the understanding given to us by our brain. Consciousness must exist in a separate domain so that the brain has a point of reference for its interpretation of the world.
As a disclaimer anything I say is my opinion and should not to be confused with fact.
morriswalters
 
Posts: 2416
Joined: Thu Jun 03, 2010 12:21 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby HungryHobo » Wed Nov 16, 2011 10:15 am UTC

Griffin wrote:Obviously. But the definition of a Turing machine includes infinite memory - it does not include infinitely quick processing speed. Thus, there are problems that some hypothetical turing machines could solve, but others could not, based on properties beyond being or not being a Turing Complete. True or false?


A turing machine which ever actually finishes a task can only use some finite amount of memory.

The tape is assumed to be infinite for simplicities sake but it would be better called "as big as it needs to be".

You can't read from or write to infinite memory in a finite number of steps.
Give a man a fish, he owes you one fish. Teach a man to fish, you give up your monopoly on fisheries.
HungryHobo
 
Posts: 1371
Joined: Wed Oct 20, 2010 9:01 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby gmalivuk » Wed Nov 16, 2011 2:48 pm UTC

Good job. Now why don't you read Griffin's post and actually answer the question therein, which wasn't about whether the tape is infinite or finite?
In the future, there will be a global network of billions of adding machines.... One of the primary uses of this network will be to transport moving pictures of lesbian sex by pretending they are made out of numbers.
Spoiler:
gmss1 gmss2
User avatar
gmalivuk
Archduke Vendredi of Skellington the Third, Esquire
 
Posts: 19301
Joined: Wed Feb 28, 2007 6:02 pm UTC
Location: Here, There, Everywhere (near Boston, anyway)

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby HungryHobo » Wed Nov 16, 2011 2:56 pm UTC

gmalivuk wrote:Good job. Now why don't you read Griffin's post and actually answer the question therein, which wasn't about whether the tape is infinite or finite?

ok

True or false?

Where "enough" is finite:

with enough time/memory/energy false. without enough time /memory/energy true.

just making the point that the infinity bit isn't the issue.
Give a man a fish, he owes you one fish. Teach a man to fish, you give up your monopoly on fisheries.
HungryHobo
 
Posts: 1371
Joined: Wed Oct 20, 2010 9:01 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby gmalivuk » Wed Nov 16, 2011 3:29 pm UTC

Yes, we get it, finite-tape machines are different from infinite-tape ones.

But that's not related to the point Griffin was making, so I was wondering whether you might go ahead and address that at some point, seeing as you quoted it in your last post...
In the future, there will be a global network of billions of adding machines.... One of the primary uses of this network will be to transport moving pictures of lesbian sex by pretending they are made out of numbers.
Spoiler:
gmss1 gmss2
User avatar
gmalivuk
Archduke Vendredi of Skellington the Third, Esquire
 
Posts: 19301
Joined: Wed Feb 28, 2007 6:02 pm UTC
Location: Here, There, Everywhere (near Boston, anyway)

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby morriswalters » Wed Nov 16, 2011 3:41 pm UTC

Griffin wrote:Obviously. But the definition of a Turing machine includes infinite memory - it does not include infinitely quick processing speed. Thus, there are problems that some hypothetical turing machines could solve, but others could not, based on properties beyond being or not being a Turing Complete. True or false?

A Turing machine can solve the problem if you can state an Algorithm that could solve the problem. That is a series of steps that would give you the answer. The power of the Turing machine is in running Algorithms.

From the Wikipedia article on Turing Machines.
The thesis states that Turing machines indeed capture the informal notion of effective method in logic and mathematics, and provide a precise definition of an algorithm or 'mechanical procedure'.


From the Wikipedia article on Turing Completeness.
To show that something is Turing complete, it is enough to show that it can be used to simulate the most primitive computer, since even the simplest computer can be used to simulate the most complicated one. All general purpose programming languages and modern machine instruction sets are Turing complete, notwithstanding finite-memory limitations.


Processing speed has nothing to do with anything, as long as the internal process is consistent within itself. Computational speed is only relevant if the output is needed in a specific finite period of time.
As a disclaimer anything I say is my opinion and should not to be confused with fact.
morriswalters
 
Posts: 2416
Joined: Thu Jun 03, 2010 12:21 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby gmalivuk » Wed Nov 16, 2011 3:47 pm UTC

morriswalters wrote:Processing speed has nothing to do with anything
No, processing speed merely has nothing to do with whether something is a Turing machine or "computable" in the theoretical sense. But Griffin's point was that there are in fact things which some TMs can do while others can't, because the internal timing and the timing of inputs don't match up properly.
In the future, there will be a global network of billions of adding machines.... One of the primary uses of this network will be to transport moving pictures of lesbian sex by pretending they are made out of numbers.
Spoiler:
gmss1 gmss2
User avatar
gmalivuk
Archduke Vendredi of Skellington the Third, Esquire
 
Posts: 19301
Joined: Wed Feb 28, 2007 6:02 pm UTC
Location: Here, There, Everywhere (near Boston, anyway)

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby morriswalters » Wed Nov 16, 2011 3:57 pm UTC

Then change that to read, speed doesn't matter unless the program is I\O dependent. An example of this would be industrial machine controller. But that says nothing about solving the problem, rather it speaks to being able to use the output in a useful way.
As a disclaimer anything I say is my opinion and should not to be confused with fact.
morriswalters
 
Posts: 2416
Joined: Thu Jun 03, 2010 12:21 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby TheGrammarBolshevik » Wed Nov 16, 2011 5:43 pm UTC

I'm a little confused; why is this discussion about computability? Searle's thesis is precisely that understanding isn't a matter of being able to compute something, so establishing that Turing-complete machines can compute the same things as brains doesn't get you anywhere unless you're already begging the question.
#xkcd-q — a pretty neat LGBTQIQ channel on Foonetic

"Grant me chastity and continence, but not yet." —St. Augustine

Ceterum autem censeo, Yalensem esse delendam.
User avatar
TheGrammarBolshevik
waldo waldorf waldron waldron's wale waler wales waley walfish walford walgreen walhalla
 
Posts: 4262
Joined: Mon Jun 30, 2008 2:12 am UTC
Location: Where.

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby gmalivuk » Wed Nov 16, 2011 5:57 pm UTC

TheGrammarBolshevik wrote:Searle's thesis is precisely that understanding isn't a matter of being able to compute something, so establishing that Turing-complete machines can compute the same things as brains doesn't get you anywhere unless you're already begging the question.
Yes, but Searle's thesis is shitty. So if we were able to actually establish that a Turing machine really truly can take the same inputs as a brain and give the same outputs, that would be a strong argument against Searle's thesis, CRE notwithstanding.

Also, the discussion of computability is to establish what it could possibly mean to say consciousness lies in both the software and hardware together. Because the theoretical meaning of "computable" merely requires that a system be Turing-complete, and so the hardware can't possibly matter beyond that. So if what the brain does is computable in this sense, it would mean consciousness *doesn't* depend on the hardware.
In the future, there will be a global network of billions of adding machines.... One of the primary uses of this network will be to transport moving pictures of lesbian sex by pretending they are made out of numbers.
Spoiler:
gmss1 gmss2
User avatar
gmalivuk
Archduke Vendredi of Skellington the Third, Esquire
 
Posts: 19301
Joined: Wed Feb 28, 2007 6:02 pm UTC
Location: Here, There, Everywhere (near Boston, anyway)

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby morriswalters » Wed Nov 16, 2011 6:02 pm UTC

TheGrammarBolshevik wrote:I'm a little confused; why is this discussion about computability? Searle's thesis is precisely that understanding isn't a matter of being able to compute something, so establishing that Turing-complete machines can compute the same things as brains doesn't get you anywhere unless you're already begging the question.

Searle's thesis is obliquely related to computability.

John Searle in Minds, Brains, and Programs wrote:"But could something think, understand, and so on solely in virtue of being a computer with the right sort of
program? Could instantiating a program, the right program of course, by itself be a sufficient condition of
understanding?"

This I think is the right question to ask, though it is usually confused with one or more of the earlier questions,
and the answer to it is no.

"Why not?"

Because the formal symbol manipulations by themselves don't have any intentionality; they are quite
meaningless; they aren't even symbol manipulations, since the symbols don't symbolize anything. In the linguistic
jargon, they have only a syntax but no semantics. Such intentionality as computers appear to have is solely in
the minds of those who program them and those who use them, those who send in the input and those who
interpret the output.

He seems to argue that intentionality isn't a computable function. He may be right but I would say that his argument is faulty.
As a disclaimer anything I say is my opinion and should not to be confused with fact.
morriswalters
 
Posts: 2416
Joined: Thu Jun 03, 2010 12:21 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby HungryHobo » Wed Nov 16, 2011 6:47 pm UTC

Only tangentially related but a fun little read:

http://www.mit.edu/people/dpolicar/writ ... gMeat.html
Give a man a fish, he owes you one fish. Teach a man to fish, you give up your monopoly on fisheries.
HungryHobo
 
Posts: 1371
Joined: Wed Oct 20, 2010 9:01 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby Griffin » Wed Nov 16, 2011 6:55 pm UTC

Processing speed has nothing to do with anything, as long as the internal process is consistent within itself. Computational speed is only relevant if the output is needed in a specific finite period of time.


You keep saying it has nothing to do with anything, but you haven't said why. The question was simply "are there things one TM can do that another one can't?". Answering with "No, not counting the situations where its yes" is what doesn't mean anything.

A Turing machine can solve the problem if you can state an Algorithm that could solve the problem. That is a series of steps that would give you the answer. The power of the Turing machine is in running Algorithms.

And the question is, are algorithms /enough/ for consciousness? Or do you need something extra? If they aren't, then hardware (perhaps) matters. Since there are plenty of other things algorithms alone aren't enough for, I don't see why this wouldn't be the case.

Also, you can quite easily write an algorithm to count apples. The point of my example is that algorithm would only be correct on some machines.
Bdthemag: "I don't always GM, but when I do I prefer to put my player's in situations that include pain and torture. Stay creative my friends."

Bayobeasts - the Pokemon: Orthoclase project.
User avatar
Griffin
 
Posts: 1364
Joined: Sun Apr 08, 2007 7:46 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby morriswalters » Wed Nov 16, 2011 7:16 pm UTC

Loved the story. It speaks to ego and self image. However back to the point. Consciousness exists nowhere. It's a model of the world created by the brain. It's no more than a push pin in a map that says this is where I am. The conscious mind, or model in my argument, sees nothing and knows nothing. Look at tests of split brain patients. They cannot consciously recognize some events which they do experience, but it can be demonstrated that the brain still does. This happens because the left side doesn't know what the right side is doing. You either do some ridiculous gymnastics or accept that consciousness doesn't represent reality, merely the brains understanding of it. Split brain patients model is faulty because their corpus callosum is damaged. Searle's fallacy is thinking that consciousness participates. It doesn't. The point of the talking rulebook was to show that things could be happening in a way Searle can't see at the nexus. His brain may not be able to acquire the meaning even if it's there.

two edits 1 to answer Griffin one to say i edited :oops:
Griffin

Separate the process from the output. Two different things are happening, machine time and real time. Machine time takes place internal to the machine, independent of the I/O. As long as the internal timing is consistent everything will happen when it is supposed to, this is controlled by the internal clock. I/O processes may cause the machine to wait, but that is because of the I/O, not the computer. The computer doesn't care how fast it operates, the user does.

As to the second I have no clue. Neither does Searle.

The apple counting algorithm itself is machine independent. It's the implementation that is different for different machines.
Last edited by morriswalters on Wed Nov 16, 2011 7:34 pm UTC, edited 2 times in total.
As a disclaimer anything I say is my opinion and should not to be confused with fact.
morriswalters
 
Posts: 2416
Joined: Thu Jun 03, 2010 12:21 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby Griffin » Wed Nov 16, 2011 7:32 pm UTC

morriswalters wrote:Consciousness exists nowhere.

Of course, its an emergent system property, not innate physical one, so of course it has no exact physical location.

morriswalters wrote:It's a model of the world created by the brain.

This, I have some problems with, though.
1) Is this a personal definition, or do you have some evidence of it being generally accepted?
2) The robot I helped build a couple years ago had this, but I don't think anyone would call it conscious. So obviously there's more to it than that.
3) A model of the world still requires some method for interaction with the world, however imprecise and inaccurate (otherwise its a model of something else), doesn't it?
4) What value does this definition of consciousness lend to the discussion, other than being trivially easy to accomplish?

The computer doesn't care how fast it operates, the user does.

What user? Also, the computer doesn't care about anything. I don't see how "who cares" is relevant. The question is about what the computer does, does not what it cares about. In one case, the computer succeeds in discovering the correct answer (the number of apples at this moment), in another it fails. There is no user.
Bdthemag: "I don't always GM, but when I do I prefer to put my player's in situations that include pain and torture. Stay creative my friends."

Bayobeasts - the Pokemon: Orthoclase project.
User avatar
Griffin
 
Posts: 1364
Joined: Sun Apr 08, 2007 7:46 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby morriswalters » Wed Nov 16, 2011 7:55 pm UTC

1. We'll find out, but as of today no. Does it have to be to be discussed?
2. Why? Show me that it is more than a difference in degree. More to the point why did your robot have it? Also, what purpose does consciousness serve for the brain?
3. The answer to the last question in two. Consciousness serves as a point of reference. The model ties it all together.
4. The point is to show the problem that Searle tripped over. This will sound trite but it isn't. The model helps the brain to survive. With no point of reference, you are lost. But it doesn't help you to understand Chinese. The brain does that, and we don't know what it needs to do so we only know where some of it happens. Think about this. Put a patch over the eye of a split brain patient which understands the written word. Put a sign on the table that says "Radiation Hazard Here". Can consciousness help him?
As a disclaimer anything I say is my opinion and should not to be confused with fact.
morriswalters
 
Posts: 2416
Joined: Thu Jun 03, 2010 12:21 am UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby PM 2Ring » Wed Nov 16, 2011 9:20 pm UTC

As a Turing fan, I have a soft spot for the Turing machine, however, they aren't very good for modeling dynamic systems that interact with data that arrives while the program is running. All of the data that a TM acts upon is on the tape before you start the machine. So to model a dynamic system you have to somehow represent that dynamic data in a static fashion on the tape.

A Turing machine is not a practical computer. It's really just a "lowest common denominator" device to illustrate the universality of computable functions, ie if a function is computable in a finite number of steps by some mechanical computing device, then it's computable in a finite number of steps by any other mechanical computing device, given enough memory. Except for a few mostly trivial computations, it's rare that a TM approach will actually be an efficient way to perform the computation. The "clock frequency" of a TM isn't very relevant - it just has to be finite, since all we're interested in is whether the computation can be performed in a finite number of steps.


I don't believe that Searle is claiming that understanding is not computable: his issue is with the idea that there's no more to intelligence than mere language processing ability. He's saying that a system capable of processing language to the degree required to have a conversation (about stories written in Chinese) does not (necessarily) understand what it's saying, or mean what it says. He might be correct on that score, to some extent. But I suspect that such conversations would not be very satisfying, and an AI would need to do better than that to pass the Turing test. OTOH, humans are quite capable of having discussions about topics that they don't really understand, yet we're still happy to assume that such people are intelligent even if the discussion seems more like an exercise in word jugglery than an actual sharing of concepts and feelings...
User avatar
PM 2Ring
 
Posts: 2593
Joined: Mon Jan 26, 2009 3:19 pm UTC
Location: Mid north coast, NSW, Australia

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby Anaphase » Wed Nov 16, 2011 10:44 pm UTC

Griffin wrote:
Processing speed has nothing to do with anything, as long as the internal process is consistent within itself. Computational speed is only relevant if the output is needed in a specific finite period of time.
You keep saying it has nothing to do with anything, but you haven't said why. The question was simply "are there things one TM can do that another one can't?". Answering with "No, not counting the situations where its yes" is what doesn't mean anything.

The answer is that a universal Turing machine can always do anything that any other Turing machine can. When people say "Turing machine" it's usually as a shorthand for UTM, while there are Turing machines that aren't UTMs there is nothing they can do that a UTM can't.

But Griffin's point was that there are in fact things which some TMs can do while others can't, because the internal timing and the timing of inputs don't match up properly.

As PM 2Ring said, TMs don't have inputs or outputs.
Anaphase
 
Posts: 33
Joined: Fri Oct 21, 2011 7:20 pm UTC

Re: The Assumptions of Searle’s Chinese Room Thought Experim

Postby Griffin » Wed Nov 16, 2011 10:59 pm UTC

Well then, if we're saying TMs can't have input and output, the whole argument is finished and the initial point of "there are problems computers can solve that Turing Machines can't" stands as de facto true. In other words, they can't solve any problem that requires an input or output - they can only solve static problems.

To repeat the very first thing I said:
There are problems some computers can solve that some other turing machine can not solve.

Which may have been the source of the confusion, as I have realized I made an error here. I should have specified this as "a turing machine" instead of "some other turing machine" to make things clear, since computers are not turing machines.

Which was a response to
First, that's wrong. If it is computable, it is computable by a Turing machine.


I made the concession that while technically true, it was only in regards to a limited computer-science version of "computable" - one which requires an operation can be implemented on a TM.

My whole point was that there are problems that are solvable that are not "computable" in the strict sense of the term.
Bdthemag: "I don't always GM, but when I do I prefer to put my player's in situations that include pain and torture. Stay creative my friends."

Bayobeasts - the Pokemon: Orthoclase project.
User avatar
Griffin
 
Posts: 1364
Joined: Sun Apr 08, 2007 7:46 am UTC

PreviousNext

Return to Serious Business

Who is online

Users browsing this forum: No registered users and 4 guests