Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!uunet!husc6!rutgers!sri-spam!ames!elroy!mahendo!jplgodo!wlbr!scgvaxd!trwrb!aero!venera.isi.edu!smoliar
From: smoliar@vaxa.isi.edu (Stephen Smoliar)
Newsgroups: comp.ai
Subject: Re: Can you sue an expert system?
Message-ID: <4260@venera.isi.edu>
Date: Mon, 7-Dec-87 11:54:08 EST
Article-I.D.: venera.4260
Posted: Mon Dec  7 11:54:08 1987
Date-Received: Sun, 13-Dec-87 12:05:55 EST
References: <1788@cup.portal.com>
Sender: daemon@venera.isi.edu
Reply-To: smoliar@vaxa.isi.edu.UUCP (Stephen Smoliar)
Organization: Information Sciences Institute
Lines: 51

In article <1788@cup.portal.com> Barry_A_Stevens@cup.portal.com writes:
>
>Consider, and please comment on, this scenario.
>
>                     * * * * * * * * * * *
>
>A well-respected, well-established expert systems(ES) company constructs
>an expert financial advisory system. The firm employs the top ES
>applications specialists in the country. The system is constructed with
>help from the top domain experts in the financial services industry. It
>is exhaustively tested, including verification of rules, verification of
>reasoning, and further analyses to establish the system's overall value.
>All results are excellent, and the system is offered for sale.
>
Anyone who is willing to accept these premises at face value may be more
interested in investing in the bridge I have between Manhattan and Brooklyn
than in expert systems.  The sort of "ideal" product envisaged here is
certainly beyond the grasp of current development technology and may remain
so for quite some time.  The most important omission from this scenario is
the assumption that any sort of disclaimer has been attached to the product.
I have encountered a variety of advertisements for human financial consultants;
and, as a rule, there is always some disclaimer about risk present.  The
idea that their would be a machine-based product which would be risk-free
borders on ludicrous.  If a customer was hooked by such a claim, most likely
the only place he would be able to complain would be to the Better Business
Bureau.
> 
>By now, you know the outcome. On the Friday morning before Black Monday,
>the expert system tells Joe to "sell everything he has and go into the
>stock market." ESs can usually explain their actions, and Joe asks for
>an explanation. The ES replies "because ... it's only been going UP for
>the past five years and there are NO PROBLEMS IN SIGHT."
> 
Would Joe have accepted such an explanation from a human advisor?  If so,
he has gotten what he deserved.  (I happened to be discussing an analogous
case with my lawyer-neighbor.  Our scenario involved medical systems and
malpractice, but the theme is basically the same.)

This raises another question:  Assuming Joe is no dummy (and that he can
afford good human advice), why would he be intersted in an machine advisor?
I would argue that the area in which machines tend to have it over humans
is that of quantitative risk assessment.  Thus, the machine is more likely
to synthesize and justify concrete quantitative predictive models than is
a human expert, whose skills are fundamentally qualitative.  Thus, the best
Joe could hope for would be such a model.  INTERPRETING the model would
remain his responsibility (although that interpretation may be linked to
the machines justification of the model, itself).

I would conclude that this scenario is far too simplistic for the real world.
I suggest that Mr. Stevens debug it a bit.  Then we might be able to have a
more realistic debate on the matter.