[LINK] Centrelink/DFCS computer boondoggle story

[Roger Clarke]

Jan:
>   From the Age, 15 April 2005:
http://theage.com.au/news/National/The-64m-question-how-was-it-wasted/2005/04/14/1113251737955.html
>The $64m question: how was it wasted?
...
http://www.anao.gov.au/WebSite.nsf/Publications/09678CB7AE1F3751CA256FE10082
495F

pjchen at optusnet.com.au:
>Any link gurus got comments on the use of ES in this area?  Is 8000 rules a
>lot or litting is ES land?

A guru's a person with a white beard, right?

Back when Adam was a pup, in 1989, I did this one:
Knowledge-Based Expert Systems:  Risk Factors and Potentially 
Profitable Application Areas
http://www.anu.edu.au/people/Roger.Clarke/SOS/KBTE.html

It's interesting to re-read it after all these years, and apply it in anger:

"Non-deterministic models are therefore an important risk factor"

"It is necessary that the domain be sufficiently well-defined and 
bounded that the knowledge engineer be able to commence with, or at 
least develop, a clear conception of the classes of problem which are 
to be addressed, and equally importantly those which are not to be 
addressed"

"It is necessary that there be a sufficient degree of stability of 
the domain phenomena.  ...  KBT may not be of much assistance in 
dealing with a quickly-mutating flu virus, because the delay in 
collecting and interpreting new data may be such that by the time the 
knowledge has been formulated, its brief period of usefulness has 
expired"  [do you like the idea of welfare policy as flu-virus?]

"The domain knowledge must also be expressible, in the sense of being 
able to be conveniently and accurately mapped onto some formalism"

"It is important that the domain-specialist(s) be eager, willing, or 
at least amenable to take part in the project, not just in principle, 
but also in practice, through the long haul of preliminary 
discussions, education of the knowledge engineer, assessment of the 
rules the knowledge engineer formulates, and testing of the 
knowledge-base against known cases. Behavioural difficulties will 
arise"

"the domain specialist(s) must be able to express the knowledge in a 
manner which the knowledge engineer can understand. This is no 
trivial matter, since most specialists have hitherto sensed or imaged 
a great deal of their knowledge, and never expressed it outwardly, 
especially in the exhaustive and precise manner in which the 
knowledge engineer seeks to capture it"

"There may also be a problem of internal inconsistency among the 
rules expressed by the specialist or adduced by the knowledge 
engineer. For example, particularly in large knowledge-bases, several 
rules may be expressed in such a way that they lead to different 
conclusions, which are (or logically should be) mutually exclusive"

"In the development of 'community knowledge bases' (Bobrow et al 1986 
p.893), where more than one domain specialist is involved, 
inter-specialist inconsistency may arise. This may be because the 
knowledge of a single domain is shared across several people 
(typically because each has experience of a sub-set of, for example, 
the customers, geographical areas, or product groups). Or it may be 
because the application is intended to straddle two related domains, 
and essentially requires the knowledge of two specialists to be 
combined. In less common cases (at least as far as can be seen in the 
literature to date), it may be the intention of the organisation 
commissioning the application to reconcile the, in part conflicting, 
expertise of two or more specialists in the same domain. A tempting, 
though probably futile, application would be macro-economic policy 
formulation"  [Which do you reckon is harder, macro-economic policy 
or social welfare policy??]

"In many applications, the need for significant modifications will 
arise, sometimes because of environmental change, in other cases as a 
result of ongoing learning by the domain-specialist or the 
organisation as a whole"

"Professional people (actuaries, financiers, insurance assessors, 
engineers, architects, solicitors, public accountants, etc) generally 
regard the exercise of their professional judgment as being critical 
to their occupation and to their self-esteem. Genuinely expert 
applications are likely to find little success in such circumstances, 
and even adviser applications are highly likely to be resisted. 
Approaches to avoiding such reactions are the (at least symbolic) 
involvement of all affected people in the design process; depiction 
of the knowledge-base as a professional colleague, or as an assistant 
which has been trained by a colleague; and delicate handling of 
contentious rules, and of differences of opinion among domain 
specialists"

"As with with any new investment requiring time to mature, the 
commitment of senior management is essential, to the technology as a 
whole, and to individual projects. Otherwise, as problems arise, 
delays occur and expectations are lowered, there is a risk that 
individual projects and even the entire investment may be abandoned 
just as the breakthrough is being neared"  [I didn't express the 
other side of the coin:  that over-commitment, pride, apathy, 
incompetence or lack of concentration might result in investments 
*not* being abandoned when break*down* is being neared.  I left it 
out because it was the bleeding obvious - every major project since 
the 1960s has faced that risk]

Even in those days, "In the services sector ..., projects have been 
commissioned to interpret and apply legal regulations, e.g. of 
government benefits, immigration and building"

[Moreover, at least one company has been successful in this 
particular space, and did a lot of work on policy databases for the 
old DSS back in the early 1990s:  http://www.softlaw.com.au/.  So 
there should have been a lot of expertise in this area long before 
this particular project started.  It's hardly a new phenomenon that 
politicians pass laws, and that agencies are mini-empires and their 
denizens are jealous of one another.

[What disturbs me is not actually the failure of the project.  It's 
the abysmal failure in governance;  and it's the scale of the 
expenditure, firstly at all, and secondly even after it was 
abundantly clear that there were serious problems with the project.

-- 
Roger Clarke              http://www.anu.edu.au/people/Roger.Clarke/
			            
Xamax Consultancy Pty Ltd, 78 Sidaway St, Chapman ACT 2611 AUSTRALIA
                 Tel: +61 2 6288 1472, and 6288 6916
mailto:Roger.Clarke at xamax.com.au            http://www.xamax.com.au/

Visiting Professor in the Baker Cyberspace Law & Policy Centre, UNSW
Visiting Professor in the eCommerce Program, University of Hong Kong
Visiting Fellow in Computer Science,  Australian National University