ARTIFICIAL INTELLIGENCE AND PUBLIC POLICY ANALYSIS

(DRAFT)

copyright 1998

Computers are making critical decisions under the supervision, but without the aid, of human beings in an increasing number of industries. Today, machines, not only approve individual credit card charges, but also decide whether to offer new cards to specific individuals. "Smart Bombs" are performing complex targeting tasks formerly rendered by humans. Airline schedules are being created by computers rather than people, and a wide variety of design tasks--in fields ranging from clothing to architecture and to drugs are conducted by thinking machines rather than people. Software programs, rather than appraisers, are taking information from massive databases to determine the value of an increasing number of homes for lending institutions. Artificial intelligence is also being used in medical diagnoses, in purchasing, in helping autistic children, in inventory control, and in nuclear engineering. Finally, Big Blue crushed many humans by defeating the finest chess player in the world. Inevitably, more and more decisions will be made in this manner.

DEFINITIONS

Computer-based management information systems gather, process, and analyze information for planning, operational, and decision making purposes. Examples of such systems range from airline reservation systems to accounts receivable software on a personal computer for the small business manager. Such systems aid in the day-to-day business of running the organization. Increasingly, modern organizations are using data bases, decision support systems, and expert systems to make strategic decisions. For example, decision support systems help city planners determine how best to allocate city funds to mitigate the effects of earthquakes.

Artificial intelligence (AI) has been defined as the subfield of computer science concerned with the concepts and methods of symbolic inference by computer and symbolic knowledge representation for use in making inferences. It can be seen as an attempt to model aspects of human thought on computers, allowing computers to act in ways that mimic human intelligence. The line between decision support and artificial intelligence systems is not always clear. Examples of artificial intelligence formats include neural networks, genetic programming, fuzzy computing and artificial life.

APPLICATIONS

Examples of problems tackled by AI, in addition to those cited above are robotics, computer vision, speech translation, and natural language processing. In "Neural network applications in business: A review and analysis of the literature (1988-95)" Bo K. Wong, Thomas A Bodnovich, and Yakup Selvi analyze applications of AI published in journals. They find that the number of articles containing applications rose fairly steadily from 1 in 1988 to 3 in 1989 and 1990 to 39 in 1994 and 78 in 1995. Applications were classified as follows

Web-search services often return thousands of responses to a simple request for information -- many of which appear to bear little or no relation to what you are seeking. EchoSearch uses artificial intelligence to analyze the Web pages to try to detect concepts and meaning, not just whether certain search words are found, and then returns its findings in a summary.

The Food and Drug Administration approved a new computer-aided test, made by Neuromedical Systems Inc., designed to improve the accuracy of Pap smear screening for cervical cancer. The Papnet test, which uses a neural network, re-examines Pap smears that are judged free of abnormal cells. If it finds abnormal cells, it displays them on a high-resolution color video screen for interpretation by a technician. The test, which is expected to cost $35, increases the detection of abnormal cells by 30%.

Many large organizations, including the White House, Disneyland and Ford Motor Co., have contracted with a company that uses artificial intelligence to analyze resumes, categorizing, for example, the applicant's primary work experience. Other businesses use less sophisticated technology that simply searches for keywords. The organizations use optical scanning devices to enter resumes into computers, then electronically search for the ideal candidate.

FINANCIAL APPLICATIONS

AI has proven useful for numerous financial applications. At least four artificial intelligence systems: Lending Assistant, Credit Assistant, CUBUS, and MOCCA are utilized in credit analysis.

The computer-assisted real estate appraisal system (CAREAS) and EVIAN's neural net search through multitudes of data to derive home values.

According to The Wall Street Journal Fidelity Investments' Brad Lewis developed a computerized artificial-intelligence system to run or assist Fidelity's Disciplined Equity, Stock Selector and Small Cap Funds. That 'black box' helped the fund -- currently with $2.1 billion in assets -- outdistance Standard & Poor's 500-stock index for six years in a row, from 1989 through 1994. Mr. Lewis boasted that the fund was an 'index killer' that could consistently beat the index through astute, automated and ever-improving stock selection. The 'artificial intelligence' component 'learns' from the market and ferret out subtle patterns in how the market values different stocks. However, in 1995 and 1996 Disciplined Equity trailed the S&P benchmark. Benham Income & Growth Fund, Dreyfus Disciplined Stock Fund and Vanguard Quantitative Portfolios are three funds that rely heavily on quantitative tools and that have been strong performers in recent years. All three outdistanced the S&P 500 in the five years through October, 1996 according to Morningstar -- a feat matched by only 10% of diversified U.S.-stock funds. S. B. Carr Investments Inc., manages $15 million in client money using a database to screen out various financial characteristics of stocks, then rank the stocks with a Multi-Factor Ranking System. The computer studies the financial characteristics for factors that are predictive then assigns weights of its own.

OptiMark Technologies Inc. recently announced a trading system aimed at institutional investors. 'Crossing networks' act as large electronic black boxes, in which numerous buy and sell orders come together at fixed prices to be matched electronically. Crossing networks either use fixed prices, such as a closing price, or allow customers to fix the price of their request, like a limit order, to match their trades. The OptiMark system, using artificial-intelligence technology, allows for a more fluid movement of price by taking into account subjective concepts such as 'satisfaction' and 'trading-aggressiveness profiles.' 'The key is that this system, housed in a supercomputer, will permit large investors to expose certain aspects of their strategy and their size to the marketplace, without creating an unfavorable price impact'.

The insurance industry's most common application of AI comes in the form of expert systems, proliferating in fraud detection in underwriting and claims applications, as well as in target marketing of insurance products, new business processing and field application validation systems. One insurer utilizing an expert system for evaluating risk exposure is The Hartford Steam Boiler Inspection and Insurance Co., which evaluates safety for equipment and processes used by large government and industrial businesses.

ARTIFICIAL INTELLIGENCE AND GOVERNMENT POLICY

Although computers have assisted in a number of ways with policy decisions, they have not often been used to actually make decisions. This is slowly changing.

Tulare County is one of California's least affluent counties (the median household income is $24,500). The unemployment rate averages 15 percent, and out of a population of 320,000, some 95,000 are on public assistance. Starting in 1988, Tulare County left vacant positions open, and reinvested the money in a system that allows potential public assistance recipients to be interviewed by computer. The system uses, on average, only one employee per five stations. The employees start the system and are available, as needed, to answer questions. Using video-disk technology, an individual can respond to application questions regarding dependents, income, and employment by touching the computer screen. When the interview has been completed, the computer makes an initial determination of eligibility using artificial intelligence. An application is printed with the blanks already filled in and reviewed by a social worker for a final determination of eligibility. A separate module lets users know the rights and responsibilities of AFDC recipients. The system has been used to screen AFDC clients since 1990 and was expanded to screen potential food stamp recipients in 1991. Although exact figures are not available, Tulare's human services administrative costs per recipient are now the lowest or next-to-lowest in the state for every category of service, according to Charles Harness, chairman of the county board of supervisors.

Police in St. Petersburg, Florida, use a pair of computer databases to root out causes of crime, sharing their conclusions with one another -- and with citizens. The first database contains crime statistics. The system sorts each type of crime by various geographic breakdowns, including individual address, and community policing district. The data reveal crime "hot spots" --helping police home in on underlying problems that cause repeat calls for service. The second database reports how they're combating those problems. It's sorted by type of problem, geography and other variables. Officers get this information out to citizens, help one another solve pending problems, and learn from closed cases. Citizens access the databases at three "community resource centers" located in small shopping centers.

States are using problem-solving computer software -- "expert systems" -- to help speed unemployment assistance to disaster stricken workers. Texas developed the "Lone Star Software" prototype after a 1989 cold-snap hit the agricultural Rio Grande Valley -- freezing more than 10,000 workers out of their jobs. Because eligibility regulations aren't easily grasped by the uninitiated, temporary workers couldn't help with the overflow of claims-seekers -- resulting in long lines, inconsistent legal interpretations and short tempers. But expert systems use if-then constructs to "think" through routine claims, so temps can handle them -- significantly increasing processing capacity and uniformity while cutting waiting time. Experienced workers are freed to concentrate on trickier cases

Revenue Canada identifies non-compliant tax returns using a software tool called Neural Works Professional II Plus. A Processing Review and Automated Techniques team uses historical cases where noncompliance has been identified to train the neural network to make predictions about future cases.

The United States Coast Guard schedules weekly assignments of cutters by computer to ensure patrol coverage, enforce equitable distribution of patrols, and honor restrictions on consecutive assignments in a manner superior to manually prepared schedules.. The cutters primarily respond to calls for search and rescue, law enforcement, and pollution control.

Multiple-objective decision making (MODEM) provides an effective framework for integrated resource assessment of agroecosystems. This framework can add noneconomic objectives as constraints in an optimization problem, and evaluate tradeoffs among competing objectives. It has been utilized to evaluate a crop farm and watershed in northern Missouri. An interactive, spatial decision support system made the MODEM framework accessible to unsophisticated users. The system assesses the socioeconomic, environmental, and ecological consequences of alternative management plans for reducing soil erosion and nonpoint source pollution.

A computer-based early warning system--the Non-life Early Warning System (N.E.W.S.)--is used by the Dutch Insurance Supervisory Board (ISB) to improve the supervision of Dutch non-life insurance companies by identifying possible problem companies at an early stage of the evaluation process. It integrates a rule-based expert system, an ordered logit model, and a neural network.

Officials at law enforcement, defense and intelligence agencies have suggested creating a sophisticated computer program to screen records of the more than 700,000 electronic money transfers involving U.S. institutions each day and to flag suspicious ones for further investigation through artificial intelligence.

The Emergent Solutions Group of Coopers & Lybrand Consulting is using "Hollywood graphics" featuring the same computer animated imagery used in Jurassic Park and Toy Story and complexity theory to assist urban and rural planners with policy decisions. "Multiple autonomous agent technology" attempts to reproduce the choices made by individual businesses, governments, households or individiuals. From the autonomous, self-directed activities of these individuals (not known in advance) group behaviors emerge. The software simulates how zoning laws, conservation programs, economic policies and other land use regulations could change local landscapes and affect residents. It produces a 3-dimensional world where planners can see potential consequences of decisions one-to-ten years down the road. The first such program under development is the "Vermont Rural Lands Simulation" program.

The above programs assist government in the implementation of policy, but steps are being taken to actually have computers guide policies. We conducted an extensive search of computerized databases, including EconLIT, Public Affairs (PAIS), the Wall Street Journal, the New York Times, Carl's Uncover and utilized five major internet search engines. We uncovered very few examples which utilize artificial intelligence to devise policies.

COST BENEFIT ANALYSIS

In order to implement the use of artificial intelligence in policy analysis computers must be furnished with decision criteria and a base of knowledge. Two tools can be utilized to provide this--Cost Benefit and Meta Analysis.

Cost Benefit Analysis merely seeks to tally expenses and gains from policies. It has been used by public and private entities. Benefits for private companies are essentially profits, which are measured relatively easily by market derived revenues less costs of inputs purchased in other markets.

However, it is not as easily applied for public projects. Public goods are rarely sold, and when transactions do occur it is generally under monopolistic circumstances. In making public policy decisions and applying cost benefit analysis policy makers must determine the value of clean air, of lungs damaged by pollution, and of reduced crime without the benefits of market transactions. A host of techniques and models have been devised to overcome the lack of market prices, including "the contingent valuation method", "hedonic valuation models", "wage differential studies", indications from insurance expenditures, and tallies of costs. Despite the difficulties, recent federal and state legislation and court decisions have mandated its use.

Hundreds of cost-benefit studies have been commissioned by various levels of government, but these have generally isolated one-shot policies under specific circumstances and are not generally subject to automation.

Cost benefit analysis has not often been employed in part due to the expense and time involved in measuring complicated phenomena. Given the heterogeneity of policies, costs, and benefits, governments seeking to comprehensively measure impacts must typically spend tens or hundreds of thousands of dollars, and wait months or years for the results. Policies are thus based upon feelings and hunches--while taxpayers and potential beneficiaries may suffer the consequences.

In addition, when decision makers do employ cost benefit analysis, they often omit significant gains and losses due to difficulties typically encountered in collecting information. For example, one of the greatest costs of crime--fear--is rarely considered in analyses of criminal justice policy. The failure to measure crucial elements can render conclusions implausible. Furthermore, analysts also often overlook information from some studies in an age when data expands exponentially.

Automation of this process can both reduce costs and broaden the scope of these analyses, thus improving the qualityovercome these difficulties.

One way computers can automate the policy process, reduce costs, and increase credibility is by facilitating the use of information from previous studies.

META ANALYSIS

Meta analysis uses the results of other studies as the data for statistical analysis to distill systematic conclusions regarding relationships between variables. Data are collected from a series of studies, which are then converted to a common scale and reanalyzed using various statistical techniques. The basic null hypothesis underlying the model is that all estimated values of the dependent variables (made comparable by appropriate adjustments if necessary) are equal to a grand mean under specified design conditions. Statistical testing of the null hypothesis can use a variety of procedures such as cross tabulations, order statistics, and t-tests. Simple averages are used but MANOVA nonorthogonal Multivariate analysis of variance is perhaps the most popular test. Regression analysis is often used to explain differences in results.

This form of research was created in the 1970's by psychometricians. It is becoming increasingly popular among social and biological scientists. The results of combining studies are reported with increasing frequency in major journals and daily newspapers, particularly with respect to medical controversies.

A search of the Carl's Uncover database of scholarly journals turned up 1,546 references to meta analysis, and 27 books were found in the Library of Congress Catalogue. The vast majority of these references were in the field of medicine and related disciplines where life and death decisions are being made on the basis of meta analyses.

Economists have recently begun to add this tool to their bag of tricks. However, relatively few meta analyses have made their way into journals. A search of Carl's Uncover using the keywords meta analysis, and economic(s), value(s), labor, output, GNP, consumption, capital, finance, investment, costs(s), benefit(s), trade, real estate, property, government, and policy revealed only 14 references (excluding extraneous results)

A number of attempts have been made to begin to bring artificial intelligence to policy analysis by computerizing cost benefit and meta analysis.

PRECURSORS OF ARTIFICIAL INTELLIGENCE SYSTEMS FOR POLICY ANALYSIS

"Pontis"

"Pontis" provides a systematic methodology for allocating funds, evaluating current and future needs of bridges and options to meet those needs and recommending the optimal policy for each bridge in the context of overall network benefits, budgets and restrictions. After a trial implementation in California and extensive testing in several states, the system was adopted by the Association of American State Highway Officials. At the heart of Pontis is a set of predictive and optimization models which derive their information from judgmental, engineering and economic models and various databases.

The Externalities Project

In order to improve environmental policy decision making the Empire State Electric Energy Research Corporation, the New York State Energy Research and Development Authority, the New York Public Service Commission, Resources for the Future, and the Electric Power Research Institute sponsored the creation of a computer model designed to estimate environmental externalities for new and relicensed electric resource options. The final report, released in January, 1995, was prepared by the Tellus Institute and RCG Hagler, Bailly, Inc. The multimillion dollar project was managed by Stephen Bernow of Tellus and Robert D. Rowe of RCG.

The study used a damage function approach--a multi-step staged calculation process. After a site and generation technology have been selected the model:

1) Calculates emissions

2) Estimates the dispersion pattern of the emissions

3) Forecasts impacts on receptors (such as decreased crop production, or additional asthma attacks)

4) Values impacts in monetary terms

The project produced a report featuring extensive methodological discussions, a user-friendly and well-documented computer model (EXMOD), information on the uncertainties associated with estimates, a large bibliographic data base, and case studies.

Emissions considered include carbon dioxide, carbon monoxide, CholoFluoroCarbons, fine particulates, methane, nitrogen oxides, nitrous oxide, sulfur dioxide, volatile organic compounds, arsenic, beryllium, cadmium, chromium, dioxin, formaldehyde, lead, mercury, nickel, PCB's, and 12 types of water emissions. Externalities analyzed include human health effects of air pollutants, contamination of ground water from ash disposal, reductions in crop production from changes in ozone, visibility impairment, and impacts upon fishing.

The geographic units of analysis are groupings of census tracts, each of which contains detailed air quality, demographic, geophysical, meteorological, and crop production data. EXMOD also presents distributions of values with probability ranges rather than single point estimates.

TER

William H. Desvouges, F. Reed Johnson, and H. Spencer Banzhaf (DJ&B) of Triangle Economic Research created a similar model. The authors examined six types of pollutants: particulate matter, sulfur dioxide, carbon monoxide, nitrogen oxide, lead and ozone. Effects considered include human health impacts (morbidity and mortality), agricultural consequences (reduced crop yields), and materials damaged (stone and metal corrosion and surface soiling). The authors reviewed over 400 individual health studies as well as EPA criteria documents and agricultural studies.

To obtain externality costs for each scenario the model 1) assessed ambient air quality, 2) predicted emission quantities and compositions for specific generation scenarios, 3) modeled the transport and dispersion of these emissions, 4) calculated the exposures to people, crops and materials, 5) assessed potential injuries resulting from these exposures and 6) estimated the willingness to pay to avoid these injuries.

The authors also used meta analysis techniques to synthesize information. The meta analyses also provides a means to estimate a range of valuations so that uncertainty can be quantified.

DJ&B also utilize "health state indexes" to derive value estimates for the full range of health effects. These indexes--developed by health scientists to prioritize treatment of physical conditions--combine many attributes of health (such as comfort and mobility) into a single scale. DJ&B also explore the relationship between health state and willingness to pay for health effects when willingness to pay estimates are available. For those effects that have not been valued, they project a value based on the index score associated with the effect.

The study also uses a comprehensive database and modeling system for agricultural policy analysis developed by the Food and Agricultural Policy Research Institute (FAPRI) The model utilizes county level information to predict yield trends, acreage planted, acreage harvested, state price linkages, value of production and deficiency payments. The often significant effects of deficiency payments, the 1985 Farm Bills O-92 voluntary acreage reduction program, and the conservation reserve program are incorporated. The model provides production values and deficiency payment estimates for given ambient air pollution levels.

Finally the study uses a Monte Carlo simulation to estimate uncertainty. A Monte Carlo simulation takes estimated ranges for all parameters, randomly selects a value from each of these ranges, and then combines the estimates. The result produces one possible damage estimate. This sequence is repeated 400 times. The resulting distribution of outcomes yields an expected value, and an estimate of the 90 percent confidence interval around the most likely value. A total of 18 million calculations are made for each scenario.

Damages are a function of quantities generated and the toxicity of pollutants. .

EC

The European Commission (EC, 1995) used a similar model to estimate externalities for nine fuel cycles (coal, lignite, biomass, nuclear, oil, natural gas, photovoltaic, hydro and wind) for West Burton in the U.K.

and Lauffer in Baden Wurttemberg in Germany.

ACB

The ACB system is a modular menu driven computer program that produces several tallies of costs and benefits arising from government policies. Tallies are based upon data from previously published studies. A decision maker can change assumptions, and add costs or benefits through a simple spreadsheet format. The system can also serve a bibliographic function--rapidly leading analysts to sources of information.

The resources devoted to the other models were far greater. However, the ACB model is designed to analyze a far wider range of policies, and permits users to update study coefficients with new information more readily than EXMOD. Its spreadsheet format may also be easier to revise than the other models. Potentially, it can be used to compare the costs and benefits of different types of expenditures. Applications to crime and environmental policy have been created. The system asks the user to select:

1) A Policy Area (such as CRIME, ENVIRONMENT, or HOUSING)

2) A Function To Be Performed (e.g. VIEW/ALTER COST BENEFIT TALLIES, or ENTER DATA)

3) The Amount Of The Proposed Expenditure

4) The Geographic Region Under Analysis

5) A Specific Policy (such as HIRING POLICE or INCREASING THE SEVERITY OF SENTENCES)

If VIEW/ALTER COST-BENEFIT SUMMARIES is chosen, the system presents the policy maker with several possible scenarios. The scenarios provide different means to estimate costs and benefits.

For crime the rows of cells within the scenarios:

1) present statistical data (coefficients) from studies showing the extent to which a policy such as hiring police tends to produce an effect, say increased arrests or convictions.

2) presents coefficients found in studies measuring the extent to which the change in the effect (increases in arrests or convictions) changes the level of a disamenity (e.g. crime), and

3) multiplies (1) and (2) resulting in the extent to which a policy cuts the disamenity

4) multiplies the results in (3) by estimates of the costs imposed on society by the disamenity (crime), which comprise the benefits of a policy.

5) adds in the effects of government spending multipliers

6) compares the benefits estimated above to costs

The system

1) contains a database filled with coefficients and other information derived from previous studies;

2) converts the units of each study into a common base;

3) places the data from studies that investigate the same phenomena together by attaching a variable name to each result in the database;

4) derives a single representative (weighted average) number for each variable by weighting the elements of the studies;

5) produces a series of potential cost benefit scenarios looking at causes and effects, as well as costs and benefits, in different ways based on the weighted variables. The system provides many potential weighting schemes, and allows the user to modify the weights. One scheme attaches greater significance to studies conducted in more recent years. Other weights are based on the apparent quality of the study, and the plausibility of results.

OTHER PRECURSORS

Other computer applications useful for computerized cost benefit models include econometric forecasting and impact analysis models, such as the Federal Reserve, DRI and Wharton models, and input models such as REMI and IMPLAN

BIBLIOGRAPHY

European Commission (1995) Externalities of Energy: ExternE Project. For the Directorate General XII, Prepared by Metroeconomica, CEPN, IER, Eyre Energy-Environment, ETSU, Ecole des Mines

Forbes ASAP "Information Technology Improves the Welfare and Health Services Application Process in Tulare County, CA" October 25, 1993, pp. 151 For More Information:Ian Strachan, 209-733-6537, fax 209-730-2568 Charles Harness, 209-733-6271, fax 209-733-6318

Hagler Bailly Consulting, Inc. (1995) The New York State Externalities Cost Study (Dobbs Ferry, New York) Oceana Publications

Krupnick, Alan J. and Dallas Burtraw "The Social Costs of Electricity: Do the Numbers Add Up" Resources for the Future Discussion Paper 96-20 May, 1996 1616 P Street, NW Washington, DC 20036 (202) 328-5000 http://www.rff.org

Lee, R. (1995) Externalities Study: Why Are the Numbers Different? Draft Paper prepared for the Third International Workshop on Externality Costs, Ladenburg, Germany, May 27-30

Lee, R., A.J. Krupnick, D. Burtraw, et.a l. (1995) Estimating Externalities of Electric Fuel Cycles: Analytical Methods and Issues, Estimating Externalities of Coal Fuel Cycles and additional volumes for other fuel cycles (Washington, D.C.), McGraw-Hill/Utility Data Institute

The Public Innovator ""Prevention rather Than Cure: Ace of Database" Issue No. 23, February 23, 1995

The Public Innovator, "Getting Results: Silver Lining" Issue No. 23, February 23, 1995

Regional Economic Research Inc. (1991) Valuing the Environmental Impacts of Alternative Energy Resources, prepared for the California Energy Commission, Sacramento, California, prepared in San Diego, California

Triangle Economic Research (1995) Assessing Environmental Externality Costs for Electricity Generation, prepared for Northern States Power Company, Minnesota Durham, N.C.

ACT SIGART, University of California at Davis, CMU Artificial Intelligence Repository.

Golabi, Kamal; Shepard, Richard "Pontis: A system for maintenance optimization and improvement of US bridge networks" Interfaces Vol: 27 Iss: 1 Date: Jan/Feb 1997 p: 71-88

Delahanty, Paul "Government using neural nets to snag tax cheaters" Computing Canada Vol: 22 Iss: 16 Date: Aug 1, 1996 p: 6

Erginbilgic,-Fusun"Artificial Intelligence Applications in Credit Analysis: An Overview with Four Examples" Middle-East-Technical-University-Studies-in-Development; 22(2), 1995, pages 135-47

Dave Kansas "Optimark Technologies to Announce Trading System Aimed at Institutions" The Wall Street Journal Sep 24, 1996 Sec: C p: 16

Schmerken, Ivy "A money manager and his self-learning machine" Wall Street & Technology Vol: 15 Iss: 2 Date: Feb 1997 p: 70

Schwartz, Susana "Artificial intelligence: Reviving the love affair" Insurance & Technology Vol: 21 Iss: 8 Date: Aug 1996 p: 42-44

Brown, Gerald G; Dell, Robert F; Farmer, Robert A "Scheduling Coast Guard District cutters" Interfaces Vol: 26 Iss: 2 Date: Mar/Apr 1996 p: 59-72

Prato,-Tony, Chris Fulcher, Shunxiang Wu, and Jian Ma "Multiple-Objective Decision Making for Agroecosystem Management" Agricultural-and-Resource-Economics-Review; 25(2), October 1996, pages 200-212.

Kramer,-Bert "A model for the evaluation of non-life insurance companies" Groningen: Rijksuniversiteit Groningen, 1996, pages vii, 248.

Falconer,-Lawrence-L.; Long,-Charles-R.; McGrann,-James-M. "A Decision Support Aid for Beef Cattle Investment Using Expert Systems" Journal-of-Agricultural-and-Applied-Economics; 28(1), July 1996, pages 180-92.

Agrell,-Per-J. "A Multicriteria Framework for Inventory Control" International-Journal-of-Production-Economics; 41(1-3), October 1995, pages 59-70.

Bart Ziegler "The Internet (A Special Report); How Can I Find What I'm Looking for?" The Wall Street Journal Dec 9, 1996 Sec: R p: 20

Karen Damato "Fund Track: a `Can't Miss' Mutual-fund Machine Sputters" The Wall Street Journal Nov 22, 1996 Sec: C p: 1

Silvia Ascarelli "Goodbye, Gutenberg: Germans Tap System to Translate Speech; but New Technology Stutters as Scientists Try to Cope with Narrow Vocabularies" The Wall Street Journal Nov 20, 1995 Sec: B p: 10

"Technology: Neuromedical's Pap Test Is Cleared by the FDA" The Wall Street Journal Nov 10, 1995 Sec: B p: 5

Margaret A. Jacobs "Law -- Legal Beat: Plans to Track Money Laundering by Computer Are in Spin Cycle" The Wall Street Journal Sep 13, 1995 Sec: B p: 2

Rochelle Sharpe "Work Week: a Special News Report about Life on the Job --and Trends Taking Shape There" The Wall Street Journal May 16, 1995 Sec: A p: 1

Detweiler, John H; Radigan, Ronald E "Computer-assisted real estate appraisal: A tool for the practicing appraiser" Appraisal Journal Vol: 64 Iss: 1 Date: Jan 1996 p: 91-101

Bo K. Wong, Thomas A Bodnovich, and Yakup Selvi "Neural network applications in business: A review and analysis of the literature (1988-95)" Decision Support Systems 19 (1997) pges 301-320