Go to
Brian Martin's publications on education
Brian Martin's publications on science, technology and society
Researchers often complain about research grant schemes, but usually within a narrow frame of reference. Looking more broadly, problems with grant schemes can be classified as bias, waste, discouragement and orientation to interests. There are various ways to allocate research funds, including administrative decision, peer review, performance-based allocation, equality and community-based bids. Each has different sorts of problems and serves different interests. By looking at diverse systems for allocating research funds, some of the assumptions underlying usual discussions become more apparent. Recent changes in Australian government policy on higher education research are examined using the framework provided here.
Any research grants scheme is likely to generate a great deal of informal complaint. Not surprisingly, dissatisfaction is most common among unsuccessful applicants, with complaints about bias and wasted effort. Administrators worry about making the system work efficiently. Reformers seek methods of making better decisions, for example by changing selection criteria and peer review systems, and ensuring accountability.
However, most of the discussions about grant schemes deal only with minor changes within the same basic structure. Yet the structure of a scheme is often the primary determinant of the sorts of problems that it generates. This article aims to provide a broader perspective on this issue. I start by outlining several key types of problems with grant schemes and then lay out several types of methods for decision making. With this foundation, it is straightforward to note the sorts of problems most commonly or likely to be associated with decision-making systems. Finally, recent changes in Australian government policy on higher education research are assessed using the broad framework offered here. (Some of the sources cited below deal with peer review for journals, but their assessments are likely to apply to peer review for grants as well.)
What constitutes a problem with a grant scheme of course depends on the observer. Hence the focus needs to be on perceived problems, acknowledging that widespread perception of a problem may be a problem in itself, whatever the 'facts'. Here, several different types of problem are outlined.
Among applicants, especially unsuccessful ones, allegations of bias are commonplace. They include the following.
To these possibilities can be added 'random bias' due to incompetence of administrators or referees.
Any grant scheme requires administrative overheads to assess applications. In addition, applicants may spend a large amount of time preparing applications. Even in an efficiently run operation, the cost associated with running the scheme and preparing applications can be a significant proportion of the money awarded in grants, especially if the grants are small and the success rate is low. In addition, not all grant schemes are run efficiently, aggravating the problem of waste.
Most of the attention in grant schemes is on those who are successful, but there can be significant effects on those who are unsuccessful. They can be disheartened by rejections or become resentful. This provides fertile soil for beliefs that decisions are biased, which help to alleviate the stigma of failure. Any competitive system creates this problem, whether it is grading in schools or promotion systems at work. There is considerable evidence that competition reduces intrinsic motivation (Kohn 1993). Even for more successful applicants, the goal of winning a grant may become more significant than doing the research.
The possibility of getting a grant provides an incentive to do research that pleases those dispensing the money. Grants provided by a corporation or government department for research in particular fields - telecommunications or pesticides - obviously orient researchers to particular problems. This is not necessarily a problem in itself, but can be seen to be one in the wider context of social priorities. In short, research is oriented to those who have money to dispense. Social problems which no one has a vested interest in solving receive few resources (Arditti et al. 1980; Dickson 1984; Martin 1979).
Many government grant schemes are oriented not to problems but to acquisition of knowledge. Grants go to applicants who best make the case that they are pushing back the frontiers in astronomy or brain structure. Even in these cases, it can be argued that there is an indirect orientation to outside interests. For example, problems in numerical analysis or oceanography may be influenced by military priorities; problems in organic chemistry or electrical engineering may be influenced by corporate priorities. This influence can occur through paradigms, potential applications of pure research, or job prospects.
Finally, researchers have a vested interest in their own careers, including positions and status. In as much as top researchers are influential in making decisions about grants, it is likely that the system of research - based on full-time professional specialists - will be perpetuated. This may be at the expense of other priorities, such as pressing social problems. The familiar example of researchers recommending further research is symptomatic of the problem that the grant system is oriented to the interests of researchers at the expense of others.
Researchers are so familiar with peer review - not least through refereed journals - that it may seem that there is no sensible alternative. But actually there are various possibilities. The following are five possibilities, chosen because they have the potential to deal effectively with one or more of the problems noted above. They are presented as ideal types. In practice, actual allocation of research funds typically combines elements of several of these methods.
In this ideal system, all decisions about research priorities and funding are made by top administrators, who may or may not be researchers themselves. In making their decisions, the administrators take advice from a range of groups: political and economic leaders, researchers, lobbyists and so forth. The approach is typical of research in large bureaucracies, especially government (in particular the military) and corporations. The justification for this approach is service to national, public or shareholder interest, which are often taken as synonymous. The research system emphasises cooperation and teamwork, the exemplar being the Manhattan project for producing the first nuclear weapons.
The patterns of research in the world today reflect high-level administrative decisions. This applies not just to most government and corporate research but also to much university research, since administrators make most of the decisions to decide the distribution of funds between different fields, provide research infrastructure, and to set up specific research centres and programmes. For example, the framework for deciding the allocation of research funding between telecommunications engineering and musicology within a university is normally decided by managers, rather than by peer comparisons, equal distribution or public opinion poll. There is, of course, no objective method to decide such an allocation.
Researchers and other pressure groups do have some influence on such administrative decisions. For example, researchers and consumer groups have some influence on the research agendas of electricity authorities and electronics firms. Although administrative decision is widely used, it has received relatively little critical attention compared to peer review; there are only a few researchers who enthusiastically advocate it (McCutchen 1977).
In this model, decisions about research priorities and funding are made by peers, namely professional researchers knowledgeable in relevant areas, in a competitive merit-based system. The justification for peer review is that peers are best able to judge the merit of research and that high-quality research is best able to advance knowledge and serve the public interest.
Peer review can be implemented in various ways. Applications can be sent to anonymous referees who rate them. Alternatively, applications might all be judged by a small panel without any outside comment. As long the panel members are knowledgeable in the field, this is still judgement by peers, though more open to accusations of bias. Some key elements in peer review systems are:
There is a large literature on peer review, and much discussion (Armstrong 1997; Campanario 1995; Chubin and Hackett 1990; Daniel 1993; Peters and Ceci 1982; Wessely 1998). However, for the purposes here the various options in peer review systems are secondary to the main point that ranking of competing applications is primarily based on judgements by peers.
After administrative decision, peer review plays a significant role in setting research priorities and making funding decisions. This operates in grants schemes and in decisions at the research project level within universities and, to a lesser extent, in some government and corporate research units. As noted above, administrative decisions play a central role in setting the framework for many peer-review decisions.
This method funds researchers according to the outputs that they have produced, such as publications and patents (Forsdyke 1993; Roy 1984). By defining the desired sorts of performance and defining the rewards, performance-based funding can be implemented without formal grant applications at all. For example, each paper in a specified journal (perhaps weighted by the journal's impact factor) could result in a defined payment to the authors. The research quantum allocated to Australian universities relies on a formula including weights for publications, research degree completions and funds received. This is similar to Roy's (1984) proposal for allocation of funds to groups of researchers based on group productivity. Obviously, peer judgements influence funding via the outputs, but performance-based funding differs from typical peer review systems in being based strictly on outputs rather than grant applications. One justification for performance-based allocation is to avoid direct biases based on reputation, gender or research findings.
Offering prizes for discoveries can be considered to be another type of performance-based funding. Nobel prize winners receive not only money but fame which often can be translated into further resources.
In this approach, every researcher gets either an equal amount of funding or an equal chance at funding. Some minimum requirements can be put on who is eligible, such as all staff at a university or in a research group. Among those who are eligible, available funding can be divided up in any of several ways.
A considerable proportion of research funds are currently allocated using this method, at least nominally. If university staff are expected to devote one third of their time to research, then one third of the budget for salaries could be said to be allocated using the method of rough equality (though since salaries are different, the nominal allocation is not equal). For some humanities research, for example, time and access to good libraries are the most crucial elements, and additional research funding is less significant. On the other hand, in many technical fields, salaries are only the beginning of what is required to undertake research.
In this model, research priorities and funding are decided by a range of community groups, such as groups of workers, parents or neighbours. The key feature is that users at the grassroots would make the decisions, rather than administrators or researchers. Currently this approach is used for only a limited amount of research, and therefore it is worthwhile spelling out briefly how it might work.
One procedure is to have a panel selected randomly from volunteers from user groups. The panel would hear submissions from researchers and other interested groups and then make decisions about research priorities and funding.
Another possible and rather different procedure is for user groups to prepare submissions for research to be carried out by particular researchers. The researchers, who would be expected to do a certain amount of research selected from the user-group applications, would choose which projects they preferred to undertake. This is analogous to the way university departments are expected to provide a certain amount of teaching selected from areas that students want to study. The justification for community-based allocation is to serve human needs.
So-called "science shops" - perhaps better described as knowledge shops - are the closest thing to community-based allocation. They take questions from community groups, trade unions, small businesses and other organisations without significant resources for research, help to turn the these questions into researchable topics, and seek to find university students or staff to carry out relevant projects (Farkas 1999; Zaal and Leydesdorff 1987).
Table 1. Each method of allocation is matched against each major type of problem.
|
Problem Method |
Bias |
Waste |
Discouragement |
Orientation to interests |
|
Administrative decision |
Potential bias in favour of insiders, dominant groups |
Preparation of proposals; decision system overheads |
Poorly supported researchers may become demoralised |
Incentive to serve interests of administrators |
|
Peer review |
Potential bias in favour of insiders, dominant groups, successful researchers |
Preparation of applications; grant scheme overheads |
Unsuccessful applicants may become demoralised |
Incentive to serve interests of granting body or peers |
|
Performance-based funding |
Potential bias in favour of successful and/or superficial researchers |
Grant scheme overheads (especially collection of performance data) |
Low output performers (including those with high quality) may become demoralised |
Incentive to serve interests associated with output measures |
|
Equal allocation |
Bias against those excluded from the equal allocation |
Money spent on those who are unproductive |
High performers may resent allocations to those who are unproductive |
Incentive for researchers to serve their own interests |
|
Community-based allocation |
Potential bias in favour of insiders, preferred groups |
Effort spent by community groups in finding researchers |
Researchers who are not sought after may feel unworthy |
Incentive to serve community interests |
Although no allocation system is free of bias, waste, discouragement or orientation to interest groups, the expression of these problems can be quite different with different systems. For example, community-based allocation provides incentives to do research serving quite different interests than administrative decision. The choice of an allocation system both reflects and shapes an ongoing connection between researchers and interest groups.
Most of the complaints commonly voiced about grant schemes - such as bias in favour of insiders or against innovators - concern more-or-less inevitable features of competitive systems of allocation. The usual focus on the problems with peer review and biases in grant schemes draws attention away from both the realities of how research priorities are set and the possibilities for greater community participation in setting research agendas.
Broad research priorities are set primarily by administrative decision. Concentrating on problems with peer review systems diverts attention from this reality. Indeed, peer review does not even provide the means for making many of the central decisions affecting research, such as decisions to set up institutions or departments or to provide greater infrastructure or funding for certain faculties. This is because peer review mainly concerns judgements within disciplines (peer groups). The key research decisions, by contrast, concern judgements between disciplines.
Concentrating on problems with grants schemes as they currently exist also diverts attention from the possibility of greater participation by community groups and individuals in setting priorities for research. There are many ways in which such participation could be increased, such as by community representatives on institutional boards or advisory panels for research groups, as well as being given a say over potential research projects (Bammer et al. 1986). But most such proposals are well and truly off the agenda. The main contenders for influence are interest groups (especially government and large corporations) and researchers (especially elite researchers).
Essentially, the perennial complaints about grants schemes reflect the dependent but privileged position of most researchers. They benefit from the allocation of substantial social resources to their salaries and research projects. They are reluctant to question the dominant institutions that control their funding, or the framework in which it occurs. Few of them believe that groups in the community - at least those with little money - should have any direct say in research priorities. That would be threatening to their prerogatives and status, built up through peer systems.
Recently, the Australian government has made major policy changes affecting funding for higher education research (DETYA 1999). Although the changes are significant so far as Australian research policy goes, they are relatively minor within the wider context outlined in this article.
Administrative decision remains the dominant method for setting research priorities and allocating funding for much Australian research, including most corporate research both in-house and sponsored as well as much in-house government research. Within the university sector, administrative decision is commonly used to deciding on major infrastructure projects and the relative staffing and funding of different departments. Often these decisions are mediated through formulas, themselves agreed to through administrative processes. None of this is greatly affected by the government's policy changes.
Academic staff are expected to do research and typically are allocated a nominal proportion of their work week to do so. This proportion can vary enormously, from full-time for research-only positions to virtually zero for staff overloaded with teaching or administration. Despite the large inequalities in available research time, nevertheless it is more equally distributed than research grants. It is certainly the element of research support that most closely fits the principle of equal allocation. In addition, full-time research students, who carry out a substantial proportion of research done in universities, typically spend most of their time engaged in research, even more closely following the principle of equal allocation. The government's policy changes do not explicitly address time available for research, though there may be changes in time allocation as an indirect effect.
Community-based allocation has never been a basis for research funding in Australia. Australian science shop initiatives have received little institutional support (Bammer et al. 1992). There would be a few cases where community groups, without research funds to disperse, approach academics or students in order to get relevant projects carried out. The government's research policy is entirely geared towards research oriented to groups that can pay, especially large corporations and government itself. The white paper makes no mention of community input into research priorities.
Peer review is the method used by the Australian Research Council (ARC) and the National Health and Medical Research Council (NHMRC) to decide allocation of research grants. However, peer review operates only within an overall framework - such as the relative allocations to engineering and humanities - that is decided administratively. The white paper signals a change to peer review procedures and their administration. Two ARC innovations will be the appointment of programme managers to oversee the peer review system and the use of external reviewers to rank grant applications, superseding the former process of relying entirely on different independent external assessors for each application. These changes may well have significant effects at the level of individuals and groups applying for grants, but at a more general level they can be seen to be a minor modification within the peer review method of allocation.
Performance-based funding is used in Australia for determining 'institutional grants', the term for a component of research funding for universities that is allocated on a competitive basis. (This role for performance-based funding is unusual in an international context.) The white paper reports a change in the method of calculating this funding: it is now to be 60% based on success in obtaining research income, 30% on research student numbers and 10% on research output measured through publications. This is a change in percentages from previously, especially in putting more emphasis on research student numbers. However, performance-based funding is not used at the level of grant applications, so the effect of the policy changes on individual researchers, which will be mediated through administrative decision-making systems within universities, probably will be limited.
Thus, the government's initiatives on research funding, while significant in relation to previous patterns, are relatively minor within the wider context of possible methods of allocating funds. Administrative decision remains a dominant force, augmented by peer review and institution-level performance-based allocation. Equality as a method of allocation appears to have a gradually declining role, while community-based allocation remains totally marginal.
As for the problems associated with grant schemes, little is likely to change. Problems of bias, waste and discouragement will remain. More significantly, pressures to orient research to dominant interests, especially corporations and government, are increasing. Grant schemes requiring alliances between university researchers and 'outside partners' - typically corporations or governments - have been increasingly important in recent years. This increases the role of administrative decision, exercised by the outside partner, and puts pressure on academics to link their research to groups that have money and resources. Outside groups without money are left out of the picture.
What does this picture imply in terms of recommendations for change? The answer depends crucially on one's goals. If the goal is to serve the interests of large corporations and governments - which are commonly legitimated by equating them with the 'national interest' - then further expansion of grants requiring alliances with 'industry' would be called for. If the goal is to promote innovation (both technological and social) then, arguably, tied money and peer review serve as deterrents and a much better strategy is to provide ample guaranteed funding for a substantial period (Horrobin 1996), along the lines of the equality model. If the goal is to help solve problems raised by those without money and power - such as poor people, communities under environmental assault and people with disabilities - then a move toward community-based allocation is the way to go.
How to move in any given direction is a big topic, beyond the scope of this paper. The key point here is that the design of grant systems involves a set of social choices that have wide-reaching effects, yet most discussion about grants takes place within a narrow set of assumptions, without mention of dramatically different allocation principles and associated consequences.
I thank two anonymous referees and Aapo Skorulis for valuable comments.
Arditti, Rita, Brennan, Pat and Cavrak, Steve, eds. 1980, Science and liberation, Boston, South End Press.
Armstrong, J Scott 1996, 'The ombudsman: management folklore and management science - on portfolio planning, escalation bias, and such', Interfaces, 26(4), pp 25-55.
Armstrong, J Scott 1997, 'Peer review for journals: evidence on quality control, fairness, and innovation,' Science and Engineering Ethics, 3(1), pp 63-84.
Bammer, Gabriele, Emery, Merrelyn, Gowing, Linda and Rainforth, Jennifer, 1992, 'Right idea, wrong time: the Wisenet Science Shop 1988-1990,' Prometheus, 10 (December), pp 300-310.
Bammer, Gabriele, Green, Ken and Martin, Brian 1986, 'Who gets kicks out of science policy?', Search, 17 (January-February), pp 41-46.
Campanario, Juan Miguel 1995, 'On influential books and journal articles initially rejected because of negative referees' evaluations', Science Communication, 16(3), pp 304-325.
Chubin, Daryl E and Hackett, Edward J 1990, Peerless science: peer review and U.S. science policy, Albany, State University of New York Press.
Daniel, H-D 1993, Guardians of science: fairness and reliability of peer review, Weinheim, Germany, VCH.
DETYA (Department of Education, Training and Youth Affairs) 1999, Knowledge and innovation: a policy statement on research and research training, Canberra, Commonwealth of Australia (http://www.detya.gov.au/highered/whitepaper/).
Dickson, David 1984, The new politics of science, New York, Pantheon.
Epstein, William 1990, 'Confirmational response bias among social work journals', Science, Technology, & Human Values, 15(1), pp 9-38.
Farkas, Nicole 1999, 'Dutch science shops: matching community needs with university R&D', Science Studies, 12(2), pp 33-47.
Forsdyke, D R 1993, 'Bicameral grant review: how a systems analyst with AIDS would reform research funding', Accountability in Research, 2, pp 237-241.
Horrobin, D F 1974, 'Referees and research administrators: barriers to scientific advance?', British Medical Journal, (27 April), pp 216-218.
Horrobin, David F 1990, 'The philosophical basis of peer review and the suppression of innovation', Journal of the American Medical Association, 263(10), pp 1438-1441.
Horrobin, David F 1996, 'Peer review of grant applications: a harbinger for mediocrity in clinical research?', Lancet, 347 (9 November), pp 1293-1295.
Kohn, Alfie 1993, Punished by rewards: the trouble with gold stars, incentive plans, A's, praise, and other bribes, Boston, Houghton Mifflin.
Martin, Brian 1979, The bias of science, Canberra, Society for Social Responsibility in Science.
McCutchen, Charles W 1977, 'Bring back patrons', The Sciences, (November), pp 19-21.
Peters, D P & Ceci, S J 1982, 'Peer-review practices of psychological journals: the fate of published articles, submitted again', Behavioral and Brain Sciences, 5, pp 187-195.
Roy, Rustum 1984, 'Alternatives to review by peers: a contribution to the theory of scientific choice', Minerva, 22 (Autumn-Winter), pp 316-328.
Wenneras, Christine and Wold, Agnes 1997, 'Nepotism and sexism in peer-review', Nature, 387 (22 May), pp 341-343.
Wesseley, Simon 1997, 'Peer review of grant applications: what do we know?', Lancet, 352 (25 July), pp 301-305.
Zaal, Rolf and Leydesdorff, Loet, 1987, 'Amsterdam Science Shop and its influence on university research: the effects of ten years of dealing with non-academic questions,' Science and Public Policy, 14 (December), pp 310-316.