The assessment of research grant proposals is made by appropriate experts drawn from academia, government and industry. Proposals are usually assessed first by individual reviewers and then by peer review panels.
Normally at least two individual reviewers are used, one of whom may be nominated by the applicant, although STFC reserves the right not to use nominated reviewers. Nominated reviewers must not be collaborators; nor should they be from the applicant's or collaborator's organisation. Applicants are given an opportunity to respond to reviewer comments prior to assessment by the peer review panel.
The membership of peer review panels is chosen to span the scientific remit of the panel and is decided in consultation with the community, usually following a call for nominations. STFC attaches great importance to ensuring that conflicts of interest among panel members and reviewers are disclosed and managed. Guidance for managing conflicts of interest is available below.
1. These guidelines set out the procedures for managing conflicts of interest that the Council expects peer review panels to follow in the assessment of proposals and the allocation of funds or other resources. The guidelines have been developed in order to protect the integrity of Council and reduce the risk of impropriety or any perception of impropriety in the conduct of peer review business and allocation of funds. The guidelines apply to all individuals involved in any way in the peer review of applications for funding.
2. Individual panel members like others who serve the public are expected to follow the seven principles of public life set out by the Committee on standards in Public Life (the Nolan Committee). The seven principles are listed at Annex A of the “Code of Practice for Council Members”.
3. In the absence of specific statutory provisions, common law requires that members of public bodies should not participate in the discussion or determination of matters in which they or a member of their family or household have a direct pecuniary or other material interest and where participation in the discussion would suggest a real danger of bias or give rise to a perception of bias.
4. In the context of peer review of project proposals or grant applications, a conflict of interest is normally deemed to arise:
5. Pre-panel meeting briefings will be used by the Council staff to bring to the Chair's attention all relevant information on any connection between an application and a panel member, including institutional conflicts of interest. These Council guidelines should be followed in order to identify where a conflict of interest will require an individual panel member to withdraw from the meeting.
6. At the start of a panel meeting, the Chair will ask panel members to make a declaration on conflicts of interest. This declaration will include the requirement for a positive declaration that there is no conflict of interest. The nature of any conflict of interest, including positive declarations, will be recorded in the minutes of the meeting. If an individual panel member finds that a conflict of interest becomes apparent only as the meeting progresses, the nature of the conflict should be declared as soon as practicable to the Chair. The panel can then recommend whether the member has to 1. Leave the meeting during discussions relating to the conflicted application; 2. Stay in the meeting but not participate in the discussion; or 3. Stay in the meeting and participate freely. The office will make the final decision in all circumstances.
7. A conflict of interest in any of the following categories will automatically require that the member withdraw from the meeting for the discussion of that application:
8. If after exploring other possibilities, it is considered that the exclusion of a member under 7(b) above would compromise the panel's ability to make a proper scientific judgement of the proposal, the automatic exclusion may be overridden. However this option only applies where the panel member does not plan to work on the project under consideration. Where such an exception is made, it must be solely on the grounds that the value to the quality and integrity of the peer review process of the panel member in question may be deemed to outweigh any residual suggestion of bias arising from the declared conflict of interest.
9. Notwithstanding any of the above, a panel member may voluntarily choose to withdraw from the discussion of a proposal at any time if s/he wishes to do so. This may arise for a number of reasons as discussion of a proposal develops. The secretary to the panel should record in the minutes the point at which an individual ceases to take part in the discussion and leaves the meeting.
If you have been asked to provide a review for an STFC proposal, the following guidance will help you determine whether you have a conflict of interest which you would want to declare.
It is vital that reviewers are seen to be completely impartial. You should not, therefore, take part in the review of any proposal where you perceive there is a conflict of interest. We accept that it is likely that academics who work in the same field may know each other, and this doesn't therefore bar you from commenting on a proposal. Most of our peer review forms contain a confidential section on 'Conflicts of Interest”, where this information can be declared. However, if your knowledge/relationship with the applicant is such that you feel it would be difficult to be impartial when commenting on the proposal, then please decline the review selecting “Conflict of Interest” as the reason why. If you are unsure whether to decline on the grounds of conflict then please contact the Grants Policy team for advice before doing so.
You are at the same/proposed HEI/RO as the PI or any Co-I.
You are at an Organisation that is a project partner or that of a visiting researcher on the proposal.
You are named as a participant on the proposal eg as an Investigator, staff member or project partner.
You have a close family relationship (spouse, partner, parent, sibling, child) or share a household with the PI/Co-I.
You stand to gain a financial or professional advantage from a particular outcome for an application which you are asked to review.
You have written a letter of support for the proposal.
You have had a PhD/PhD Supervisor relationship with any of the applicants in the last ten years (FELLOWSHIPS ONLY).
You are at the applicant’s proposed organisation i.e. the organisation of the proposal (FELLOWSHIPS ONLY).
You are at an Organisation where the candidate has a current application for a non-STFC funded position (FELLOWSHIPS ONLY).
STFC recognises the importance of objectivity in making funding decisions and in ensuring that this is done fairly, transparently and without bias. To support the safeguarding of objective decision making, the following documents cover the expectations that we have of our peer review panel members and Chairs and the role we expect them to fulfil. They also cover the role of STFC staff and what our panels can expect from us along with the principles we set for peer-review meetings.
Cross-Council guidelines have been developed in order to protect the integrity of Council and reduce the risk of impropriety or any perception of impropriety in the conduct of peer review business and allocation of funds. The guidelines apply to all individuals involved in any way in the peer review of applications for funding.
Individual panel members like others who serve the public are expected to follow the seven principles of public life set out by the Committee on standards in Public Life (the Nolan Committee).
Reviewers are required to make a statement in every completed report to declare whether they have/have not any conflicts of interest. Failure to make this declaration will invalidate the report.
Names of reviewers are not disclosed to applicants and neither are those of the lead introducer on the grant. There are exceptions to the latter for those grant rounds which operate clarification meetings which are led by the lead introducer or where the lead introducer also acts as a mentor to the applicant.
Applicants for research grants should note that canvassing of members of the Peer Review Panel will lead to disqualification.
This page provides general advice and guidance for reviewers on completing reports for the STFC peer review process. You should also refer to the JeS Helptext when preparing to complete a review.
Reviewers' input is the single most important element in the peer review process, providing advice on the qualities of the many research proposals we receive each year. For the process to work effectively, reviewer comments should be timely, objective, fair and informed.
Reviews are based around a series of assessment criteria:
In addition some funding schemes have further scheme-specific assessment/selection criteria which can be found in the relevant guidance notes:
To maximise their value to the peer review process reviewer reports should aim to:
It is important to bear in mind how your report will be used. Your report will be fed back anonymously to the applicant, who will then be allowed to respond to factual inaccuracies or questions you raise. Following this, members of peer review panels will be asked to use your reports as a tool for ranking between proposals.
When you are asked to review a proposal, all the necessary documents will be sent to your Joint electronic Submission (Je-S) account. If you are asked to review a proposal submitted against a particular call for proposals, we will also include a web link to the call document which will set out all the call requirements providing you with background information as to what the applicant was advised. You should also check the “Instructions to Reviewer” notes specific to the request which are available within the JeS form. These will explain, for example, where you are only required to comment on certain aspects/projects on the proposal.
The proposal you are asked to review includes a case for support. In some instances, the case for support may include a link to a web site containing information on the research proposed. Reviewers are not required to consider this additional information when providing comments on a proposal. If you do choose to look at this information, it is possible that your anonymity to the applicant will be compromised.
All reviews on our behalf are submitted using the electronic forms available through the UK Shared Business Services Je-S system.
If you cannot comment within the indicated timescale, please confirm this immediately so we have time to approach an alternative reviewer or perhaps extend the deadline. In addition, please confirm immediately if you do not feel qualified to comment at all.
STFC has adopted the following standard assessment criteria for research grant proposals. Additional criteria may apply to specific funding calls and will be described in the specific call guidance.
Supporting evidence which increases the confidence in a successful outcome. Where any of these are not met the risk and any proposed remedial or mitigation action must be identified. Where any criteria are not met any recommendation for funding would be subjected to close scrutiny by STFC. If approved for funding, STFC is likely to make an award contingent on remedial action to address the concerns highlighted before funds are committed.
Important additional criteria, the opportunities and plans for which must be addressed in the application.
Applicants should always refer to the specific call guidance for further details.
We are committed to support the recommendations and principles set out by the San Francisco Declaration on Research Assessment (DORA; https://sfdora.org/read/). You should not use journal-based metrics, such as journal impact factors, as a surrogate measure of the quality of individual research articles, to assess an investigator’s contributions, or to make funding decisions.
For the purpose of research assessment, please consider the value and impact of all research outputs (including datasets, software, inventions, patents, preprints, other commercial activities, etc.) in addition to research publications. You should consider a broad range of impact measures including qualitative indicators of research impact, such as influence on policy and practice.
The content of a paper is more important than publication metrics, or the identity of the journal, in which it was published, especially for early-stage investigators. Therefore, you should not use journal impact factor (or any hierarchy of journals), conference rankings and metrics such as the H-index or i10-index when assessing UKRI grants.
We encourage you to challenge research assessment practices that rely inappropriately on journal impact factors or conference rankings and promote and teach best practice that focuses on the value and influence of specific research outputs. If you are unsure about DORA, please speak to the panel convener or the panel chair.
A particular equality issue in peer review is unconscious bias. Despite striving to be objective, people often hold implicit or unconscious assumptions that influence their judgement. Examples range from expectations or assumptions about physical or social characteristics associated with gender, ethnicity and age to those associated with certain jobs, academic institutions and fields of study. A briefing note on unconscious bias is available below.
Despite striving to be objective, people often hold implicit or unconscious assumptions that influence their judgement. Examples range from expectations or assumptions about physical or social characteristics associated with gender, ethnicity and age to those associated with certain jobs, academic institutions and fields of study.
One example reported by Goldin and Rouse1 concerned the hiring of musicians by major symphony orchestras. When the identity of auditioners was withheld from the judging panel and they were placed behind a screen, the orchestras increasingly hired women.
Examples from STEM-related fields have also been reported. Steinpreis et al.2 reported a study where 238 academic psychologists evaluated an early-career CV which had been randomly assigned a male or female name. Both male and female assessors gave the male candidates better evaluations for teaching, research and service and were more likely to employ the male than the female candidate.
Wenneras and Wold3 reported a study of evaluators’ rating sheets for postdoctoral fellowships awarded by the Medical Research Council in Sweden. They found that women candidates need substantially more publications to achieve the same rating as men: the equivalent of three more papers in Nature or Science, or 20 more papers in speciality journals.
Trix and Psenka4 studied over 300 letters of recommendation for medical faculty who were hired by a large American medical school in the 1990s. They found that letters written for women were shorter, raised more doubts and portrayed women more as students and teachers while portraying men more as researchers and professionals.
Similar effects have been reported for race and ethnicity.
The difficulty with unconscious bias is of course that individuals are unaware that bias is influencing their decisions. However, there are some general points of good practice that panel members should adopt in order to minimise the effects of unconscious bias:
As part of STFC’s commitment to eliminating unlawful discrimination and promoting equality of opportunity, we collect, analyse and review equality data relating to the people we award funding to and appoint to our boards, committees and panels. This helps us to monitor how successful we are in eliminating discrimination and promoting equality and to identify any issues that need addressing. We use this information for monitoring purposes only, treat it in strictest confidence and process it separately from any activities concerned with the selection or operation of boards, committees and panels. Reporting is in anonymised, aggregate form only.
STFC seeks the best members possible for its advisory bodies but also seeks to ensure that membership reflects the diversity within the relevant research communities and, therefore, particularly welcomes nominees from groups historically under-represented on these bodies. We also welcome nominations of appropriate experts based outside the UK. Where appropriate we take positive action to encourage nominations of people from under-represented groups and to broaden the pool of nominees more generally.
We also encourage members to tell us about any adjustments that would help them to participate in meetings and other activities.
Applications, independent reviews and PI Responses are available to panel members via the Peer Review Extranet (STFC’s preferred method for sharing data). Strict controls on data security and data handling are currently in place for Government departments and Government-funded organisations, including the Research Councils. Panel members must not save data (on to laptops, discs or hard drives) and if printed copies of any of the documents are made, these must be shredded after use.
Research Councils have obligations under both the Freedom of Information Act and Data protections acts. In line with this, a cross Council Peer Review Framework has been agreed to provide guidance on information relating to the peer review process including what is routinely disclosed/published and what is not. The document is designed for use by applicants, research organisations, board/panel members and members of the public. The full framework can be found below.
Last updated: 10 February 2020