2018 Quality Evaluation FAQs - assessment process

2018 Quality Evaluation FAQs - assessment process

Last updated 25 November 2016
Last updated 11/25/2016

These FAQs relate to the assessment process in the PBRF 2018 Quality Evaluation.   

Assessment process

Q. Can we change the primary peer review panel we nominate between submission and reporting?

A. No. You must nominate one peer review panel that best matches the majority of your Nominated Research Outputs (NROs).

Where your research outputs involve interdisciplinary research that is covered by more than one panel, you should nominate the panel and the subject area that best matches the majority of the NROs. You should also note the interdisciplinary nature of the Evidence Portfolio (EP) in the Field of Research Description.

Panel Chairs might recommend we transfer an EP to another panel. If this happens, we will tell you when you receive the results of the Quality Evaluation.

You can no longer request a cross-referral to another panel (with the exception of cross-referrals to the Māori Knowledge and Development Panel and the Pacific Research Panel).

Q. I work in cross-disciplinary research. How do I request that some of the NROs in my EP be cross-referred to another panel?

A. Staff or TEOs cannot request a cross-referral (with the exception of cross-referrals to the Māori Knowledge and Development Panel and the Pacific Research Panel). Panel chairs will determine whether or not an EP needs to be cross-referred to another panel as part of the assessment. Staff who work in cross-disciplinary research should provide sufficient explanation in the “Field of Research” and “Platform of Research – Contextual Summary” sections of their EP to enable panel Chairs to determine whether the EP should be cross-referred to another panel.

Q. Why are some tie-points descriptors elaborated in the panel-specific guidelines, but not others?

A. In many cases the tie-point descriptors in the main set of guidelines contained sufficient information, and further elaboration was not required. Elaboration was only included where it provided useful additional information for that panel.

Q. The panel I’m submitting to intends to examine 50 percent of NROs, whereas some panels intend to examine 100 percent of NROs. Will this affect the quality of the assessment of my EP?

A. Each peer review panel is expected to examine at least 50 percent of the NROs listed in each EP. Panel members are expected to examine at least two of the NROs listed in an EP to enable them to form a reliable judgment about the overall quality of the Research Output component and to score it appropriately. The actual number reviewed may vary from panel member to panel member. 

Panels may examine more than 50 percent of NROs if they consider this to be appropriate and necessary to determine an appropriate score. In 2012, the percentage of NROs assessed was on average 80 percent. 

Q. I haven’t included my publication metrics in my EP – will the panel look them up?

A. No, panels will only assess the evidence provided in the EP. If you wish the panel to consider any publication metrics (such as the number of citations received or the number of papers published in the assessment period), include them in your EP. 

Any metrics included in an EP must be contextualised as part of the wider story about the quality of your research. The TEC does not produce or provide journal rankings for the Quality Evaluation assessment process.  You should refer to the appropriate panel-specific guidelines for more information about including publication metrics in your EP.

 

Cross-referral to the Māori Knowledge and Development or Pacific panels

Q. Will only the five reference items listed by the staff member in the Māori Research element or the Pacific Research element in an Evidence Portfolio (EP) be assessed if the EP is cross-referred?

A. No. The five referenced items are to help direct the Chair of the Māori Knowledge and Development Panel or the Chair of the Pacific Research Panel to relevant items, rather than be the only items that could be assessed. The relevant Chair(s) will use this information to decide whether an EP is cross-referred. This cross-referral assessment may relate to part of an EP or to specific items within the EP. The Chair(s) will advise the cross-referral panel member on what part or parts of the EP should be considered in the assessment. 

The Māori Research element and Pacific Research element in the EP was introduced to facilitate cross-referrals to the Māori Knowledge and Development Panel and the Pacific Research Panel and allow staff members to guide the Chairs of the panels in their consideration of cross-referrals. 

Q. In regards to cross-referencing to the Māori Knowledge and Development Panel, the Guidelines for TEOs (page 36) state that “The Māori Research element in the EP allows researchers to complete a comment (500 characters)… Commentary may include research based on Māori world views or Māori methods of research.”

Will there be a negative impact on the assessment of an EP if the staff member has not completed the Māori Research or Pacific Research element, such as the commentary or reference five items, but the primary panel Chair chooses to cross-refer to the Māori Knowledge and Development Panel or the Pacific Research Panel?

A. No. The cross-referral process between all panels is designed to operate without input from TEOs, with extra time added to support Chairs to make appropriate cross-referral decisions. Chair and panel training will also include the cross-referral process to ensure that it is being applied fairly and consistently.

Conflicts of interest

Q. Does a panellist need to remove themselves from their TEO’s PBRF preparation leading up to the 2018 round?

A. No. However, there are restrictions on what they can assess if they have been involved in internal assessment, advice on, and/or preparation of Evidence Portfolios (EPs) in their TEOs. Namely:   

  • If the panellist is involved in the internal assessment of their TEO’s EPs, or they have provided specific advice or guidance on individual EPs at their TEO while serving on a panel, they cannot assess EPs from their TEO at the individual assessment stage and can only contribute to panel discussions at the request of the Chair. 

  • If the panellist has no involvement in the internal assessment of their TEO’s EPs, they have not provided specific advice or guidance on individual EPs at their TEO while serving on a panel, and they have no other conflict of interest, they cannot be a lead assessor for EPs from their TEO but they may be assigned as a second assessor. 

For more information please see the Conflict of Interest Policy in the Guidelines for the 2018 Quality Evaluation assessment process. 

Guidelines for the 2018 Quality Evaluation assessment process (PDF, 1 Mb) 

Q. What should I do if I think that a panellist hasn’t identified a particular conflict of interest?

A. We have made significant changes to the conflict of interest policy and procedure to ensure that this is as transparent as possible. Our internal auditor will also review our processes and panel actions for managing conflicts of interest. 

If you have a family member or close friend or colleague who is a panellist, you should speak to them to remind them or confirm with them that they have declared a conflict of interest to the TEC rather than submit a conflict of interest notice.

In exceptional circumstances, TEOs may submit a notice of conflict of interest in relation to a panel member on behalf of staff members. For information about the criteria and process please refer to the Guidelines for tertiary education organisations participating in the 2018 Quality Evaluation.

Guidelines for tertiary education organisations participating in the 2018 Quality Evaluation (PDF, 2 Mb)

Q. If I have any questions about submitting an EP, can I ask my colleague who is a member of a PBRF peer review panel?

A. Members of peer review panels can provide general advice in some circumstances. In all cases, however, they will first take into account the PBRF Conflict of Interest policy. This may mean that the panel member declines to provide you with advice in some circumstances in order to avoid conflicts of interest at the panel assessment stage. In the first instance, it is best to consult your TEO research office or the key PBRF contact person at your TEO for any advice about submitting an EP.