2018 Quality Evaluation FAQs - Evidence Portfolios

2018 Quality Evaluation FAQs - Evidence Portfolios

Last updated 25 November 2016
Last updated 11/25/2016

These FAQs relate to completing and submitting Evidence Portfolios (EP) for the PBRF 2018 Quality Evaluation.   

Evidence Portfolios

Q. Why is my area of research not specifically mentioned in the Description of Panel Coverage section of the panel-specific guidelines? 

A. The descriptions of possible subject areas covered by a panel should be considered a guide – they are not intended to be exhaustive. As a result, not all disciplines or sub-disciplines are listed. Panels will recognise research across a wide spectrum of subject areas and specialisations. Staff members should submit their Evidence Portfolio (EP) to the panel that most closely aligns with the majority of their research outputs. More information is available in the Completing the Panel Details section of the Guidelines for tertiary education organisations participating in the 2018 Quality Evaluation. 

Guidelines for tertiary education organisations participating in the 2018 Quality Evaluation (PDF, 2.2 Mb)

Q. How do I choose which panel to submit my EP to?

A. Select the panel that is the best match to the research outputs (Nominated Research Outputs (NROs) and Other Research Outputs (OROs)) described in your EP. More information is available in the Completing the Panel Details section of the Guidelines for tertiary education organisations participating in the 2018 Quality Evaluation. The research output component is worth 70 percent of the total score, so it is important that these outputs are assessed by the most appropriate panel. Reading the panel-specific guidelines for the panels you are considering will also help you determine which panel to submit to.

Q. My research is in a subject-specific area of educational research (for example, mathematics education). Should I submit my EP to the Education Panel or to the relevant subject panel?

A. The panel-specific guidelines for the Education Panel provide guidance on the circumstances in which you should submit an EP to the Education Panel, and the circumstances in which it would be more appropriate to submit your EP to the subject-specific panel.

Some other panels (Biological Sciences, Creative and Performing Arts, Health, Humanities and Law, Mathematical and Information Sciences and Technology, and Medicine and Public Health) also include specific guidance relating to educational/pedagogical research outputs in their guidelines, including likely scenarios for cross-referrals to the Education Panel.

Q. In the panel-specific guidelines, another panel provides information about a topic that my panel doesn’t cover. Should I refer to the other panel’s guidelines for that topic?

A. No – refer to the information on that topic in the main set of the Guidelines for the PBRF 2018 Quality Evaluation (Guidelines for tertiary education organisations participating in the 2018 Quality Evaluation, Guidelines for the 2018 Quality Evaluation assessment process, A guide for staff members participating in the 2018 Quality Evaluation). 

For topics where the panel-specific guidelines for a panel don’t provide guidance or information, the advice provided in the set of main guidelines is considered to be sufficient for that panel. As there are significant differences in the research approaches and practices of the sectors covered by the 13 panels, the research activities of some sectors are more closely aligned than others with the general advice found in the main set of guidelines. This creates variation in the amount and type of advice required in the panel-specific guidelines.

Q. The panel-specific guidelines for my subject area don’t provide any advice on a particular topic. Where else can I go for advice?

A. The panel-specific guidelines provide advice on specific areas related to the subject areas of each of the 13 peer review panels, in addition to the comprehensive information in the main set of guidelines. For topics where the panel-specific guidelines don’t provide guidance or information, the advice provided in the main set of guidelines is considered to be sufficient. Your TEO will have a key contact person or people for PBRF queries, and if your TEO has a research office this may be the first place to contact. Your TEO may have also made information available to you online, for example, an organisational intranet. 

Q. If I have any questions about submitting an EP, can I ask a colleague who submitted an EP in previous Quality Evaluations?     

A. There are some changes to the types and amounts of information and evidence required in the EP compared to the 2012 Quality Evaluation. Staff members should consult your TEO research office or key PBRF contact person at your TEO, or refer to the Guidelines for the PBRF 2018 Quality Evaluation (Panel-specific guidelines and A guide for staff members participating in the 2018 Quality Evaluation), to ensure you have the most up-to-date information.

  

Research outputs

Q. Will every Evidence Portfolio have at least one Nominated Research Output (NRO) main object examined? 

A. Yes.

Q. The guidelines state “If an NRO is not of sufficient quality to complete the examination, or does not meet the evidence requirements for the research output type, the panel member is not required to assess that NRO… Panel members cannot request a new version of the NRO or additional material for the NRO”

If a link to an NRO or an NRO pdf becomes corrupted for some reason, will the TEO be alerted to the problem and given the opportunity to correct? 

A. Panel members are able to identify issues relating to broken links and corrupt documents. We will manage this process with the TEO if this occurs to ensure that a corrupted PDF and a new PDF contain the same information. The PBRF IT System will be released a year prior to final submission, so please ensure that you upload documentation early and test all links and PDFs thoroughly.

Q. According to the guidelines the scoring weighting for the Research Output component is 70 percent. As most of the panel review is focused on the four NROs, is there a breakdown clarifying the percentage (weight) of the NROs and that of the 12 Other Research Outputs (OROs)?

A. No, there isn’t a specific breakdown. The NROs are the primary outputs that form the assessment while the OROs speak to the overall platform of research. The Research Output tie-point descriptors in the assessment guidelines provide guidance about what is expected. For example, a score is more likely to be awarded where all outputs could be seen to meet the standard for that score. 

Q. My research output is a commercially sensitive report and there are areas of it which are confidential. If the commissioning agency is reluctant to give permission for it to be released for assessment in its entirety, can the sensitive areas be redacted? 

A. We have specific processes in place to enable confidential outputs to be assessed securely. If the commissioning agency is still reluctant to give permission, then we would raise this with the panel Chair to determine whether a redacted version could be reasonably evaluated. 

Q. I have a research output type that isn’t mentioned in the panel-specific guidelines. Can I still submit it in my EP?

A. Yes, if it meets the eligibility criteria for research outputs in the Guidelines for tertiary education organisations participating in the 2018 Quality Evaluation.

Guidelines for tertiary education organisations participating in the 2018 Quality Evaluation (PDF, 2.2 Mb)

The lists of examples of research outputs in the panel-specific guidelines are not intended to be exhaustive. Panels will recognise research across a wide spectrum of research types.

Q. Are there limits on the file size or file types that can be uploaded as NRO evidence in the EP?

A. To assist TEOs with providing high-quality evidence of research outputs, the PBRF technical specifications set out the requirements for electronic media formats. These are available in the PBRF 2018 Quality Evaluation EP Schema Definition.

PBRF 2018 Quality Evaluation EP Schema Definition (PDF, 1 Mb)

If you have any concerns about the file size or file type of your evidence, contact your TEO’s key PBRF contact person. This may be your TEO’s research office.

 

Research contribution

Q. What level of information is required to assess the Research Contribution?

A. Each Research Contribution item needs to include a comprehensive description of the nature and significance of the item. Where the contribution has had an impact outside of academia, it will be particularly important for the panel to understand the reach of this impact. This description must include sufficient information and evidence of the quality and prestige of the research and research-related activity to enable the panel to distinguish between the tie-points.

Information on the tie-points can be found in the Assessing the Research Contribution component section in the Guidelines for the 2018 Quality Evaluation assessment process.

Guidelines for the 2018 Quality Evaluation assessment process (PDF, 1 Mb) 

Q. How much detail does the panel need about student supervision as a research contribution? Should we list just the degree, details on the stage of the supervision (completed vs. in progress), and the role in the supervision (chief supervisor or other supervisor), or do we need student name and thesis title for each?

A. Names of students and thesis titles are not required in the Evidence Portfolio (EP). The panels do not authenticate entries and where they have concerns about the validity of a research contribution entry they will log an Audit Concern which will be taken to the TEO by the auditors. 

Q. Can participation or involvement in Track II dialogues (such as those led by the Asia New Zealand Foundation, with think-tanks in the Asian region) be included in the Research Contribution component of my EP? If so, what would I include it under?

A. Yes, it can. The Research Contribution component covers activities that are wider than standard academia. This type of activity could be included under Facilitation, Networking and Collaboration.