Pre-registration for behavioral studies

Summary

 

The purpose of this document is to help with the implementation of our lab’s vision to pre-register behavioral studies. Included among researcher’s complaints about pre-registration are that it’s difficult, tedious, and the process is relatively unknown. We hope to improve on some of these pain points. Therefore, we are clarifying some of the mystery behind pre-registration by providing practical steps, as well as an easy to use template to pre-register a behavioral study.

 

Dr. David Mellor, Project Manager at the Center for Open Science, informed us that he has not heard of any preference from any academic journal editor or reviewer for one pre-registration template over another, such that currently no one pre-registration template provides a benefit for future journal acceptance. By consensus, Aaron and Chujun prefer this pre-registration template due to its ease of use, thoroughness, and the additional flexibility it allows. Although using this template does not guarantee that a journal, reviewers, or a badge certifying organization will accept the pre-registration, it is ultimately set-up to satisfy this goal.

Relevant Sources

  • Journals recommending the Open Science Framework as a repository for pre-registration, material, and data include Nature, Science, and Psychological Science. Journals promoting the scientific values of open-sharing, transparency, and reproducible research can be found here, Click “Current Signatories”
  • Detailed pre-registration instructions can be found here
  • Pre-registration badge specifications can be found here
  • v’ant Veer and Giner-Sorelli (2016) can be found here

 

How to pre-register your study

Step 1:     Fill out the pre-registration form – (see page 3)

If there are essential elements of this form you do not wish to include, type N/A or a small explanation

Step 2:     Create a project on the Open Science Framework – Easy instructions

Step 3:     Register your research project – Easy instructions

For the prompt “choose a registration form” select Pre-Registration in Social Psychology (van ‘t Veer & Giner-Sorolla, 2016)

Step 4:     Run your study

Step 5:     Include a link to the time stamped pre-registration as part of your paper

Explain deviations from the pre-registration methods and analyses. Make a distinction between confirmatory (a-priori) analyses and exploratory (post-hoc) analyses not described in your pre-registration

 

Pre-registration template (van ‘t Veer & Giner-Sorolla, 2016) – Version: 2.1 (May 10, 2016)

Check for updates to the template here

 Project working title:

Authors: (leave this out for blind review of the pre-registration)

Affiliation: (leave this out for blind review of the pre-registration)

A. Hypotheses

Description of essential elements

  1. Describe the (numbered) hypotheses in terms of directional relationships between your (manipulated or measured) variables.
  2. For interaction effects, describe the expected shape of the interactions.
  3. If you are manipulating a variable, make predictions for successful check variables or explain why no manipulation check is included.

Recommended elements

  1. A figure or table may be helpful to describe complex interactions; this facilitates correct specification of the ordering of all group means.
  2. For original research, add rationales or theoretical frameworks for why a certain hypothesis is tested.
  3. If multiple predictions can be made for the same IV-DV combination, describe what outcome would be predicted by which theory.

 

B. Methods

Description of essential elements

Design

List, based on your hypotheses from section A:

  1. Independent variables with all their levels: (a.) whether they are within- or between-participant (b.)the relationship between them (e.g., orthogonal, nested).
  2. Dependent variables, or variables in a correlational design
  3. Third variables acting as covariates or moderators.

Planned sample

  1. If applicable, describe pre-selection rules.
  2. Indicate where, from whom and how the data will be collected.
  3. Justify planned sample size (if applicable, you can upload a file related to your power analysis here (e.g., a protocol of power analyses from G*Power, a script, a screenshot, etc.).
  4. Describe data collection termination rule.

Exclusion criteria

  1. Describe anticipated specific data exclusion criteria. For example: (a.) missing, erroneous, or overly consistent responses ; (b.) failing check-tests or suspicion probes ; (c.) demographic exclusions; (d.) data-based outlier criteria; (e.) method-based outlier criteria (e.g. too short or long response times).

Procedure

  1. (Recommended element, in the online form see next page) Set fail-safe levels of exclusion at which the whole study needs to be stopped, altered, and restarted. If applicable, you can upload any files related to your methods and procedure here (e.g., a paper describing a scale you are using, experimenter instructions, etc.).
  2. Describe all manipulations, measures, materials and procedures including the order of presentation and the method of randomization and blinding (e.g., single or double blind), as in a published Methods section

 

C. Analysis plan

Confirmatory analyses

Describe the analyses that will test each main prediction from the hypotheses section. For each one, include:

  1. the relevant variables and how they are calculated;
  2. the statistical technique;
  3. each variable’s role in the technique (e.g., IV, DV, moderator, mediator, covariate);
  4. rationale for each covariate used, if any;
  5. if using techniques other than null hypothesis testing (for example, Bayesian statistics), describe your criteria and inputs toward making an evidential conclusion, including prior values or distributions.

(the online form asks the above for the first, second, third, fourth and further predictions separately)

Recommended elements

Specify contingencies and assumptions, such as:

  1. Method of correction for multiple tests.
  2. The method of missing data handling (e.g., pairwise or listwise deletion, imputation, interpolation).
  3. Reliability criteria for item inclusion in scale.
  4. Anticipated data transformations.
  5. Assumptions of analyses, and plans for alternative/corrected analyses if each assumption is violated.

(Optionally, upload files here related to your analyses (e.g., syntaxes, scripts, etc.).

Answer the following final questions:

Has data collection begun for this project?

  • No, data collection has not begun
  • Yes, data collection is underway or complete

If data collection has begun, have you looked at the data?

  • Yes
  • No

Any additional comments before I pre-register this project (optional)

 

Example Pre-registration

Link

To view the pre-registration form, click the button “View Registration Form.”

To view the pre-registration’s attached files, click “+” by the folder Archive of OSF Storage.