On Prolific, you have the option to reject low-quality submissions. However, before you reject submissions, please think carefully whether the rejection is justified. Nobody likes to get rejected, so please try to keep rejections to a minimum. Participants are penalised for rejections and if they receive too many they will be removed from the pool. If you don't feed like a participants has been negligent then please avoid rejecting, participants are able to voluntarily return submissions and will often do so upon request if it's demonstrated that a mistake was made.
General rule of thumb when reviewing submissions: If a participant completed the survey in time and submitted a completion code, then they should be compensated.
Valid reasons for rejections are, for example (see also the decision tree diagram below):
- The participant completed your study exceptionally fast. By "exceptionally fast" we mean participants who are statistical outliers.
- The participant skipped crucial questions. Crucial questions are those that are inevitable to answering your research question. Demographic questions, for example, do not necessarily fall under "crucial questions" because participants have the right to withhold such sensitive information (according to guidelines by the British Psychological Society).
- The participant failed one or multiple attention checks (for more on attention checks or instructional manipulation checks please click here).
- The participant did not sufficiently engage in a task where the required level of engagement was clearly specified. e.g. the participant was asked to write a paragraph but only put a few words or gibberish.
Below are not valid reasons for rejection:
- A participant completed your study "too fast" when "too fast" isn't clearly specified in advance. e.g. I don't think participants can complete this in less than X mins. Participants completion time varies around an average. They have variable reading speeds and variable knowledge of keyboard shortcuts and other browser tools. Please use attention checks/IMCs to check for attention rather than completion time. Completion time cannot be used as a reason for rejection unless they are a statistical outlier below the mean.
- I can't use the data because:
- The participant completed the study but failed an internal measure so I can't use their data. e.g. the participant's variance on a measure was not above an arbitrary threshold. Unfortunately, in most cases, if you use internal measures as a quality control that are not explicitly explained to participants then you'll still need to pay for their response even if you can't use their data. Please get in touch if you need clarity on your study.
- There was a technical error in my study that caused the data not to be saved but wasn't the participant's fault. Unfortunately, technical errors sometimes occur but if it was not the participant's fault then this is not a valid reason for rejection. Please get in touch if this happens and we'll help as best we can.
Please be respectful to participants and try to resolve issues with participants as best you can - remember that these are people that have taken their time to participate in your research :)
Please note that Prolific may overturn rejections in certain occasions.
You may also be interested in: