Researchers cannot return or reject a submissions without negatively affecting a participants score. Either the participant has done a sufficiently diligent job as to get paid (even if you can't use the data) or not.
If you don't believe their Prolific Score should be affected then these participants should be approved and paid (even if you decided not to use their data).
If you think they have done a poor enough job to not deserve their reward then this should be reflected via a rejection.
The 3rd option is to request that the participant returns the submission themselves via a message or ask them to retake the study more carefully.