The poll also found Leave voters were less likely to think it was worth it if they or a family member lost their job.