I found a solution. This error was fixed in release 2021.1.3, but the PsychoPy download page links to 2021.1.2. You have to go to the github releases page to grab the latest version.
After this, within your project folder, delete index.html and the lib folder to make sure those files are recreated correctly.