Huge shoutout to @damienbarrett on the Mac Admins Slack Team for the fix. I'm just documenting it here, in case anyone else runs into this problem and is searching for the solution.
You launch up Wabbit Emu and try to run your TI-## calculator ROM, and the onscreen buttons don't match up with what's actually happening on the emulated screen. For example, you might click the Clear button and see the number 1 appear instead.
First, quit out of Wabbit Emu completely (Cmd-Q).
Now when you open up WabbitEmu, the appropriate onscreen buttons should work.
If you want to script this change, check out pudquick's Python script to set LSHighResolutionModeIsMagnified. You may have to modify it, of course, since it uses TextEdit as an example app:
lowres_app_id = u'org.revsoft.WabbitEmu'
Just keep in mind that after the script runs, the user will have to log out and log back in again for the change to take effect.