Sorry, I figured it out, here is the solution I found. When I added a Collider to the Button, the Collider was super tiny. I didn't realize that until I tried to edit it. I made the Collider larger and tried to "fit it around the button" so to speak, and that fixed the issue. The Collider was just so small I was never hitting it.