In French, Subscribe is S'abonner
I have been unable to enter the single quote character.
Exactly, I can enter it, but it displays as the HTML code on the checkout page.
Thank you for bringing this to our notice. This is an issue on our system with the way special characters are rendered in checkout pages. Our team has been notified about this and they would be taking this up in the future when hosted pages revamp is done. We'll inform you when it's fixed.