Pressing the "Escape" button should close the settings window. It surprised me that this is not how it worked, because I think it's a pretty intuitive UX response. As far as I can tell, the only way to close the settings window is by manually clicking the small "close" button on the top left.