A speaker vibrates air in front if the speaker cone and behind it. You need to stop the air that is moving at the front mixing with the air that is moving at the back. If these two vibrations mix, you don’t get much sound out of the speaker. Mounting the speaker on a very large (infinite) panel does that as the air vibrating at the back can’t travel round the panel to the front. Unfortunately fitting an infinite sized panel in a model boat ( or anywhere else) isn’t practical, hence the need to mount the speaker in a box to stop the vibrations from the back mixing with those from the front.
The hull can act as the box. You just need to work out where you can mount a panel with the speaker on it so that the back of the speaker is sealed, and the front is open to let the sound out.
As for size, the general advice would be ‘as big as possible’😀 I’ suggest something like 70mm diameter. If you can fit that in. As you are wanting to generate steam sounds, which tend to be higher frequency, you might get away with something a bit smaller.
The link should show a photo of the speaker I fitted in my rowing boat. ( Scroll down to ‘radio upgrade’) It is positioned under the foredeck which is quite a small space but it works well. You can hear Bionic Bill singing right across the lake. 🤣