Hi Phil, could be a dodgy ESC/loose motor wire connection, (bullet connectors need to be tight) Could be something with the RX throttle lead connection. Is the ESC auto battery setting ? if not, is it set to the correct number of cells and battery type? Try testing each thing (ESC, RX) one at a time with another one of poss. Are you using a BEC from the ESC or a separate UBEC? Usually trial by elimination will pick it up.
I have around 50 ESCs of many types, and have never had that problem. If you can, start with the motor on another ESC and work backwards. The noise is the prop, only way to reduce that is to use a lower KV motor (around 800kv and a bigger prop) Mine is really loud with a 36/48 1450kv motor with a 9"x 6" prop (its' max load on 3s) (see recent vid). It can run a 5s but would require a 6"prop and would be ear splitting (over 100dB/29000 rpm). I've got it at a balance between speed and lots of complaints LOL.
Check your amps full throttle static with that prop, and if below the max A spec for the motor, prop it to just under rated max for the motor static, IE more pitch (it will unload when under way) That's as quiet as you will get with a 3s. Try going to a 2s and a higher pitched prop which will reduce the noise, (2s will give the motor more torque for a bigger prop at a lower tip speed. Check amps as above. A 2s 40c will give you more power as well if required,- you don't have to buy a bigger motor to get more power, just up your C rating)