your description of the material you are seeking could be ripstop nylon. I have used this for many years both in kite making and also for sails on my boats. The lightest weight generally available is 2 oz / 60 gsm which I have found just fine for my boats. It is available on-line (ebay) and is only around £4 a metre. The next weight up (heavier) is 3.8 oz, ie nearly twice as heavy. I have also used this for boat sails but it is stiffer. Sometimes this is preferable.
With ripstop nylon the cut edges do not tend to fray, so I have turned the material just once for the hems. This makes for a flatter edge than turning it twice which is necessary with other types of material. Yes, I make all my own sails and machine sew hems. On some sails I have used bias binding along the edges rather than hem the material. It makes for a firmer edge when using the lightest sail material, which helps when it comes to sewing cringles.
Chugalone has given some excellent information for sail making which is appropriate to cotton sails that replicate the originals as close as possible. However, should you be wanting to create good looking working sails without needing to replicate the look of the originals then perhaps ripstop nylon is the stuff for you.
Happy sewing ! 😉