Gentlemen, you illustrate exactly why I built my own first ESC (ignorance and mistrust of purchased products,) and partly why I then continued with my Tug dual ESC blog. Advert over...but in a few blogs time I will be covering H bridges, which are present in every ESC and may, I repeat may, be a real source of some of your problems.
With no disrespect intended, you don't really know what, if anything, is wrong.
I can say that battery specifications and tests can be horribly misleading. In particular, battery voltage cannot be interpreted with a DVM. In my testing I have seen battery voltage, on NiMh at 7.2V nominal, dip and recover by over 500mV in a single 3msecond cycle while drawing only 3A. That is more than enough to trigger under voltage trips rather earlier than expected, and I'm sure that other technologies or poorer state of charge would give even worse figures. Figures given by marketing men are totally worthless, and I understand that Chinese batteries can be only 10℅ of what is promised.
Going back to H bridges, several aspects of the design are critical. In particular, if the design allows even momentary "shoot through" - google that- you will get poor runtime at best and early under voltage tripping as a certainty. Higher voltages make that a higher possibility, again depending on any short cuts in the design. The ESC numbers you mention only seem to have options for up to 9 cells, which means absolute max running at start of around 12.6V and rapidly decaying on a few seconds to 12.2 and below.?? But your 12V batteries will start at 14.4 and should not be run below 8V, so a 5V drop out is extremely low, the battery is over discharged by then. If you could select the 9 cell option, and if I was designing the ESC I'd be putting dropout at around 7.5V and hope for recovery to 9V to allow a limp home mode.
What is there to do?
Testing battery discharge time at a load nearly equal to the motor load is a first step only. A stack of light bulbs is easy and convenient, although a purist will argue. That will confirm that the battery is basically good or bad. You just time how long it takes for volts to drop below 1.1V/cell NiMhl (1.5 for lead acid, 3.3V for lithium) but do NOT flatten the battery much more than this or you may wreck it, especially lithiums.
5 mins after you start this test, take the light off measure the battery volts, put the light back and see how much the battery drops. That is a crude measure of battery internal impedance. You can repeat that test at any time during the discharge cycle. High battery impedance will always result in problems. For NiCd and NiMh battery holders with springs are an absolute no go because they will double the battery impedance. Strong clips may be OK for a small motor. So connections are critical. Soldering some materials needs care because you may get a poor joint even if mechanically it seems strong. Gold plating is a well known culprit. It starts OK and vibrates off.
The gold standard is the oscilloscope. You just set the boat going and look at the power supply. (Light motor loads can be misleading so pushing against the side of a tank is best)
All is revealed by this test, if you know how to interpret what you see.
In time, I'll be putting some pictures on my blog when I cover H bridges, and how to sleep at night.