I buy my small drill bits from
https://www.microscrews-shop.nl/ Excellent quality. I use them in steel and stainless as well as aluminium. For brass I dull them slightly and keep in separate box.
Are you drilling tool steel (aka silver steel) or High Speed Steel? Tool steel is no problem for the drills I buy above.
His hardware is also of excellent quality.
(I don't pay the VAT because in Canada, which offsets the shipping cost.)