Brace yourself, lecture incoming.
Reliability in coupling is a function of six principal distinct factors:
Uniform coupler mounting and alignment on all vehicles.
Free rolling vehicles.
Track laying standard.
Smooth mechanical performance of traction
Coupler design. This is simple, the old 'Rapido' coupler used for European N gauge is very poor. There are superior alternatives, Microtrains about the best in RTR autocouplers, but simple DIY ' bent wire' body mounted permanent couplings will perform just as well for fixed formations.
Uniform coupler mounting and alignment on all vehicles. It doesn't matter what coupler is used, this is essential. If there isn't a manufacturer positioning gauge for a RTR coupler, make one yourself, and use it without fail.
Free rolling vehicles. Really important, one draggy vehicle can ruin all in a train of more than 3 or 4 coaches. A good starting standard is vehicles should roll away from rest if placed on a true 1 in 100.
Track laying standard. A smooth rail top and longitudinal alignment. Largest possible radius curves, and no kinks or 'dog legs' horizontally or vertically; no protrusions above chair tops of track pins, ballast, 'scenic features' or anything else. (Hilarious example, a layout and stock which was fully 'sorted' and really reliable, suddenly suffering from random uncouples. To the great chagrin of the owner it was the prototypical newly arranged loose rails in the 'four foot' that just disturbed the occasional coupler, resulting in an unexpected uncouple, sometime after.)
Smooth mechanical performance of traction. The traction should 'creep' in and out of motion and perform smoothly throughout the speed range, no bumping or jerking.
Operator skill. Operate your superbly performing traction the way the real railway driver does. (A major disappointment for me at shows are operators that don't use the excellent inertia simulation available on DCC, which does this way better than a human hand on knob, slider, button etc.. But no, the train arrives on platform still doing 50mph, and crash stops to rest in two coach lengths. Sigh!)
You have to be the engineer in observing all the above, and making the corrections as required to deliver the desired performance.