if space is at a premium, you will want to use the largest radius curves that will fit, then you need to be a bit more 'scientific' than suggested so far.....
you need to leave sufficient room for the vehicle with the longest overhang to get around without clouting backscenes or overhanging the edge of the board
as a good rule of thumb, I would suggest the same clearances as one would find between double tracks on the curve.
I have used a 2" gap on my son's layout......upgrading from approx no.1 curves to No.2 or 3......which is a bit academic as I used flex track, but you get my drift?
try temporarily laying some track, and try your longest coaches round them.....hold a pencil on the outer corner of a coach,and trace the line on the board.....this will give some idea of what space to leave???
also remember, unless you are using curved boards, then the actual points where tracks really get close to the edge will be few.
I reckon a minimum of a coupla inches, all round, should be ok.........I ignore little fingers....