I built a small plywood work table for the drill press:

Obviously, that was a long time ago. It’s a plywood scrap with a small cleat screwed to its bottom, upon which one can position / clamp / hold / finagle smallish workpieces without worrying about drilling into the surface.
The most recent batch of aluminum backing plates prompted me to finally replace that relic:

The mill vise under the plywood grips the cleat and the whole affair rides on a Sears “Drill Press Milling Attachment Stock No 27585” which is basically a simple XY table with hand dials. It’s not rigid enough for actual milling (which you should never do on a drill press, anyway, because the end mill will pull itself out of the Jacobs chuck), but it’s good for tweaking the position before you drill something.
One should never hand-hold workpieces while drilling.
Don’t do as I do, do as I say. OK?