So I messed arounf a bit and tried a few different things suggested here.
THe one thing I did not try yet was to champfer the holes. I dont have a champfer that big and my champfer bit for my router is crappy.
I did sand the shafts again and also dinged up the sides of the the shafts using the afore mentioned bigger hammer. That seemed to help the hold.
I also roughed up the inside of the holes but that didn't matter mush seeing that it doesn't contact the sides of the holes.
Phil mentioned the hourglass shape of his holes and as I kept trying things in the same hole it started to oval out in one direction which dramatically hlpe the holding to the point that if I pushed it down by hand it held really well.
I may try the champfer later this weekend and I think I may ding up the sides a bit more with the bigger hammer.
If all else fails I may try and get a holdfast created by a blacksmith, Flex Seal, or duct tape. ( Spare the duct tape, Spoil the job. Red Green)
Thank you all very much for the suggestions.
I will keep you posted.
Larry