Doubt you're doing anything wrong - you've just subscribed to MF's and other's new view (erroneous in my opinion) that you should align the blade to the table. Your table just doesn't have enough adjustment to get you there.
A fence that adjusted for drift would though.
Which, yours does except that it's a total PIA and doesn't hold well.