Using the right kind of drill and bit should help with the wandering...a hammer drill, for example. You might also try to hit the grout line as that's easier to drill into than the actual brick for your anchors. A properly sized anchor and associated screw shouldn't cause an excessive force issue in the masonry.