I’ve always thought the cut-off is whether it’s near the speaker (“here”) or near the person being spoken to (“there”). My native language has a three-way distinction (near the speaker (“dito”), near the person spoken to (“diyan”), far from both (“doon”)), so it’s pretty easy to just collapse it to “here” and “there”.