Text this: Model-independent test of the cosmic anisotropy with inverse distance ladder