Text this: Learning More May Not Be Better: Knowledge Transferability in Vision-and-Language Tasks