This research is driven by our curiosity about the limits to which the current technology for natural language processing could be pushed: if humans are capable of reading jumbled sentences, can machines do? To answer this question, we evaluate the ability of state-of-the-art models to recognize and comprehend wordand character-level jumbled sentences, analyze factors that may influence machine’s performance, and devise potential cures for enhancing models’ robustness. The quality of sentences embeddings is investigated on three classic downstream tasks mimicking human cognitive abilities, including sentiment classification, information retrieval, and semantic similarity, to reflects machines’ capability in reading jumbled sentences. We discover induced biases in embeddings for jumbling sentences which impairs machines performance. The removal of these induced biases significantly improves machines’ robustness of reading character-level jumbled sentences on all three tasks.
Authors
Runzhe Yang, Zhongqiao Gao