The Language of Deep Learning

Have you ever asked your phone’s intelligent personal assistant a question, and gotten something unintelligible or irrelevant in reply?

Clearly, we still have some way to go before reality catches up with sci-fi movies such as Star Wars or Star Trek, in which humans communicate easily and naturally with droids and computers.

Human languages, with all their complexities and subtleties — say, double meanings, puns, paraphrases and slang, for instance — are a huge challenge for computers to understand and respond to.

The field of natural language processing seeks to address this, combining computer science, artificial intelligence and computational linguistics to teach computers to process and understand language.

With natural language processing capabilities, computers can be used to extract information and insights from large bodies of unstructured textual data, such as feedback emails from customers, or social media posts mentioning a company or product.

New techniques in deep learning have greatly advanced the field of natural language processing, said Facebook software engineer Dr Vikramank Singh, who was speaking on 27 April 2017 at the Deep Learning Summit Singapore.

This has allowed the development of algorithms that are better able to deal with the nuances of human languages, he added.

Deep neural networks

In traditional natural language processing, words are converted into numerical representations, a format that computers can understand. While machine learning techniques have been used in the field for some time, said Dr Singh, these have mostly been used to optimise word representations that still have to be designed by humans.

“We want to move beyond handcrafting features, and towards models that allow for human-level representations for learning and reasoning,” he added.

Deep learning, on the other hand, uses deep neural networks to develop representations that are better suited to more complex interpretation tasks, said Dr Singh.

For example, instead of assigning each word its own representation, as is done in traditional natural language processing, deep learning techniques allow semantically related words (‘bro’ and ‘brother’, for example) to be clustered together. In addition, they can also help computers do a better job when dealing with the recursive nature of human languages.

A sentence such as “John told Mary that we knew that she had gone to see you,” for example, contains several embedded phrases that can be difficult for computers to parse using traditional techniques.

These capabilities give the computer a more human-like understanding of language, said Dr Singh.

“Deep learning models allow for biologically inspired learning,” he added.

We can’t get no supervision

Deep neural networks can be trained in a supervised or unsupervised fashion. In the former, more commonly used approach, computers are trained on a set of inputs for which the targets, or answers, are known. In the latter approach, computers are left to figure out these relationships on their own.

While unsupervised approaches allow for more generalised AI applications that are not limited to one function, they can also be more difficult to set up.

Dr Singh and his colleagues found that including an unsupervised pre-training component prior to supervised training improved the performance of their algorithms, which made fewer classification errors as compared to when only supervised training was used.

They have also used recurrent neural networks, which contain loops that allow information to be passed between different parts of the network, to improve their algorithms.

Recurrent neural networks allow information to persist for longer, meaning that the computer does not have to start from scratch each time a new set of input data arrives.

Using these recursive techniques, the researchers are training computers to carry out paraphrase detection — recognising the degree to which sentences such as “The cats catch mice” and “The cats eat mice” mean similar things.

Deep diving into text

Facebook uses natural language processing capabilities to make sense of the millions of largely text-based communications that go through its platforms on a daily basis.

In 2016, the social media giant announced DeepText, a deep learning-based algorithm that can understand several thousand posts per second, in more than 20 languages, with “near-human accuracy.” While the algorithm is still in the testing phase, the company says that it will help show users more relevant content, and at the same time filter out spam.

But DeepText can also perform more complex tasks, such as intelligent search within Messenger, Facebook’s chat programme.

Typing “I need a ride” or “Let’s take a ride there” prompts DeepText to offer the option to book a taxi; on the other hand, the algorithm is smart enough not to offer this option in response to phrases such as “I don’t need a ride” or “I like to ride donkeys.”

DeepText also learns from experience — the more posts and conversations it processes, the better it gets at doing so.

This capability, combined with new deep learning techniques such as the ones Dr Singh is developing, should help to move the field of natural language processing forward.

And thus one day, digital agents and robots might converse on the level seen in Hollywood movies.