a thoughtful web.
Good ideas and conversation. No ads, no tracking.   Login or Take a Tour!
comment
hogwild  ·  3385 days ago  ·  link  ·    ·  parent  ·  post: Favorite Science Journal Articles

Almost everyone in my field (natural language processing) is currently working on word embeddings (and is sick of everyone working on word embeddings). Word embeddings are continuous vector representations of words based on their representation that can be used as inputs in place of specifying the exact word as input to a statistical model like a neural net. So you don't have to learn how to deal with the word "hamburger" in every situation as long as you know how to deal with words that are distributed similarly to "hamburger".

The cause is one paper from a team at Google, which used a shallow neural net that developed representations for tasks like predicting a word from its context. The representations they found had the property that you could perform analogical reasoning with them, so roughly speaking you could add and subtract vectors to find "paris" - "france" + "italy" = "rome".

Mikolov et al. (2013). Distributed Representations of Words and Phrases and their Compositionality.

And a paper last year found that this neural model, with its regularizer and everything, was actually implicitly performing matrix factorization on a matrix where cell (i,j) described how often word i and word j occur together. This was incredible, because NLP researchers had been using matrix factorization of this sort to find word embeddings for a long time, and this fancy new technique for learning embeddings suddenly had a clear intuition for why it worked!

Levy and Goldberg (2014). Neural Word Embedding as Implicit Matrix Factorization.