Improving Negative Sampling for Word Representation Using Self-embedded Features.

Long Chen, Fajie Yuan, Joemon M Jose, Weinan Zhang

Research output: Chapter in Book/Report/Conference proceedingConference proceedingpeer-review

18 Citations (Scopus)

Abstract

Although the word-popularity based negative sampler has shown superb performance in the skip-gram model, the theoretical motivation behind oversampling popular (non-observed) words as negative samples is still not well understood. In this paper, we start from an investigation of the gradient vanishing issue in the skip-gram model without a proper negative sampler. By performing an insightful analysis from the stochastic gradient descent (SGD) learning perspective, we demonstrate, both theoretically and intuitively, negative samples with larger inner product scores are more informative than those with lower scores for the SGD learner in terms of both convergence rate and accuracy. Understanding this, we propose an alternative sampling algorithm that dynamically selects informative negative samples during each SGD update. More importantly, the proposed sampler accounts for multi-dimensional self-embedded features during the sampling process, which essentially makes it more effective than the original popularity-based (one-dimensional) sampler. Empirical experiments further verify our observations and show that our fine-grained samplers gain significant improvement over the existing ones without increasing computational complexity.
Original languageEnglish
Title of host publicationWSDM '18: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining
PublisherACM
Pages99-107
Number of pages9
ISBN (Print)978-1-4503-5581-0
DOIs
Publication statusPublished - 2 Feb 2018
Externally publishedYes
Event11th ACM International Conference on Web Search and Data Mining - New York, United States
Duration: 5 Feb 20189 Feb 2018

Conference

Conference11th ACM International Conference on Web Search and Data Mining
Abbreviated titleWSDM'18'
Country/TerritoryUnited States
CityNew York
Period5/02/189/02/18

Fingerprint

Dive into the research topics of 'Improving Negative Sampling for Word Representation Using Self-embedded Features.'. Together they form a unique fingerprint.

Cite this