Finding communities in networks: message-passing, phase transitions, and new spectral methods

Detecting communities is an important problem in the study of networks.  Recently, we developed scalable Belief Propagation algorithms that update probability distributions of node labels until they reach a fixed point.  In addition to being of practical use, these algorithms can be studied analytically, revealing phase transitions in the ability of any algorithm to solve this problem.  Specifically, there is a "detectability transition" in the stochastic block model, below which no algorithm can label nodes better than chance, or even tell with high probability whether or not communities exist.

 

I'll explain this transition, and give an accessible introduction to Belief Propagation and the analogy with "free energy" and the cavity method of statistical physics.  We'll see that the consensus of many good solutions is a better labeling than the "best" solution --- something that is true for many real-world optimization problems.

 

I'll then turn to spectral methods.  It is popular to classify nodes according to the first few eigenvectors of the adjacency matrix or the graph Laplacian.  However, in the sparse case these operators get confused by localized eigenvectors, focusing on high-degree nodes or dangling trees rather than large-scale communities. As a result, they fail significantly above the detectability transition.  I will describe a new spectral algorithm based on the "non-backtracking matrix," which avoids these localized eigenvectors: it is optimal in the sense that it succeeds all the way down to the transition.

 

This is joint work with Aurelien Decelle, Florent Krzakala, Elchanan Mossel, Joe Neeman, Allan Sly, Lenka Zdeborova, and Pan Zhang.