BishtRef
Articles Journals Publishers Authors Subjects Most Cited New Papers Year Stats Open Access
journal article Nov 11, 2019

Attention Models in Graphs

John Boaz Lee Ryan A. Rossi ORCID Sungchul Kim Nesreen K. Ahmed Eunyee Koh
ACM Transactions on Knowledge Discovery from Data Vol. 13 No. 6 pp. 1-25 · Association for Computing Machinery (ACM)
View at Publisher Save 10.1145/3363574
Abstract
Graph-structured data arise naturally in many different application domains. By representing data as graphs, we can capture entities (i.e., nodes) as well as their relationships (i.e., edges) with each other. Many useful insights can be derived from graph-structured data as demonstrated by an ever-growing body of work focused on graph mining. However, in the real-world, graphs can be both large—with many complex patterns—and noisy, which can pose a problem for effective graph mining. An effective way to deal with this issue is to incorporate “attention” into graph mining solutions. An attention mechanism allows a method to focus on task-relevant parts of the graph, helping it to make better decisions. In this work, we conduct a comprehensive and focused survey of the literature on the emerging field of graph attention models. We introduce three intuitive taxonomies to group existing work. These are based on problem setting (type of input and output), the type of attention mechanism used, and the task (e.g., graph classification, link prediction). We motivate our taxonomies through detailed examples and use each to survey competing approaches from a unique standpoint. Finally, we highlight several challenges in the area and discuss promising directions for future work.
Topics

No keywords indexed for this article. Browse by subject →

References
90
[1]
Sami Abu-El-Haija , Bryan Perozzi , Rami Al-Rfou , and Alex Alemi . 2018 . Watch your step: Learning node embeddings via graph attention . In Proc. NeurIPS. 9198--9208 . Sami Abu-El-Haija, Bryan Perozzi, Rami Al-Rfou, and Alex Alemi. 2018. Watch your step: Learning node embeddings via graph attention. In Proc. NeurIPS. 9198--9208.
[2]
Charu C. Aggarwal , Amotz Bar-Noy , and Simon Shamoun . 2017. On sensor selection in linked information networks. Computer Networks 126 , C ( 2017 ), 100--113. Charu C. Aggarwal, Amotz Bar-Noy, and Simon Shamoun. 2017. On sensor selection in linked information networks. Computer Networks 126, C (2017), 100--113.
[3]
Charu C. Aggarwal and Haixun Wang . 2010 . Graph Data Management and Mining: A Survey of Algorithms and Applications. In Advances in Database Systems, Vol. 40 . Springer . Charu C. Aggarwal and Haixun Wang. 2010. Graph Data Management and Mining: A Survey of Algorithms and Applications. In Advances in Database Systems, Vol. 40. Springer.
[4]
Nesreen K. Ahmed , Nick Duffield , Theodore L. Willke , and Ryan A . Rossi . 2017 . On sampling from massive graph streams. In Proc. VLDB. 1430--1441. Nesreen K. Ahmed, Nick Duffield, Theodore L. Willke, and Ryan A. Rossi. 2017. On sampling from massive graph streams. In Proc. VLDB. 1430--1441.
[6]
Nesreen K. Ahmed , Ryan Rossi , John Boaz Lee , Xiangnan Kong, Theodore L. Willke, Rong Zhou, and Hoda Eldardiry. 2018 . Learning role-based graph embeddings. arXiv:1802.02896. Nesreen K. Ahmed, Ryan Rossi, John Boaz Lee, Xiangnan Kong, Theodore L. Willke, Rong Zhou, and Hoda Eldardiry. 2018. Learning role-based graph embeddings. arXiv:1802.02896.
[7]
Graph based anomaly detection and description: a survey

Leman Akoglu, Hanghang Tong, Danai Koutra

Data Mining and Knowledge Discovery 10.1007/s10618-014-0365-y
[11]
Dzmitry Bahdanau , KyungHyun Cho , and Yoshua Bengio . 2015 . Neural machine translation by jointly learning to align and translate . In Proc. ICLR. 1--15 . Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. ICLR. 1--15.
[13]
Andy Brown Aaron Tuor Brian Hutchinson and Nicole Nichols. 2018. Recurrent neural network attention mechanisms for interpretable system log anomaly detection. arXiv:1803.04967v1. Andy Brown Aaron Tuor Brian Hutchinson and Nicole Nichols. 2018. Recurrent neural network attention mechanisms for interpretable system log anomaly detection. arXiv:1803.04967v1. 10.1145/3217871.3217872
[14]
Hongyun Cai , Vincent W. Zheng , and Kevin Chen-Chuan Chang . 2018 . A comprehensive survey of graph embedding: Problems, techniques and applications . ACM TKDE 30 , 9(2018), 1616 -- 1637 . Hongyun Cai, Vincent W. Zheng, and Kevin Chen-Chuan Chang. 2018. A comprehensive survey of graph embedding: Problems, techniques and applications. ACM TKDE 30, 9(2018), 1616--1637.
[15]
Jianfei Chen , Jun Zhu , and Le Song . 2018 . Stochastic training of graph convolutional networks with variance reduction . In Proc. of ICML. 941--949 . Jianfei Chen, Jun Zhu, and Le Song. 2018. Stochastic training of graph convolutional networks with variance reduction. In Proc. of ICML. 941--949.
[17]
Edward Choi , Mohammad Taha Bahadori , Jimeng Sun , Joshua Kulas , Andy Schuetz , and Walter Stewart . 2016 . RETAIN: An interpretable predictive model for healthcare using reverse time attention mechanism . In Proc. NeurIPS. 3504--3512 . Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. 2016. RETAIN: An interpretable predictive model for healthcare using reverse time attention mechanism. In Proc. NeurIPS. 3504--3512.
[18]
Wenyuan Dai , Gui-Rong Xue , Qiang Yang , and Yong Yu . 2007 . Transferring naive bayes classifiers for text classification . In Proc. AAAI. 540--545 . Wenyuan Dai, Gui-Rong Xue, Qiang Yang, and Yong Yu. 2007. Transferring naive bayes classifiers for text classification. In Proc. AAAI. 540--545.
[19]
Shuiguang Deng , Longtao Huang , Guandong Xu , Xindong Wu , and Zhaohui Wu . 2017 . On deep learning for trust-aware recommendations in social networks . IEEE TNNLS 28 , 5 (2017), 1164 -- 1177 . Shuiguang Deng, Longtao Huang, Guandong Xu, Xindong Wu, and Zhaohui Wu. 2017. On deep learning for trust-aware recommendations in social networks. IEEE TNNLS 28, 5 (2017), 1164--1177.
[21]
David Duvenaud , Dougal Maclaurin , Jorge Aguilera-Iparraguirre , Rafael Gomez-Bombarelli , Timothy Hirzel , Alan Aspuru-Guzik , and Ryan P . Adams . 2015 . Convolutional networks on graphs for learning molecular fingerprints. In Proc. NeurIPS. 2224--2232. David Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Gomez-Bombarelli, Timothy Hirzel, Alan Aspuru-Guzik, and Ryan P. Adams. 2015. Convolutional networks on graphs for learning molecular fingerprints. In Proc. NeurIPS. 2224--2232.
[22]
Jun Feng , Minlie Huang , Yang Yang , and Xiaoyan Zhu . 2016 . GAKE: Graph aware knowledge embedding . In Proc. COLING. 641--651 . Jun Feng, Minlie Huang, Yang Yang, and Xiaoyan Zhu. 2016. GAKE: Graph aware knowledge embedding. In Proc. COLING. 641--651.
[23]
Andrea Galassi Marco Lippi and Paolo Torroni. 2019. Attention please! A critical review of neural attention models in natural language processing. arXiv:1902.02181v1. Andrea Galassi Marco Lippi and Paolo Torroni. 2019. Attention please! A critical review of neural attention models in natural language processing. arXiv:1902.02181v1.
[24]
Learning to Forget: Continual Prediction with LSTM

Felix A. Gers, Jürgen Schmidhuber, Fred Cummins

Neural Computation 10.1162/089976600300015015
[26]
Community structure in social and biological networks

M. Girvan, M. E. J. Newman

Proceedings of the National Academy of Sciences 10.1073/pnas.122653799
[27]
Ian J. Goodfellow , Jean Pouget-Abadie , Mehdi Mirza , Bing Xu , David Warde-Farley , Sherjil Ozair , Aaron C. Courville , and Yoshua Bengio . 2014 . Generative adversarial nets . In Proc. NeurIPS. 2672--2680 . Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proc. NeurIPS. 2672--2680.
[28]
Alex Graves Greg Wayne and Ivo Danihelka. 2014. Neural turing machines. arXiv:1410.5401. Alex Graves Greg Wayne and Ivo Danihelka. 2014. Neural turing machines. arXiv:1410.5401.
[30]
Junliang Guo , Linli Xu , and Enhong Cheng . 2018 . SPINE: Structural identity preserved inductive network embedding . In Proc. of IJCAI. 2399--2405 . Junliang Guo, Linli Xu, and Enhong Cheng. 2018. SPINE: Structural identity preserved inductive network embedding. In Proc. of IJCAI. 2399--2405.
[31]
Will Hamilton , Zhitao Ying , and Jure Leskovec . 2017 . Inductive representation learning on large graphs . In Proc. NeurIPS. 1--11 . Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Proc. NeurIPS. 1--11.
[32]
Xu Han , Zhiyuan Liu , and Maosong Sun . 2018 . Neural knowledge acquisition via mutual attention between knowledge graph and text . In Proc. AAAI. 1--8. Xu Han, Zhiyuan Liu, and Maosong Sun. 2018. Neural knowledge acquisition via mutual attention between knowledge graph and text. In Proc. AAAI. 1--8.
[35]
Binbin Hu , Chuan Shi , Wayne Xin Zhao, and Philip S. Yu . 2018 . Leveraging meta-path based context for Top-N recommendation with a neural co-attention model. In Proc. KDD. 1531--1540. Binbin Hu, Chuan Shi, Wayne Xin Zhao, and Philip S. Yu. 2018. Leveraging meta-path based context for Top-N recommendation with a neural co-attention model. In Proc. KDD. 1531--1540.
[37]
Thomas N. Kipf and Max Welling . 2017 . Semi-supervised classification with graph convolutional networks. In Proc. ICLR. 1--14. Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proc. ICLR. 1--14.
[38]
Ankit Kumar , Ozan Irsoy , Peter Ondruska , Mohit Iyyer , James Bradbury , Ishaan Gulrajani , Victor Zhong , Romain Paulus , and Richard Socher . 2016 . Ask me anything: Dynamic memory networks for natural language processing . In Proc. ICML. 2397--2406 . Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016. Ask me anything: Dynamic memory networks for natural language processing. In Proc. ICML. 2397--2406.
[44]
Qi Liu , Biao Xiang , Nicholas Jing Yuan , Enhong Chen, Hui Xiong, Yi Zheng, and Yu Yang. 2017 . An influence propagation view of PageRank. ACM TKDD 11, 3 (2017), 30:1--30:30. Qi Liu, Biao Xiang, Nicholas Jing Yuan, Enhong Chen, Hui Xiong, Yi Zheng, and Yu Yang. 2017. An influence propagation view of PageRank. ACM TKDD 11, 3 (2017), 30:1--30:30.
[46]
Ye Liu , Lifang He , Bokai Cao , Philip S. Yu , Ann B. Ragin , and Alex D . Leow . 2018 a. Multi-view multi-graph embedding for brain network clustering analysis. In Proc. AAAI. 117--124. Ye Liu, Lifang He, Bokai Cao, Philip S. Yu, Ann B. Ragin, and Alex D. Leow. 2018a. Multi-view multi-graph embedding for brain network clustering analysis. In Proc. AAAI. 117--124.
[48]
Qing Lu and Lise Getoor . 2003 . Link-based classification . In Proc. ICML. 496--503 . Qing Lu and Lise Getoor. 2003. Link-based classification. In Proc. ICML. 496--503.
[49]
Thang Luong , Hieu Pham , and Christopher D . Manning . 2015 . Effective approaches to attention-based neural machine translation. In Proc. EMNLP. 1412--1421. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proc. EMNLP. 1412--1421.

Showing 50 of 90 references

Cited By
186
Unfolding graph convolutional networks: Architectures, use cases, and trends

Nematollah Saeidi, Vahid Hajipour · 2026

Applied Soft Computing
A Multiple Attention Layer-shareable Method for Link Prediction in Multilayer Networks

Huan Wang, Yu Teng · 2025

ACM Transactions on Knowledge Disco...
Visual attention methods in deep learning: An in-depth survey

Mohammed Hassanin, Saeed Anwar · 2024

Information Fusion
2F-TP: Learning Flexible Spatiotemporal Dependency for Flexible Traffic Prediction

Jie Zhao, Chao Chen · 2023

IEEE Transactions on Intelligent Tr...
Efficient Node PageRank Improvement via Link-building using Geometric Deep Learning

Vincenza Carchiolo, Marco Grassia · 2023

ACM Transactions on Knowledge Disco...
A General Survey on Attention Mechanisms in Deep Learning

Gianni Brauwers, Flavius Frasincar · 2023

IEEE Transactions on Knowledge and...
Deep Learning on Graphs: A Survey

Ziwei Zhang, Peng Cui · 2022

IEEE Transactions on Knowledge and...
Joint Communication and Action Learning in Multi-Target Tracking of UAV Swarms with Deep Reinforcement Learning

Wenhong Zhou, Jie Li · 2022

Drones
Deep Graph Generators: A Survey

Faezeh Faez, Yassaman Ommi · 2021

IEEE Access
Computing Graph Neural Networks: A Survey from Algorithms to Accelerators

Sergi Abadal, Akshay Jain · 2021

ACM Computing Surveys
Hypergraph convolution and hypergraph attention

Song Bai, Feihu Zhang · 2021

Pattern Recognition
Graph Convolutional Neural Network for Human Action Recognition: A Comprehensive Survey

Tasweer Ahmad, Lianwen Jin · 2021

IEEE Transactions on Artificial Int...
Deep multi-level fusion network for multi-source image pixel-wise classification

Licheng Jiao, Lingling Li · 2021

Knowledge-Based Systems
Metrics
186
Citations
90
References
Details
Published
Nov 11, 2019
Vol/Issue
13(6)
Pages
1-25
License
View
Authors
J
John Boaz Lee

WPI, MA

R
Ryan A. Rossi

Adobe Research, San Jose, CA

S
Sungchul Kim

Adobe Research, San Jose, CA

N
Nesreen K. Ahmed

Intel Labs, Santa Clara, CA

E
Eunyee Koh

Adobe Research, San Jose, CA

Cite This Article
John Boaz Lee, Ryan A. Rossi, Sungchul Kim, et al. (2019). Attention Models in Graphs. ACM Transactions on Knowledge Discovery from Data, 13(6), 1-25. https://doi.org/10.1145/3363574
Related

You May Also Like

Graph evolution

Jure Leskovec, Jon Kleinberg · 2007

2,024 citations

Isolation-Based Anomaly Detection

Fei Tony Liu, Kai Ming Ting · 2012

1,600 citations

Hierarchical Density Estimates for Data Clustering, Visualization, and Outlier Detection

Ricardo J. G. B. Campello, Davoud Moulavi · 2015

673 citations

A Survey on Causal Inference

Liuyi Yao, Zhixuan Chu · 2021

376 citations

Temporal Link Prediction Using Matrix and Tensor Factorizations

Daniel M. Dunlavy, Tamara G. Kolda · 2011

351 citations