The hardware lottery

Sara Hooker

doi:10.1145/3467017

journal article Open Access Nov 19, 2021

The hardware lottery

Sara Hooker

Communications of the ACM Vol. 64 No. 12 pp. 58-65 · Association for Computing Machinery (ACM)

View at Publisher Save 10.1145/3467017

Abstract

After decades of incentivizing the isolation of hardware, software, and algorithm development, the catalysts for closer collaboration are changing the paradigm.

Topics

No keywords indexed for this article. Browse by subject →

References

50

[1]

Amodei, D., Hernandez, D., Sastry, G., Clark, J., Brockman, G., and Sutskever, I. AI and compute. OpenAI (2018), https://openai.com/blog/ai-and-compute/.

[2]

ARM. Enhancing AI performance for IoT endpoint devices. (2020), https://www.arm.com/company/news/2020/02/new-ai-technology-from-arm

[3]

10.1145/3317550.3321441

[4]

When and where do we apply what we learn?: A taxonomy for far transfer.

Susan M. Barnett, Stephen J. Ceci

Psychological Bulletin 10.1037/0033-2909.128.4.612

[5]

Synaptic Modifications in Cultured Hippocampal Neurons: Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type

Guo-qiang Bi, Mu-ming Poo

The Journal of Neuroscience 10.1523/jneurosci.18-24-10464.1998

[6]

10.3389/fnhum.2010.00025

[7]

Chellapilla, K., Puri, S., and Simard, P. High performance convolutional neural networks for document processing. Tenth International Workshop on Frontiers in Handwriting Recognition (2006).

[8]

Coates, A., Huval, B., Wang, T., Wu, D., Catanzaro, B., and Andrew, N. Deep learning with COTS HPC systems. In Proceedings of the 30th Intern. Conf. on Machine Learning (2013), Sanjoy Dasgupta and David McAllester (Eds.). PMLR, Atlanta, GA, USA, 1337--1345. http://proceedings.mlr.press/v28/coates13.html

[9]

Collier, B. Little engines that could've: The calculating machines of Charles Babbage. Garland Publishing, Inc. (1991) USA.

[10]

Computer history 1949--1960: Early vacuum tube computers overview. Computer History Archives Project (2018), https://www.youtube.com/watch?v=WnNm_uJYWhA

[11]

Dean, J. The deep learning revolution and its implications for computer architecture and chip design. IEEE International Solid-State Circuits Conference (2020), 8--14.

[12]

10.1109/jssc.1974.1050511

[13]

10.1109/jproc.2018.2868961

[14]

Feldman, M. The era of general-purpose computers is ending. The Next Platform (2019), https://bit.ly/3hP8XJh

[15]

Neocognitron: A new algorithm for pattern recognition tolerant of deformations and shifts in position

Kunihiko Fukushima, Sei Miyake

Pattern Recognition 10.1016/0031-3203(82)90024-3

[16]

Gupta, S. and Tan, M. EfficientNet-Edge TPU: Creating accelerator-optimized neural networks with AutoML. Google AI Blog (2019), https://ai.googleblog.com/2019/08/efficientnet-edgetpu-creating.html

[17]

Hauck, S. and DeHon, A. Reconfigurable Computing: The Theory and Practice of FPGA-Based Computation. (2017), Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.

[18]

Herculano-Houzel, S., et al. The elephant brain in numbers. Frontiers in Neuroanatomy 8 (2014).

[19]

Hinton, G. and Anderson, J. Parallel Models of Associative Memory. (1989), L. Erlbaum Associates Inc., USA.

[20]

Horowitz, M. Computing's energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC). 10--14.

[21]

Hotel, H., Johansen, H., Bernholdt, D., Héroux, M., and Hornung, R. Software productivity for extreme-scale science. U.S. Department of Energy Advanced Scientific Computing Research (2014).

[22]

Isaacson, W. Grace Hopper, computing pioneer. The Harvard Gazette (2014). https://news.harvard.edu/gazette/story/2014/12/grace-hopper-computing-pioneer/

[23]

10.1145/3140659.3080246

[24]

Kuhn, T. The Structure of Scientific Revolutions. (1962), University of Chicago Press, Chicago.

[25]

Kurzweil, R. The Age of Intelligent Machines. (1990), MIT Press, Cambridge, MA, USA.

[26]

10.1145/1506409.1506425

[27]

Backpropagation Applied to Handwritten Zip Code Recognition

Y. Lecun, B. Boser, J. S. Denker et al.

Neural Computation 10.1162/neco.1989.1.4.541

[28]

10.1109/mm.2011.68

[29]

10.1007/bf01931367

[30]

Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory.

James L. McClelland, Bruce L. McNaughton, Randall C. O'Reilly

Psychological Review 10.1037/0033-295x.102.3.419

[31]

A graph placement methodology for fast chip design

Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan et al.

Nature 10.1038/s41586-021-03544-w

[32]

10.1080/20018091094349

[33]

Moore, G. 1965. Cramming more components onto integrated circuits. Electronics 38, 8 (April 1965). https://www.cs.utexas.edu/~fussell/courses/cs352h/papers/moore.pdf

[34]

Moravec, H. When will computer hardware match the human brain. Journal of Transhumanism 1 (1998).

[35]

10.1145/2692916.2557966

[36]

Posselt E.A. The Jacquard Machine Analyzed and Explained: The Preparation of Jacquard Cards and Practical Hints to Learners of Jacquard Designing. (1888).

[37]

Prabhakar, R., et al. Plasticine: A reconfigurable architecture for parallel patterns. In 2017 ACM/IEEE 44th Annual Intern. Symposium on Computer Architecture (ISCA). 389--402.

[38]

10.1109/isca45697.2020.00045

[39]

Rumelhart, D., Hinton, G., and Williams, R. Learning representations by back-propagating errors. MIT Press (1988), 696--699.

[40]

Sabour S. Frost N. and Hinton G. Dynamic routing between capsules. (2017) 3856--3866. http://papers.nips.cc/paper/6975-dynamic-routing-between-capsules.pdf

[41]

Shalf, J. The future of computing beyond Moore's law. Philosophical Transactions of the Royal Society A, 378 (2020).

[42]

10.1145/2688498.2688499

[43]

10.1109/pgec.1963.263588

[44]

Studies of interference in serial verbal reactions.

J. R. Stroop

Journal of Experimental Psychology 10.1037/h0054651

[45]

Thompson, N. and Spanuth, S. The decline of computers as a general purpose technology: Why deep learning and the end of Moore's Law are fragmenting computing. (November 2018).

[46]

Thompson, N., Greenewald, K., Lee, K., and Manso, G. The computational limits of deep learning. arXiv e-prints, Article arXiv:2007.05558 (July 2020), arXiv:2007.05558 pages. arXiv:2007.05558 [cs.LG]

[47]

10.1093/owc/9780198748847.001.0001

[48]

10.1007/978-3-642-70911-1_20

[49]

Von Neumann, J., Churchland, P.M., and Churchland, P.S. The Computer and the Brain. Yale University Press (2000), https://books.google.com/books?id=Q30MqJjRv1gC

[50]

Warden, P. and Situnayake, D. TinyML: Machine Learning With TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers. (2019), O'Reilly Media, Inc. https://books.google.com/books?id=sB3mxQEACAAJ

Metrics

104

Citations

50

References

Details

Published: Nov 19, 2021
Vol/Issue: 64(12)
Pages: 58-65
License: View

Authors

S

Sara Hooker

Google Brain, Mountain View, CA

Cite This Article

Sara Hooker (2021). The hardware lottery. Communications of the ACM, 64(12), 58-65. https://doi.org/10.1145/3467017

The hardware lottery

You May Also Like