DeepLabCut: markerless pose estimation of user-defined body parts with deep learning

Alexander Mathis; Pranav Mamidanna; Kevin M. Cury; Taiga Abe; Venkatesh N. Murthy; Mackenzie Weygandt Mathis; Matthias Bethge

doi:10.1038/s41593-018-0209-y

journal article Aug 20, 2018

DeepLabCut: markerless pose estimation of user-defined body parts with deep learning

Alexander Mathis

Pranav Mamidanna Kevin M. Cury Taiga Abe Venkatesh N. Murthy

Mackenzie Weygandt Mathis

Matthias Bethge

Nature Neuroscience Vol. 21 No. 9 pp. 1281-1289 · Springer Science and Business Media LLC

View at Publisher Save 10.1038/s41593-018-0209-y

Topics

No keywords indexed for this article. Browse by subject →

References

51

[1]

Tinbergen, N. On aims and methods of ethology. Z. Tierpsychol. 20, 410–433 (1963). 10.1111/j.1439-0310.1963.tb01161.x

[2]

Bernstein, N. A. The Co-ordination and Regulation of Movements Vol. 1 (Pergamon, Oxford and New York, 1967).

[3]

Krakauer, J. W., Ghazanfar, A. A., Gomez-Marin, A., MacIver, M. A. & Poeppel, D. Neuroscience needs behavior: correcting a reductionist bias. Neuron 93, 480–490 (2017). 10.1016/j.neuron.2016.12.041

[4]

Ota, N., Gahr, M. & Soma, M. Tap dancing birds: the multimodal mutual courtship display of males and females in a socially monogamous songbird. Sci. Rep. 5, 16614 (2015). 10.1038/srep16614

[5]

Wade, N. J. Capturing motion and depth before cinematography. J. Hist. Neurosci. 25, 3–22 (2016). 10.1080/0964704x.2015.1070029

[6]

Dell, A. I. et al. Automated image-based tracking and its application in ecology. Trends Ecol. Evol. 29, 417–428 (2014). 10.1016/j.tree.2014.05.004

[7]

Gomez-Marin, A., Paton, J. J., Kampff, A. R., Costa, R. M. & Mainen, Z. F. Big behavioral data: psychology, ethology and the foundations of neuroscience. Nat. Neurosci. 17, 1455–1462 (2014). 10.1038/nn.3812

[8]

Anderson, D. J. & Perona, P. Toward a science of computational ethology. Neuron 84, 18–31 (2014). 10.1016/j.neuron.2014.09.005

[9]

Winter, D. A. Biomechanics and Motor Control of Human Movement (Wiley, Hoboken, NJ, USA, 2009). 10.1002/9780470549148

[10]

Vargas-Irwin, C. E. et al. Decoding complete reach and grasp actions from local primary motor cortex populations. J. Neurosci. 30, 9659–9669 (2010). 10.1523/jneurosci.5443-09.2010

[11]

Wenger, N. et al. Closed-loop neuromodulation of spinal sensorimotor circuits controls refined locomotion after complete spinal cord injury. Sci. Transl. Med. 6, 255ra133 (2014). 10.1126/scitranslmed.3008325

[12]

Maghsoudi, O. H., Tabrizi, A. V., Robertson, B. & Spence, A. Superpixels based marker tracking vs. hue thresholding in rodent biomechanics application. Preprint at https://arxiv.org/abs/1710.06473 (2017). 10.1109/acssc.2017.8335168

[13]

Pérez-Escudero, A., Vicente-Page, J., Hinz, R. C., Arganda, S. & de Polavieja, G. G. idTracker: tracking individuals in a group by automatic identification of unmarked animals. Nat. Methods 11, 743–748 (2014). 10.1038/nmeth.2994

[14]

Nakamura, T. et al. A markerless 3D computerized motion capture system incorporating a skeleton model for monkeys. PLoS One 11, e0166154 (2016). 10.1371/journal.pone.0166154

[15]

de Chaumont, F. et al. Computerized video analysis of social interactions in mice. Nat. Methods 9, 410–417 (2012). 10.1038/nmeth.1924

[16]

Matsumoto, J. et al. A 3D-video-based computerized analysis of social and sexual interactions in rats. PLoS One 8, e78460 (2013). 10.1371/journal.pone.0078460

[17]

Uhlmann, V., Ramdya, P., Delgado-Gonzalo, R., Benton, R. & Unser, M. FlyLimbTracker: An active contour based approach for leg segment tracking in unmarked, freely behaving Drosophila. PLoS One 12, e0173433 (2017). 10.1371/journal.pone.0173433

[18]

Pictorial Structures for Object Recognition

Pedro F. Felzenszwalb, Daniel P. Huttenlocher

International Journal of Computer Vision 2005 10.1023/b:visi.0000042934.15159.49

[19]

Toshev, A. & Szegedy, C. DeepPose: human pose estimation via deep neural networks. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 1653–1660 (IEEE, Piscataway, NJ, USA, 2014).

[20]

Cascaded pose regression

Piotr Dollar, Peter Welinder, Pietro Perona

2010 IEEE Computer Society Conference on Computer... 2010 10.1109/cvpr.2010.5540094

[21]

Machado, A. S., Darmohray, D. M., Fayad, J., Marques, H. G. & Carey, M. R. A quantitative framework for whole-body coordination reveals specific deficits in freely walking ataxic mice. Elife 4, e07892 (2015). 10.7554/elife.07892

[22]

Cortex commands the performance of skilled movement

Jian-Zhong Guo, Austin R Graves, Wendy W Guo et al.

eLife 2015 10.7554/elife.10774

[23]

Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. in Advances in Neural Information Processing Systems Vol. 25 (eds. Pereira, F. et al.) 1097–1105 (Curran Associates, Red Hook, NY, USA, 2012).

[24]

Deep Residual Learning for Image Recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren et al.

2016 IEEE Conference on Computer Vision and Patter... 10.1109/cvpr.2016.90

[25]

Wei, S.-E., Ramakrishna, V., Kanade, T. & Sheikh, Y. Convolutional pose machines. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 4724–4732 (IEEE, Piscataway, NJ, USA, 2016).

[26]

Pishchulin, L. et al. DeepCut: joint subset partition and labeling for multi person pose estimation. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 4929–4937 (IEEE, Piscataway, NJ, USA, 2016).

[27]

Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M. & Schiele, B. DeeperCut: a deeper, stronger, and faster multi-person pose estimation model. in European Conference on Computer Vision 34–50 (Springer, New York, 2016).

[28]

Feichtenhofer, C., Pinz, A. & Zisserman, A. Detect to track and track to detect. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 3038–3046 (IEEE, Piscataway, NJ, USA, 2017).

[29]

Insafutdinov, E. et al. ArtTrack: articulated multi-person tracking in the wild. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 1293–1301 (IEEE, Piscataway, NJ, USA, 2017).

[30]

Andriluka, M., Pishchulin, L., Gehler, P. & Schiele, B. 2D human pose estimation: new benchmark and state of the art analysis. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 3686–3693 (IEEE, Piscataway, NJ, USA, 2014).

[31]

Donahue, J. et al. DeCaf: a deep convolutional activation feature for generic visual recognition. in I nternational Conference on Machine Learning 647–655 (PMLR, Beijing, 2014).

[32]

Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. How transferable are features in deep neural networks? in Advances in Neural Information Processing Systems 3320–3328 (Curran Associates, Red Hook, NY, USA, 2014).

[33]

Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning Vol. 1 (MIT Press, Cambridge, MA, USA, 2016).

[34]

Kümmerer, M., Wallis, T. S. & Bethge, M. DeepGaze II: reading fixations from deep features trained on object recognition. Preprint at https://arxiv.org/abs/1610.01563 (2016).

[35]

Khan, A. G., Sarangi, M. & Bhalla, U. S. Rats track odour trails accurately using a multi-layered strategy with near-optimal sampling. Nat. Commun. 3, 703 (2012). 10.1038/ncomms1712

[36]

Li, Y. et al. Neuronal representation of social information in the medial amygdala of awake behaving mice. Cell 171, 1176–1190.e17 (2017). 10.1016/j.cell.2017.10.015

[37]

Robie, A. A., Seagraves, K. M., Egnor, S. E. & Branson, K. Machine vision methods for analyzing social interactions. J. Exp. Biol. 220, 25–34 (2017). 10.1242/jeb.142281

[38]

Mathis, M. W., Mathis, A. & Uchida, N. Somatosensory cortex plays an essential role in forelimb motor adaptation in mice. Neuron 93, 1493–1503.e6 (2017). 10.1016/j.neuron.2017.02.049

[39]

Drai, D. & Golani, I. SEE: a tool for the visualization and analysis of rodent exploratory behavior. Neurosci. Biobehav. Rev. 25, 409–426 (2001). 10.1016/s0149-7634(01)00022-7

[40]

Sousa, N., Almeida, O. F. X. & Wotjak, C. T. A hitchhiker’s guide to behavioral analysis in laboratory rodents. Genes Brain Behav. 5 (Suppl. 2), 5–24 (2006). 10.1111/j.1601-183x.2006.00228.x

[41]

Gomez-Marin, A., Partoune, N., Stephens, G. J., Louis, M. & Brembs, B. Automated tracking of animal posture and movement during exploration and sensory orientation behaviors. PLoS One 7, e41642 (2012). 10.1371/journal.pone.0041642

[42]

Ben-Shaul, Y. OptiMouse: a comprehensive open source program for reliable detection and analysis of mouse body and nose positions. BMC Biol. 15, 41 (2017). 10.1186/s12915-017-0377-3

[43]

Zhang, C., Bengio, S., Hardt, M., Recht, B. & Vinyals, O. Understanding deep learning requires rethinking generalization. Preprint at https://arxiv.org/abs/1611.03530 (2016).

[44]

Berman, G. J. Measuring behavior across scales. BMC Biol. 16, 23 (2018). 10.1186/s12915-018-0494-7

[45]

Kim, C. K., Adhikari, A. & Deisseroth, K. Integration of optogenetics with complementary methodologies in systems neuroscience. Nat. Rev. Neurosci. 18, 222–235 (2017). 10.1038/nrn.2017.15

[46]

Adaptive background mixture models for real-time tracking

C. Stauffer, W.E.L. Grimson

Proceedings. 1999 IEEE Computer Society Conference... 10.1109/cvpr.1999.784637

[47]

Ristic, B., Arulampalam, S. & Gordon, N. Beyond the Kalman Filter: Particle Filters for Tracking Applications (Artech House, Norwood, MA, USA, 2003).

[48]

Carreira, J. & Zisserman, A. Quo vadis, action recognition? A new model and the kinetics dataset. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 4724–4733 (IEEE, Piscataway, NJ, USA, 2017).

[49]

Fiji: an open-source platform for biological-image analysis

Johannes Schindelin, Ignacio Arganda-Carreras, Erwin Frise et al.

Nature Methods 2012 10.1038/nmeth.2019

[50]

Abadi, M. et al. TensorFlow: a system for large-scale machine learning. Preprint at https://arxiv.org/abs/1605.08695 (2016).

Showing 50 of 51 references

Cited By

4,863

Dynamic modulation of social gaze by sex and familiarity in marmoset dyads

Feng Xing, Alec G Sheffield · 2026

eLife

Species-specific spectral tuning of motion vision in butterflies

Jack A. Supple, Uroš Cerkvenik · 2026

Current Biology

Applying Artificial Intelligence (AI) Innovative Tools for Ecological Research and Monitoring of Transitional Water Ecosystems: A Systematic Review

Armando Cazzetta, Francesco Zangaro · 2026

Environments

Next-generation multicolor indicators for in vivo imaging of norepinephrine

Valentin Lu Rohner, Sebastiano Curreli · 2026

Nature Methods

Lateral hypothalamus to dorsal raphe nucleus projections modulate intraspecific attack behavior in male mice

Koshiro Mitsui, Yoan Chérasse · 2026

iScience

Modeling diseases of aging in larval zebrafish, a paradoxical yet powerful strategy

Güliz Gürel Özcan, Jason Rihel · 2026

Genetics

Using Machine Learning to Support Behavioral Data Analysis in Physical Education

Christian Leukel, Timo Leuders · 2026

Journal of Teaching in Physical Edu...

Establishment of a novel deep learning-based method for gait analysis after peripheral nerve injury in pigs

Junpei Nakayama, Kentaro Okuyama · 2026

Regenerative Therapy

Stereotypic behaviors across species: From neurobiological mechanisms to farm animal welfare

Jie Gao, Haiyin Shu · 2026

Neuroscience & Biobehavioral Re...

Knee and trunk biomechanics during a single leg drop vertical jump. The effects of horizontal distance, sex, and limb dominance

Brian Lynch, Colin Bond · 2026

Gait & Posture

Dentate gyrus and CA3 activity mediates light-tone second-order conditioning expression in mice

Marc Canela, Jordi Bach Adell · 2026

iScience

Transcranial temporal interference stimulation precisely targets deep brain regions to regulate eye movements

Mo Wang, Sixian Song · 2025

Neuroscience Bulletin

Linking functional activity and connectivity of neuronal circuits via fast cross-layer all-optical physiology

Chi Liu, Yuejun Hao · 2025

Cell Reports

Soft, skin-interfaced wireless electrogoniometry systems for continuous monitoring of finger and wrist joints

Hee-Sup Shin, Jihye Kim · 2025

Nature Communications

Adaptive wheel exercise for mouse models of Parkinson’s Disease

Henry Skelton, Dayton Grogan · 2025

Journal of Neuroscience Methods

An Interdisciplinary Review of the Zygomaticus Muscles: Anatomical Variability, Imaging Modalities, and Clinical Implications

Ingrid C. Landfald, Łukasz Olewnik · 2025

Journal of Clinical Medicine

Horizontal, vertical, and torsional optokinetic responses and their adaptations in fish

Shin Tadokoro, Shuntaro Miki · 2025

Journal of Neurophysiology

Foundation model of neural activity predicts response to new stimulus types

Eric Y. Wang, Paul G. Fahey · 2025

Nature

Brain-wide analysis reveals movement encoding structured across and within brain areas

Ziyue Aiden Wang, Balint Kurgyis · 2025

Nature Neuroscience

Towards Intelligent Assessment in Personalized Physiotherapy with Computer Vision

Victor Garcia, Olga C. Santos · 2025

Sensors

Metrics

4,863

Citations

51

References

Details

Published: Aug 20, 2018
Vol/Issue: 21(9)
Pages: 1281-1289
License: View

Authors

Mackenzie Weygandt Mathis

M

Matthias Bethge

University of Tübingen, Tübingen, Germany; matthias.bethge@bethgelab.org

Cite This Article

Alexander Mathis, Pranav Mamidanna, Kevin M. Cury, et al. (2018). DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, 21(9), 1281-1289. https://doi.org/10.1038/s41593-018-0209-y

DeepLabCut: markerless pose estimation of user-defined body parts with deep learning

You May Also Like