Ning Zhang


Email: ningzhang@meta.com

Google Scholar Linkedin

My team is hiring full time research scientists, engineers and research interns, email me if you are interested.

About Me

I am a researcher, entrepreneur and practitioner working on artificial intelligence, deep learning and computer vision. I am currently a senior research scientist manager at Facebook(Meta) AI, working on computer vision, natural language processing and multimodal models for commerce and monetization applications.

Prior to this, I was head of computer vision at Dawnlight, working on next-generation privacy preserving indoor patient monitoring system for social good. My team built first prototype of activity recognition model running real-time on edge devices.

Before that, I led the computer vision research group at Snapchat. My team worked on projects including object recognition, object detection, efficient deep learning inference, semantic segmentation, pose estimation and tracking, text recognition, Generative Adversarial networks. I had a great time applying deep learning to enhance products that brought joy to hundreds of millions of users.

I earned my Ph.D. in Computer Science at UC Berkeley in 2015, advised by Professor Trevor Darrell. My Ph.D. thesis is about fine-grained image categorization using deep learning. I have also spent two summers interning at Facebook AI Research (FAIR). I graduated from Tsinghua University with a B.S. in Computer Science in 2010, working with Professor Jie Tang.

Services

Selected Publications

Tell me what happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generation

Tsu-Jui Fu, Licheng Yu, Ning Zhang, Cheng-Yang Fu, Jong-Chyi Su, William Yang Wang, Sean Bell.
Computer Vision and Pattern Recognition (CVPR), 2023





FaD-VLP: Fashion Vision-and-Language Pre-training towards Unified Retrieval and Captioning

Suvir Mirchandani, Licheng Yu, Mengjiao Wang, Animesh Sinha, Wenwen Jiang, Tao Xiang, Ning Zhang.
Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022





CommerceMM: Large-Scale Commerce Multimodal Representation Learning with Omni Retrieval

Licheng Yu, Jun Chen, Animesh Sinha, Mengjiao Wang, Hugo Chen, Tamara Berg, Ning Zhang.
SIGKDD Conference on Knowledge Discovery and Data Mining, 2022





Unsupervised Vision-and-Language Pre-training via Retrieval-based Multi-Granular Alignment

Mingyang Zhou, Licheng Yu, Amanpreet Singh, Mengjiao Wang, Zhou Yu, Ning Zhang.
Computer Vision and Pattern Recognition (CVPR) (Oral), 2022



Connecting What to Say With Where to Look by Modeling Human Attention Traces

Zihang Meng, Licheng Yu, Ning Zhang, Tamara L. Berg, Babak Damavandi, Vikas Singh, Amy Bearman.
Computer Vision and Pattern Recognition (CVPR), 2021
Code



Context-Aware Zero-Shot Recognition

Ruotian Luo, Ning Zhang, Bohyung Han, Linjie Yang.
Thirty-Fourth AAAI Conference on Artifial Intelligence (AAAI), 2020
Code



Laplace Landmark Localization

Joseph P. Robinson, Yuncheng Li, Ning Zhang, Yun Fu, Sergey Tulyakov.
The IEEE International Conference on Computer Vision (ICCV), 2019



Dynamic Kernel Distillation for Efficient Pose Estimation in Videos

Xuecheng Nie, Yuncheng Li, Linjie Yang, Ning Zhang, Jiashi Feng.
The IEEE International Conference on Computer Vision (ICCV), 2019



Feedback Adversarial Learning: Spatial Feedback for Improving Generative Adversarial Networks

Minyoung Huh, Shao-hua Sun, Ning Zhang.
Computer Vision and Pattern Recognition (CVPR), 2019



Multi-view to Novel view: Synthesizing novel views from Self-Learned Confidence

Shao-Hua Sun, Minyoung Huh, Yuan-Hong Liao, Ning Zhang, Joseph J. Lim.
European Conference on Computer Vision (ECCV), 2018
Project page Code



Visual Attention Model for Name Tagging in Multimodal Social Media

Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, Heng Ji.
56th Annual Meeting of the Association for Computational Linguistics (ACL), 2018



AutoScaler: Scale-Attention Networks for Visual Correspondence

Shenlong Wang, Linjie Luo, Ning Zhang, Li-Jia Li.
British Machine Vision Conference (BMVC), 2017(Oral)

Deep Reinforcement Learning-Based Image Captioning With Embedding Reward

Zhou Ren, Xiaoyu Wang, Ning Zhang, Xutao Lv, Li-Jia Li.
Computer Vision and Pattern Recognition (CVPR), 2017(Oral)

Fine-grained pose prediction, normalization, and recognition

Ning Zhang, Evan Shelhamer, Yang Gao, Trevor Darrell.
International Conference on Learning Representations (ICLR) workshop, 2016

Compact Bilinear Pooling

Yang Gao, Oscar Beijbom, Ning Zhang, Trevor Darrell.
Computer Vision and Pattern Recognition (CVPR), 2016

Beyond Frontal Faces: Improving Person Recognition Using Multiple Cues

Ning Zhang, Manohar Paluri, Yaniv Tagiman, Rob Fergus, Lubomir Bourdev.
Computer Vision and Pattern Recognition (CVPR), 2015
PDF arXiv Project page



Do Convnets Learn Correspondence?

Jonathan Long, Ning Zhang, Trevor Darrell.
Neural Information Processing Systems Foundation (NIPS), 2014
arXiv



Part-based R-CNNs for Fine-grained Category Detection.

Ning Zhang, Jeff Donahue, Ross Girshick, Trevor Darrell.
European Conference on Computer Vision (ECCV), 2014 (Oral)
PDF Slides Poster Code



PANDA: Pose Aligned Networks for Deep Attribute Modeling.

Ning Zhang, Manohar Paluri, Marc'Aurelio Ranzato, Trevor Darrell, Lubomir Bourdev.
Computer Vision and Pattern Recognition (CVPR), 2014 (Oral)
PDF Code Slides Arxiv




Open-vocabulary Object Retrieval

Sergio Guadarrama, Erik Rodner, Kate Saenko, Ning Zhang, Ryan Farrell, Jeff Donahue, Trevor Darrell.
Robotics Science and Systems (RSS), 2014
PDF




DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition

Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, Trevor Darrell.
International Conference on Machine Learning (ICML), 2014
PDF Arxiv




Deformable Part Descriptors for Fine-grained Recognition and Attribute Prediction

Ning Zhang, Ryan Farrell, Forrest Iandola, Trevor Darrell.
International Conference on Computer Vision (ICCV), 2013
PDF Matlab Code Poster



Pose Pooling Kernels for Sub-category Recognition

Ning Zhang, Ryan Farrell, Trevor Darrell.
Computer Vision and Pattern Recognition (CVPR), 2012
PDF



Birdlets: Subordinate Categorization Using Volumetric Primitives and Pose-Normalized Appearance.

Ryan Farrell, Om Oza, Ning Zhang, Vlad I. Morariu, Trevor Darrell, Larry S. Davis.
International Conference on Computer Vision (ICCV), 2011 (Oral)
PDF




Teaching