Research Experience

• 

  Adobe

  San Jose, California, USA (April 2024 - present)

  Research Scientist

  • Segmentation Foundation Model: Developed cutting-edge segmentation technologies to enhance Adobe products. Contributions include core features deployed in Adobe Photoshop’s newest select tools[Media Review], and segmentation enhancements in Adobe Lightroom, Adobe Express
  • Grounding Multimodal Large Language Model: Enhance the grounding ability of multimodal large language models by integrating advanced reasoning capabilities and fine-grained understanding.

• 

  Adobe

  San Jose, California, USA (May 2023 - Nov 2023)

  Research Intern | Advisors: Dr. Brian Price, Dr. Jason Kuen, Dr. Yifei Fan, Dr. Zijun Wei, Luis Figueroa, Markus Woodson

  • Segmentation Foundation Model: Engaged in a large-scale segmentation initiative that utilizes an expansive dataset of over 100 million diverse images. The work involved a model-in-the-loop data engine and close collaboration with teams from Adobe Research, Photoshop, and Lightroom
  • SAM Model Reproducing and Beyond: Successfully reproduced the large-scale multi-node training and interactive segmentation capabilities of the SAM (Segment Anything Model), and further enhanced its features for accuracy and efficiency, potentially influencing the roadmap for future products.
  • Human-Centric Feedback Loop: Initiated a human feedback mechanism by designing meticulous evaluation experiments and coordinating with the labeling team. This ensures that the models are attuned to nuanced user requirements and can be iteratively improved based on real-world feedback
  • Benchmarking and Quality Assurance: Elevated model performance through algorithmic advancements, pushing the boundaries of model performance, achieving state-of-the-art segmentation accuracy across 20+ diverse test sets. This underscores the model’s potential for real-world applications.

• 

  Google

  Mountain View, California, USA (May 2022 - Sep 2022)

  Research Intern | Advisors: Dr. Xuhui Jia, Dr. Yu-Chuan Su, Dr. Ruijin Cang

  • One-shot Face Video Synthesis: Developed deep learning algorithms optimized for real-time face video synthesis, targeting efficient deployment on mobile devices such as smartphones.
  • Improved Quality: Innovated a method that enhances both the geometric correspondence (7% in AKD) and overall quality (%15 in AED) of the face videos produced.
  • Emotion Manipulation: Incorporated a convenient interface to manipulate the expression and emotion of the generated video in a plug-in manner.

• 

  Center for Data Science, New York University

  New York, USA (Sept 2019 - March 2024)

  Research Assistant | Advisors: Prof. Carlos Fernandez-Granda, Prof. Krzysztof J. Geras

  • Weakly Supervised High-Resolution Segmentation: Pioneered vision-based methods for weakly supervised object localization in the domain of breast cancer diagnosis.
  • Noise-Resilient Deep Segmentation: Engineered robust algorithms tailored for segmentation tasks, showing resilience against annotation noise. This ensures the model’s reliability, especially in real-world scenarios where annotations noise is prevalent.
  • Video Understanding: Created deep learning models aimed at video action classification and segmentation, applicable in predicting motion primitives.
  • Unsupervised Model Finetuning via Weak Supervision: Devised an innovative multiple-instance learning framework that iteratively refines instance-level features and outperforms existing methods across three real-world datasets.
  • Inverse Problem for Time Series Data: Engineered deep learning solutions for MRI parameter estimation.

• 

  Computer Vision Laboratory, ETH Zurich

  Zurich, Switzerland (Oct 2018 - Aug 2019)

  Research Assistant | Advisors: Prof. Luc Van Gool, Prof. Radu Timofte, Prof. Shuhang Gu

  • Domain Adaptation: Improved cross-domain semantic segmentation methods for urban scene images through GAN models, harnessing weak paired information.
  • Video and Image Synthesis: Achieved state-of-the-art performance in unsupervised multimodal video translation through bidirectional recurrent neural networks.

Publications

See also Google Scholar.

Preprint

• 

  Controllable One-Shot Face Video Synthesis With Semantic Aware Prior

  [Paper link]

  Kangning Liu, Yu-Chuan Su, Wei(Alex) Hong, Ruijin Cang, Xuhui Jia.

  We propose a method that leverages rich face prior information to generate face videos with improved semantic consistency and expression preservation. Our model improves the baseline by 7% in average keypoint distance and outperforms the baseline by 15% in average emotion embedding distance, while also providing a convenient interface for highly controllable generation in terms of pose and expression.

Published

• 

  Refer to Any Segmentation Mask Group With Vision-Language Prompts

  [Paper link]

  Shengcao Cao, Zijun Wei, Jason Kuen, Kangning Liu, Lingzhi Zhang, Jiuxiang Gu, HyunJoon Jung, Liang-Yan Gui, Yu-Xiong Wang

  [ICCV 2025]

  We introduce a novel task of omnimodal referring expression segmentation (ORES). In this task, a model produces a group of masks based on arbitrary prompts specified by text only or text plus reference visual entities. We propose a novel framework to "Refer to Any Segmentation Mask Group" (RAS), which augments segmentation models with complex multimodal interactions and comprehension via a mask-centric large multimodal model.

• 

  ConSAMmé: Achieving Consistent Segmentations with SAM

  [Paper link]

  Josh Myers-Dean, Kangning Liu, Brian Price, Yifei Fan, Danna Gurari.

  [CVPR 2025 NTIRE Workshop.]

  Multi-output interactive segmentation methods generate multiple binary masks when given user guidance, such as clicks. However, it is unpredictable whether the order of the masks will match or whether those masks will be the same when given slightly different user guidance. To address these issues, we propose conSAMmé, a contrastive learning framework that conditions on explicit hierarchical semantics and leverages weakly supervised part segmentation data and a novel episodic click sampling strategy. Evaluation of conSAMmé’s performance, click robustness, and mask ordering show substantial improvements to baselines with less than 1% extra training data compared to the amount of data used for the baseline.

• 

  Uncertainty-aware Fine-tuning of Segmentation Foundation Models

  [Paper link] [Github Link] [Project Website]

  Kangning Liu, Brian Price, Jason Kuen, Yifei Fan, Zijun Wei, Luis Figueroa, Krzysztof J. Geras, Carlos Fernandez-Granda.

  [NeurIPS 2024]

  The Segment Anything Model (SAM) is a large-scale foundation model that has revolutionized segmentation methodology. Despite its impressive generalization ability, the segmentation accuracy of SAM on images with intricate structures is unsatisfactory in many cases. We introduce the Segmentation with Uncertainty Model (SUM), which enhances the accuracy of segmentation foundation models by incorporating an uncertainty-aware training loss and prompt sampling based on the estimated uncertainty of pseudo-labels. We evaluated the proposed SUM on a diverse test set consisting of 22 public benchmarks, where it achieves state-of-the-art results. Notably, our method consistently surpasses SAM by 3-6 points in mean IoU and 4-7 in mean boundary IoU across point-prompt interactive segmentation rounds.

• 

  Multiple Instance Learning via Iterative Self-Paced Supervised Contrastive Learning

  [Paper link] [Github Link]

  Kangning Liu*,Weicheng Zhu* (*Equal contribution), Yiqiu Shen, Sheng Liu, Narges Razavian, Krzysztof J. Geras, Carlos Fernandez-Granda.

  [CVPR 2023]

  In this paper, we introduce Iterative Self-Paced Supervised Contrastive Learning (Its2CLR), a novel method for learning high-quality instance-level representations in Multiple Instance Learning (MIL). Key features of our method: 1) self-paced learning to handle label noise and uncertainty; 2) supervised contrastive learning to learn discriminative instance-level embeddings; 3) iterative refinement of instance labels for robust and accurate classification.

• 

  Adaptive early-learning correction for segmentation from noisy annotations

  [Paper Link][Github Link]

  Sheng Liu*, Kangning Liu* (*Equal contribution, order decided by coin flip.), Weicheng Zhu, Yiqiu Shen, Carlos Fernandez-Granda.

  [CVPR 2022. Oral 4.2% acceptance rate]

  Deep learning in the presence of noisy annotations has been studied extensively in classification, but much less in segmentation tasks. In this project, we study the learning dynamics of deep segmentation networks trained on inaccurately-annotated data and propose a new method for semantic segmentation ADaptive Early-Learning corrEction (ADELE).

• 

  StrokeRehab: A Benchmark Dataset for Sub-second Action Identification

  [Paper Link]

  Aakash Kaku*, Kangning Liu* (*Equal contribution), Avinash Parnandi*, Haresh Rengaraj Rajamohan, Anita Venkatesan, Audre Wirtanen, Natasha Pandit, Kannan Venkataramanan, Heidi Schambra, Carlos Fernandez-Granda.

  [NeurIPS 2022]

  We introduce a new benchmark dataset for the identification of subtle and short-duration actions. We also propose a novel seq2seq approach, which outperforms the existing methods on the new as well as standard benchmark datasets.

• 

  Are All Losses Created Equal: A Neural Collapse Perspective

  [Paper Link]

  Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, Zhihui Zhu.

  [NeurIPS 2022]

  A broad family of loss functions leads to neural collapse solutions hence are equivalent on training set; moreover, they exhibit largely identical performance on test data as well.

• 

  Cramér-Rao bound-informed training of neural networks for quantitative MRI

  [Paper Link]

  Xiaoxia Zhang*, Quentin Duchemin*, Kangning Liu* (*Equal contribution), Sebastian Flassbeck, Cem Gultekin, Carlos Fernandez-Granda, Jakob Asslander.

  [Magnetic Resonance in Medicine 2021]

  To address the parameter estimation problem in heterogeneous parameter spaces, we propose a theoretically well-founded loss function based on the Cramér-Rao bound (CRB), which provides a theoretical lower bound for the variance of an unbiased estimator.

• 

  Weakly-supervised High-resolution Segmentation of Mammography Images for Breast Cancer Diagnosis

  [Paper Link][Github Link]

  Kangning Liu, Yiqiu Shen, Nan Wu, Jakub Piotr Chdowski, Carlos Fernandez-Granda, and Krzysztof J. Geras

  [Medical Imaging with Deep Learning 2021]

  In this study, we unveil a new neural network model for weakly-supervised, high-resolution image segmentation. Focused on breast cancer diagnosis via mammography, our method first identifies regions of interest and then segments them in detail. Validated on a substantial, clinically-relevant dataset, our approach significantly outperforms existing methods, improving lesion localization performance by up to 39.6% and 20.0% for benign and malignant cases, respectively.

• 

  Unsupervised Multimodal Video-to-Video Translation via Self-Supervised Learning

  [Paper Link] [Project Link]

  Kangning Liu*, Shuhang Gu*(*Equal contribution), Andrs Romero, and Radu Timofte

  [WACV 2021]

  In this project, we introduce an unsupervised video-to-video translation model that decouples style and content. Leveraging specialized encoder-decoder and bidirectional RNNs, our model excels in propagating inter-frame details. This architecture enables style-consistent translations and offers a user-friendly interface for cross-modality conversion. We also implement a self-supervised training approach using a novel video interpolation loss that captures sequence-based temporal information.

• 

  An interpretable classifier for high-resolution breast cancer screening images

  [Paper Link]

  Shen, Yiqiu, Nan Wu, Jason Phang, Jungkyu Park, Kangning Liu, Sudarshini Tyagi, Laura Heacock et al.

  [Medical image analysis 2021]

  Medical images differ from natural images in significantly higher resolutions and smaller regions of interest. Because of these differences, neural network architectures that work well for natural images might not be applicable to medical image analysis. In this work, we propose a novel neural network model to address these unique properties of medical images.

Teaching

• Probability and Statistics for Data Science, NYU, Center for Data Science

  Teaching Assistant (Sept - Dec 2021)

• Advanced Machine Learning, ETH Zurich, Department of Computer Science

  Teaching Assistant (Sept - Dec 2018)

Service

• Award

  Top Reviewer for NeurIPS 2024

• Conference Reviewer for

  CVPR, ICCV, ECCV, NeurIPS, ICLR, AISTATS, AAAI, WACV and MIDL

• Journal Reviewer for

  TPAMI, TNNLS, CVIU

California Style

NYC Cityscapes

Light and Night

SteelStacks

Vermont Fall