530 West 120th St
New York, NY 10027
616 CEPSR
um2171 [at] columbia.edu

Download CV

Utkarsh Mall

Postdoctoral Researcher

Computer Science, Columbia University

I am a postdoctoral research scientist at Columbia University, where I am advised by Carl Vondrick. I did my PhD from Cornell University co-advised by Kavita Bala and Bharath Hariharan. Prior to that, I obtained my bachelor’s degree in Computer Science and Engineering from Indian Institute of Technology Bombay.


I am on the job market this year (2024-2025).

Research Statement     CV     Teaching Statement

Research

My research interest lies in computer vision and its application. More specifcally, I aim to build recognition models that can learn with little to no supervision. I also use these models to make discoveries and provide scientific insights from visual data in various scientific domains. I have applied my work to a range of application domains from fashion to satellite (remote sensing) images.

Note: If you are an undergrad or masters student at Columbia or Cornell and are interested in Vision for Science Research, reach out to me for potential project opportunities.

Here is a list of my publications:

Publications Show all publications

DiSciPLE: Learning Interpretable Programs for Scientific Visual Discovery

Utkarsh Mall, Cheng Perng Phoo, Mia Chiquier, Bharath Hariharan, Kavita Bala, Carl Vondrick
Computer Vision and Pattern Recognition (CVPR), 2025
Paper (PDF) Webpage BibTeX

TL;DR: A neurosymbolic framework to learn programs explaining visual observations in visual spatial scientific domains.

Scale-Aware Recognition in Satellite Images under Resource Constraints

Shreelekha Revankar, Cheng Perng Phoo, Utkarsh Mall, Bharath Hariharan, Kavita Bala
International Conference on Learning Representations (ICLR), 2025
Paper (PDF) Webpage BibTeX

TL;DR: A method to efficiently search concepts in satellite images under high-resolution acquisition cost contraints.

AllClear: A Comprehensive Dataset and Benchmark for Cloud Removal in Satellite Imagery

Hangyu Zhou, Chia-Hsiang Kao, Cheng Perng Phoo, Utkarsh Mall, Bharath Hariharan, Kavita Bala
NeurIPS Datasets and Benchmarks Track, 2024
Paper (PDF) Webpage BibTeX

TL;DR: A large scale dataset for training cloud removal algorithms for satellite images.

FacET: How Video Meetings Change Your Expression

Sumit Sarin, Utkarsh Mall, Purva Tendulkar, Carl Vondrick
European Conference on Computer Vision (ECCV), 2024
Paper (PDF) Webpage Code BibTeX

TL;DR: A generative domain translation method that identifies and reports spatio-temporal features distinguishing facial expressions in different communication contexts, enhancing understanding of behavioral differences.

Evolving Interpretable Visual Classifiers with Large Language Models

Mia Chiquier, Utkarsh Mall, Carl Vondrick
European Conference on Computer Vision (ECCV), 2024
Paper (PDF) Webpage Code BibTeX

TL;DR: An evolutionary search algorithm using LLMs to iteratively discover interpretable and discriminative classifiers for visual recognition.

Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment

Utkarsh Mall*, Cheng Perng Phoo*, Meilin Kelsey Liu, Carl Vondrick, Bharath Hariharan, Kavita Bala
International Conference on Learning Representations (ICLR), 2024
Paper (PDF) Webpage Code BibTeX

TL;DR: A vision-language model for satellite images, trained by using geo-located internet images as intermediary between text and satellite images.

Change-Aware Contrastive Learning for Satellite Images

Utkarsh Mall, Bharath Hariharan, Kavita Bala
Computer Vision and Pattern Recognition (CVPR), 2023
Paper (PDF) Webpage Code BibTeX

TL;DR: A self-supervised representation learning approach for satellite images that uses temporal and change information to learn better representation.

Change Event Dataset for Discovery from Spatio-temporal Remote Sensing Imagery

Utkarsh Mall, Bharath Hariharan, Kavita Bala
NeurIPS Datasets and Benchmarks Track, 2022 (Featured)
Paper (PDF) Webpage Code BibTeX

TL;DR: A method to create benchmarks for discovering meaningful multi-step change events from satellite images with no labels.

Zero-shot Learning Using Multimodal Descriptions

Utkarsh Mall, Bharath Hariharan, Kavita Bala
Computer Vision and Pattern Recognition (CVPR), 2022 (L3D-IVU Workshop)
Paper (PDF) BibTeX Supplementary

TL;DR: A practical improvement on zero-shot learning, allowing annotators to provide multiple descriptors for a concept with multiple modes of appearance.

Discovering Underground Maps from Fashion

Utkarsh Mall, Kavita Bala, Tamara Berg, Kristen Grauman
Winter Applications of Computer Vision (WACV), 2022
Paper (PDF) Webpage BibTeX News

TL;DR: A method to discover neighborhood similarity in a city using the fashion characteristics withing a city.

Field Guide-inspired Zero-Shot Learning

Utkarsh Mall, Bharath Hariharan, Kavita Bala
International Conference on Computer Vision (ICCV), 2021
Paper (PDF) Code Webpage BibTeX

TL;DR: A practical active-learning interface to efficiently specify attributes in zero-shot learning.

PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering

Jang Hyun Cho, Utkarsh Mall, Kavita Bala, Bharath Hariharan
Computer Vision and Pattern Recognition (CVPR), 2021
Paper (PDF) Code Webpage BibTeX

TL;DR: An unsupervised semantic segmentation model by clustering and encouraging equivariance to geometric transforms and invariance to photometric ones.

GeoStyle: Discovering Fashion Trends and Events

Utkarsh Mall, Kevin Matzen, Bharath Hariharan, Noah Snavely, Kavita Bala
International Conference on Computer Vision (ICCV), 2019
Paper (PDF) Code Webpage BibTeX News

TL;DR: An automated framework analyzing fashion from street photos for accurate forecasting of fashion trends/style and discovering social/cultural and sporting events.

Batch-Switching Policy Iteration

Shivaram Kalyanakrishnan, Utkarsh Mall, and Ritish Goyal
International Joint Conferences on Artificial Intelligence (IJCAI), 2016
Paper (PDF) BibTeX

TL;DR: A method offering a potentially tighter bound on iterations compared to previous variants of Policy Iteration (PI) algorithms.

Interdisciplinary Research

How physical neighborhood features drive differences in health impacts of tropical cyclones

Utkarsh Mall, Carl Vondrick, Marianthi-Anna Kioumourtzoglou, Robbie Parks
ISEE Conference Abstracts, 2024
Abstract BibTeX

TL;DR: We used computer vision and elevation data to predict Hurricane Sandy-related flood damage in NYC, finding that combining both yielded the best results, which could aid disaster planning and public health in vulnerable areas.

Computing colorism: skin tone in online retail imagery

Chelsea Butkowski, Lee Humphreys, Utkarsh Mall
Visual Communication, March 2022
Paper BibTeX News

TL;DR: Quantitative comparison of how mainstream clothing retail brands represent model skin tones across still and video media modes.

ML for Tracking Fashion Trends: Documenting the Frequency of the Baseball Cap on Social Media and the Runway

Rachel Rose Getman, Denise Nicole Green, Kavita Bala, Utkarsh Mall, Nehal Rawat, Sonia Appasamy, Bharath Hariharan.
Clothing and Textiles Research Journal, June 2020
Paper BibTeX

TL;DR: A tool to analyze large datasets of fashion imagery, revealing trends of fine-grained concepts such as baseball caps.

Sliding of Microtubules by A Team of Dynein motors

Hanumant Pratap Singh, Anjneya Takshak, Utkarsh Mall and Ambarish Kunwar
IJMPC 2016
Abstract BibTeX

TL;DR: In silico study of dynein motors and the affect of their distribution on the efficiency.

Teaching

CS 5670: Computer Vision

Teaching Assistant (Outstanding Teaching Award)
Spring 2018, Cornell University

CS 1620: Visual Imaging in the Electronic Age

Teaching Assistant
Fall 2017, Cornell University

CS475/675: Computer Graphics

Teaching Assistant
Fall 2016, IIT Bombay

BB 101: Introduction to Biology

Teaching Assistant
Spring 2017, Fall 2014, IIT Bombay

Education

Postdoctoral Research Scientist

COLUMBIA UNIVERSITY
2023-PRESENT

Advisor: Carl Vondrick

Ph.D IN COMPUTER SCIENCE

CORNELL UNIVERSITY
2017-2023

Advisor: Kavita Bala and Bharath Hariharan

Minor in Cognitive Science

B.TECH (HONORS) IN COMPUTER SCIENCE AND ENGINEERING

INDIAN INSTITUTE OF TECHNOLOGY BOMBAY
2013-2017

Advisor: Siddhartha Chaudhuri

Minor in Bio-sciences and Bio-engineering