530 West 120th St
New York, NY 10027
616 CEPSR
um2171 [at] columbia.edu
Download CV

Utkarsh Mall

Postdoctoral Researcher

Computer Science, Columbia University

I am a postdoctoral research scientist at Columbia University, where I am advised by Carl Vondrick. I did my PhD from Cornell University co-advised by Kavita Bala and Bharath Hariharan. Prior to that, I obtained my bachelor’s degree in Computer Science and Engineering from Indian Institute of Technology Bombay.

Research

My research interest lies in computer vision and its application. More specifcally, I aim to build recognition models that can learn with little to no supervision. I also use these models to make discoveries and provide scientific insights from visual data in various scientific domains. I have applied my work to a range of application domains from fashion to satellite (remote sensing) images.

Note: If you are an undergrad or masters student at Columbia or Cornell and are interested in Vision for Science Research, reach out to me for potential project opportunities.

Here is a list of my publications:

Publications

FacET: How Video Meetings Change Your Expression

Sumit Sarin, Utkarsh Mall, Purva Tendulkar, Carl Vondrick
European Conference on Computer Vision (ECCV), 2024
Paper (PDF) Webpage Code BibTeX

TL;DR: A generative domain translation method that identifies and reports spatio-temporal features distinguishing facial expressions in different communication contexts, enhancing understanding of behavioral differences.

Evolving Interpretable Visual Classifiers with Large Language Models

Mia Chiquier, Utkarsh Mall, Carl Vondrick
European Conference on Computer Vision (ECCV), 2024
Paper (PDF) Webpage Code BibTeX

TL;DR: An evolutionary search algorithm using LLMs to iteratively discover interpretable and discriminative classifiers for visual recognition.

Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment

Utkarsh Mall*, Cheng Perng Phoo*, Meilin Kelsey Liu, Carl Vondrick, Bharath Hariharan, Kavita Bala
International Conference on Learning Representations (ICLR), 2024
Paper (PDF) Webpage Code BibTeX

TL;DR: A vision-language model for satellite images, trained by using geo-located internet images as intermediary between text and satellite images.

Change-Aware Contrastive Learning for Satellite Images

Utkarsh Mall, Bharath Hariharan, Kavita Bala
Computer Vision and Pattern Recognition (CVPR), 2023
Paper (PDF) Webpage Code BibTeX

TL;DR: A self-supervised representation learning approach for satellite images that uses temporal and change information to learn better representation.

Change Event Dataset for Discovery from Spatio-temporal Remote Sensing Imagery

Utkarsh Mall, Bharath Hariharan, Kavita Bala
NeurIPS Datasets and Benchmarks Track, 2022 (Featured)
Paper (PDF) Webpage Code BibTeX

TL;DR: A method to create benchmarks for discovering meaningful multi-step change events from satellite images with no labels.

Zero-shot Learning Using Multimodal Descriptions

Utkarsh Mall, Bharath Hariharan, Kavita Bala
Computer Vision and Pattern Recognition (CVPR), 2022 (L3D-IVU Workshop)
Paper (PDF) BibTeX Supplementary

TL;DR: A practical improvement on zero-shot learning, allowing annotators to provide multiple descriptors for a concept with multiple modes of appearance.

Discovering Underground Maps from Fashion

Utkarsh Mall, Kavita Bala, Tamara Berg, Kristen Grauman
Winter Applications of Computer Vision (WACV), 2022
Paper (PDF) Webpage BibTeX News

TL;DR: A method to discover neighborhood similarity in a city using the fashion characteristics withing a city.

Field Guide-inspired Zero-Shot Learning

Utkarsh Mall, Bharath Hariharan, Kavita Bala
International Conference on Computer Vision (ICCV), 2021
Paper (PDF) Code Webpage BibTeX

TL;DR: A practical active-learning interface to efficiently specify attributes in zero-shot learning.

PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering

Jang Hyun Cho, Utkarsh Mall, Kavita Bala, Bharath Hariharan
Computer Vision and Pattern Recognition (CVPR), 2021
Paper (PDF) Code Webpage BibTeX

TL;DR: An unsupervised semantic segmentation model by clustering and encouraging equivariance to geometric transforms and invariance to photometric ones.

GeoStyle: Discovering Fashion Trends and Events

Utkarsh Mall, Kevin Matzen, Bharath Hariharan, Noah Snavely, Kavita Bala
International Conference on Computer Vision (ICCV), 2019
Paper (PDF) Code Webpage BibTeX News

TL;DR: An automated framework analyzing fashion from street photos for accurate forecasting of fashion trends/style and discovering social/cultural and sporting events.

Batch-Switching Policy Iteration

Shivaram Kalyanakrishnan, Utkarsh Mall, and Ritish Goyal
International Joint Conferences on Artificial Intelligence (IJCAI), 2016
Paper (PDF) BibTeX

TL;DR: A method offering a potentially tighter bound on iterations compared to previous variants of Policy Iteration (PI) algorithms.

Interdisciplinary Research

Computing colorism: skin tone in online retail imagery

Chelsea Butkowski, Lee Humphreys, Utkarsh Mall
Visual Communication, March 2022
Paper BibTeX News

TL;DR: Quantitative comparison of how mainstream clothing retail brands represent model skin tones across still and video media modes.

ML for Tracking Fashion Trends: Documenting the Frequency of the Baseball Cap on Social Media and the Runway

Rachel Rose Getman, Denise Nicole Green, Kavita Bala, Utkarsh Mall, Nehal Rawat, Sonia Appasamy, Bharath Hariharan.
Clothing and Textiles Research Journal, June 2020
Paper BibTeX

TL;DR: A tool to analyze large datasets of fashion imagery, revealing trends of fine-grained concepts such as baseball caps.

Sliding of Microtubules by A Team of Dynein motors

Hanumant Pratap Singh, Anjneya Takshak, Utkarsh Mall and Ambarish Kunwar
IJMPC 2016
Abstract BibTeX

TL;DR: In silico study of dynein motors and the affect of their distribution on the efficiency.

Teaching

CS 5670: Computer Vision

Teaching Assistant (Outstanding Teaching Award)
Spring 2018, Cornell University

CS 1620: Visual Imaging in the Electronic Age

Teaching Assistant
Fall 2017, Cornell University

CS475/675: Computer Graphics

Teaching Assistant
Fall 2016, IIT Bombay

BB 101: Introduction to Biology

Teaching Assistant
Spring 2017, Fall 2014, IIT Bombay

Education

Postdoctoral Research Scientist

COLUMBIA UNIVERSITY
2023-PRESENT

Advisor: Carl Vondrick

Ph.D IN COMPUTER SCIENCE

CORNELL UNIVERSITY
2017-2023

Advisor: Kavita Bala and Bharath Hariharan

Minor in Cognitive Science

B.TECH (HONORS) IN COMPUTER SCIENCE AND ENGINEERING

INDIAN INSTITUTE OF TECHNOLOGY BOMBAY
2013-2017

Advisor: Siddhartha Chaudhuri

Minor in Bio-sciences and Bio-engineering