RESEARCH

Summary | Projects | Recent Invitated Talks

Write A Classifier: Learning Fine-Grained Visual Classifiers from Text and Images

NSF RI Medium Collaborative Research award in collaboration with Ahmed Elgammal (Rutgers University).

This project develops the learning strategy using textual narrative and images makes the learning effective without a huge number of images that a typical visual learning algorithm would need to learn the class boundaries. The research team investigates computational models for joint learning of visual concepts from images and textual descriptions of fine-grained categories, for example, discriminating between bird species. The research activities have broader impact in three fields: computer vision, natural language processing, and machine learning. There is a huge need to develop algorithms to automatically understand the content of images and videos, with numerous potential applications in web searches, image and video archival and retrieval, surveillance applications, robot navigation and others. There are various applications for developing an intelligent system that can use narrative to define and recognize categories.

This project addresses two research questions: First, given a visual corpus and a textual corpus about a specific domain, how to jointly and effectively learn visual concepts? Second, given these two modalities how to facilitate learning novel visual concepts using only pure textual descriptions of novel categories in the domain? The research team approaches the problem on three integrated fronts: Learning, Natural Language Processing (NLP), and Computer Vision. On the learning front, the project investigates and develops algorithms suitable for learning and predicting visual classifiers with side textual information. On the NLP front, the project aims to develop novel methods for learning global and local discriminative category-level attributes and their values from text, with feedback from human computation and visual signal. The project investigates supervised and unsupervised methods for detecting visual text, and learning methods for deep language understanding to build such rich domain models from the noisy visual text. On the Vision front, the project addresses the tasks of detection and classification with side textual information. The project investigates models for the shape and appearance of a general category that can specialize to different subordinates, in a way that allows interpreting information from text within a proper geometric context, and handle variability in viewpoints and articulation.

Reducing Racial and Gender Achievement Gaps in STEM: Use of Natural Language Processing to Understand Why Affirmation Interventions Improve Performance

NSF award in collaboration with Valerie Purdie Vaughn (Columbia University), Co-PIs Geoffrey Cohen (Stanford University), Jonathan Cook (Penn State University).

Addressing issues related to reducing the size of the achievement gaps in STEM disciplines among subpopulations of students is important to helping the Nation meet its 21st Century science and technology needs. Research shows that causes of achievement gaps in STEM arise from reciprocal interactions between societal, social, and environmental factors that might suppress students' true academic potential in challenging academic STEM domains. This project focuses on environmental factors (identified as social identity threats) that devalue, marginalize, or discriminate against students based on a social identity like race, gender, disability status, or socioeconomic status; such factors can eventually lead students to withdraw and disengage in STEM learning and careers. The objectives of this research are to: (1) synthesize and systematically analyze data from interventions (affirmation writing essays) shown to help reduce the impact of social identity threats on student participation in STEM; and (2) apply results of the synthesis and analyses to enhance existing interventions (e.g., maximize impact on subpopulations of students whose achiement in STEM fields is below their potential).

The research project will proceed in two phases. First, the investigators will create an encrypted online repository of data from more than 2,500 affirmation writing essays, previously collected through randomized double-blind experiments involving approximately 1,400 students who vary by race, ethnicity, age, gender, and social class. The researchers will link this online repository of information to academic and psychological outcomes for middle school and college students. Using natural language processing (NLP), topic modeling, and other methods the investigators will identify sematic content and essay structure processes that mediate affirmation effects and highlight meaning of the effectiveness of the essay writing interventions.

Results of these analyses will be used to develop and test a more robust intervention for reducing social identity threats involving African Americans, White, and female students. One hundred eighty (180) students (90 females and 90 males) will participate in two separate laboratory studies. One, conducted at Columbia University, will focus on race as a social factor; the second, conducted at Penn State University, will focus on gender. The ultimate goal of this work is to uncover and address psychological factors that might otherwise hinder students' participation in STEM careers.

DRATS: Detecting Relations and Anomalies in Text and Speech

DARPA Deep Exploration and Filtering of Text (DEFT) Program in collaboration with Owen Rambow, Kathleen McKeown, Julia Hirschberg (Columbia University), Mona Diab (GWU) and Mari Ostendorg (University of Washington)

The project aims at understanding the individual and interactional conditions under which an author produces discourse, and to use this deep understanding to extrapolate to her state of mind at the time of discourse production. This allows us to obtain an understanding of how to interpret the linguistic signal and to identify information (such as relations) that is only implicit. This deeper interpretation of the linguistic signal, in turn, rovides us with a much broader basis for detecting anomalies in written and spoken communication. The project is part of the DARPA Deep Exploration and Filtering of Text (DEFT) Program. The overall effort, led by Columbia University, has five algorithms to address two key challenges. Three algorithms contribute to the Relational Analysis component in the DEFT System vision, all of which fall under the general problem of cognition and interaction analysis, which aims to determine the discourse participants' beliefs and opinions. Within this task, the focus at Rutgers is on the belief recognition algorithm. Two additional algorithms contribute to the Anomaly Analysis component in the DEFT System vision, specifically detection of anomalies, novelty and contradiction. The Rutgers focus in this task is on contradiction detection. The specific contributions of the proposed Rutgers effort lead by PI Smaranda Muresan include:

• Belief Recognition Algorithm: Sarcasm Detection When someone is sarcastic, she says the opposite of what she believes. Failing to recognize an utterance as being sarcastic will lead to erroneous interpretation of what the author actually believes. The main focus will be on advancing the use of contextual features for sarcasm detection. In addition the work will explore the effect of lexical and speech-related features, the work being done on conversational English and foreign text and speech.
• Contradiction Detection This line of work will focus on the detection of new information that contradicts earlier beliefs.This work is an integral part of the team's efort to detect novel information in online conversations. Smaranda Muresan is the Technical Lead for the Novelty and Contradiction Detection task. Work will be carried out on both English and foreign language text and speech.

Teaching Computers to Follow Verbal Instructions

NSF RI Medium Collaborative Research project (IIS-1065195) in collaboration with Michael Littman (Rutgers) and Marie desJardin UMBC)

The goal of this research is to develop techniques that will permit a computer or robot to learn from examples to carry out multipart tasks specified in natural language on behalf of a user. It will study each of these components in isolation, but a significant focus will be on integrating them into a coherent system. The project will also leverage this technology to provide an entry point to educate non- or pre-computer science students about the capabilities and utility of computers as tools.

Our approach uses three main subcomponents, each of which requires innovative research to solve its portion of the overall problem. In addition, the integrated architecture is a novel contribution of this work. The three components are (1) recognizing intention from observed behavior using extensions of inverse reinforcement learning, (2) translating instructions to task specifications using novel techniques in the area of natural language processing, and (3) creating generalized task specifications to match user intentions using probabilistic methods for creating and managing abstractions.

The goal of the work is develop technology for an improved ability for human users to interact with intelligent agents, the incorporation of novel AI research insights and activities into education and outreach activities, and the development of resources for the AI educator community. In addition to permitting intelligent agents to be developed and trained in the future for a broad range of complex application domains, the interactive agents that we will develop will be used for outreach and student learning.

Learnable Constraint-based Grammars for Deep Language Understanding

The core of my research focuses on developing computational models for language understanding and learning. I introduce a new grammar formalism, Lexicalized Well-Founded Grammar, which captures syntax and semantics, has ontology constraints at the grammar rule level, and is learnable from a small set of annotated examples (Muresan 2006; Muresan and Rambow, 2007; Muresan 2008; Muresan 2010). The semantic representation is an ontology-based representation, which is expressive enough to capture various phenomena of natural language, yet restrictive enough to facilitate learning. I introduce a new grammar learning model, Grammar Approximation by Representative Sublanguage, based on the concept of representative examples, defining the importance to the model linguistically, and not simply by frequency, as in most previous work. The search space for grammar induction is a complete grammar lattice, which guarantees the uniqueness of the solution. More generally, my research objective is to explore what computational formalisms and machine learning techniques are adequate to model both context and cumulative learning for language understanding in a unified way. The goal is to bring us closer to achieve scalable, robust and human-inspired computational models for deep language understanding and language learning.

Indentifying the Language of Opposition in Online Interractions

Project in collaboration with Mark Aakus and Nina Wacholder.

This project contributes to the long-standing interest in developing socially intelligent systems to augment human reasoning and interaction in large-scale online communities by focusing on a specific, yet ubiquitous, phenomenon of human behavior: opposition. This research uses theories from communication sciences and natural language processing techniques for identifying and characterizing the flow of opposition in online interactions.This project will offer a key reframing of prior natural language processing research related to opposition by introducing a robust and scalable deep-linguistic approach for modeling how texts relate to each other through opposition. The computational models of opposition will be tested in two applications for social collaborative environments: 1) The Controversy Alert System will detect and highlight controversial utterances and blunders in Wikipedia articles. 2) The Opposition Monitor will help users track the flow of opposition in an extended online interaction, such as who opposes whom and on what issue.

Past Project

Exploring Richer Representations in Statistical Machine Translation

Project in collaboration with Philip Resnik at University of Maryland, College Park, funded by NSF SGER:"Exploiting Alternative Packagings of Source Meaning in Statistical Machine Translation".

Current approaches in statistical machine translation (MT) miss a key fact: the source language sentence is not the only way the author's meaning could have been expressed. The idea that the source sentence is just one of various ``packagings'' of underlying meaning was, of course, one familiar motivation for interlingual approaches to translation; however, interlingual semantic representations have generally been abandoned as notoriously difficult to define, and equally difficult to obtain accurately with broad coverage once defined. In this project, we are revisiting the idea of "packagings" of meaning, but exploring it in practical ways consistent with current practice in statistical MT. Unlike semantic transfer or interlingual approaches, we encode alternatives as source paraphrase lattices, a representation that allows us to exploit generalizations about the source language while still maintaining the surface-to-surface orientation that characterizes the statistical state of the art. Our exploratory work focuses on capturing syntactic and semantic variation using Lexicalized Well Founded Grammars (LWFG), a recent formalism that balances expressiveness with practical and provable learnability results (Muresan, 2006; Muresan and Rambow, 2007; Muresan 2008). We are quantifying and characterizing the information available in source paraphrase lattices, assessing the value of shallow paraphrasing, and exploring the relative promise of deeper techniques for source paraphase generation using LWFG and other constraint-based grammatical frameworks. The ability to capture generalizations via source paraphrase may open new possibilities in the translation of minority and endangered languages, which lack training corpora on the scale necessary to support standard statistical MT techniques.

Learning Consumer Health Terminologies from Text

Project in collaboration with Judith Klavans, University of Maryland Institute for Advanced Computer Studies.

Understanding and sharing terminology, both by systems and humans, are important aspects of communication. In this project, we propose a two-step approach towards building terminological knowedge bases for consumer-health systems from text (Muresan and Klavans(in prep)). First, we automatically extract defnitions from consumer-oriented on-line articles and web documents, which reflects language in use, rather than rely solely on dictionaries. This step is based on our previous work on DEFINDER ((Klavans and Muresan, 2000;2001; Muresan and Klavans 2002), which we plan to extend by using machine learning aproaches. Second, we learn a grammar that directly maps natural language to graph-based meaning representations, rather than use hand-written patterns, grammars, or semantic transfer rules used on top of syntactic parses. We use our LWFG grammar formalism which capture syntax and semantics, and models semantic interpretation as grammar constraints.

Smaranda Muresan
Research Scientist
Center for Computational Learning Systems
Columbia University
email: smara --AT-- ccls.columbia.edu
Curriculum Vitae

Mailing Address:
475 Riverside Drive MC 7717
(850 Interchurch Center)
New York, NY 10115