The workshop lasts for two days: 1st day, 2nd day.

Monday March 12th 2012

Location: The Interchurch Center (corner of 120th and Claremont Ave) Room C & D on first floor. You will have to sign in at the front desk. Your names will be there.

8:30 - 9:00am Breakfast and Webex set up

9:00 - 9:30am Introductions and Overarching goals of workshop (Mona & Eneko)

9:30 - 10:30am Discussion of What is STS? [Item A]
  • 1. STS granularity (document, paragraph, sentence, phrase, word, subword)
  • 2. Similarity between gradeable and binary characterizations
  • 3. How do we characterize textual similarity? (lexical, syntactic, semantic, pragmatic levels of representation)
  • 4. What are the different dimensions of semantic similarity?
  • 5. How is semantic similarity different from semantic relatedness/inference?
  • 6. How is STS different from textual entailment?
  • 7. Desiderata for an STS system
  • 8. Current approaches to textual similarity
10:30 - 11:00 Coffee Break

11:00 - 11:30am Semeval STS task (Eneko & Dan)
  • 1. Task design
  • 2. Data sets
  • 3. Amazon Mechanical Turk Experiments
  • 4. Metrics and Initial evaluation
11:30 - 12:00pm Sample Manual Annotation by participants

12:00 - 1:00pm Discussion of annotations

1:00 - 2:00pm Lunch

2:00 - 2:30pm Evaluation of STS (Mona & Eneko) [Item B]
  • 1. intrinsic vs extrinsic considerations
  • 2. Metrics
2:30 - 4:00pm NLP applications that would benefit from STS (10 min presentations from Participants, please volunteer, below is a list of suggested NLP applications) [Item C]
  • 1. MT
  • 2. MT evaluation
  • 3. Summarization
  • 4. Machine Reading
  • 5. Watson Jeopardy
  • 6. Distillation
  • 7. Generation
  • 8. Opinion Mining
  • 9. Social Media Mining (trending)
  • 10. Inference
4:00 - 4:30pm coffee break

4:30 - 5:30 How to create an STS blackbox? (Discussion, please send us your thoughts ahead of the workshop) [Item D]
  • 1. What semantic components contribute to STS?
  • 2. Component interface issues
6:00pm Dinner together for those interested

Tuesday March 13th

Location: (Change of Location from previous day) Room 750, Interschool lab, CEPSR building on campus, entrance on 120th St, between Broadway and Amsterdam Ave.

8:30-9:00am Breakfast and webex set up

9:00 - 9:30am Review of day 1 discussions (Mona & Eneko)

9:30 - 10:30 Discussion on how to create an STS system [Item E]
  • 1. What components exist that are relevant for the task
  • 2. what desired components are missing that would complete the STS pipeline?
10:30-11:00 Coffee break

11:00 - 12:30 Infrastructure desiderata [Item F]
  • 1. Interoperability between components
  • 2. What kind of platform would be of interest: UIMA, webservices, distributed architecture?
12:30-1:30 Lunch

1:30-3:00 Discussion of Open issues [Item G]
  • 1. Evaluation Revisited
  • 2. issues of interpretability
  • 3. towards a multilingual STS
  • 4. Possibility of an empirical semantic framework
3:00- 3:30 coffee break

3:30-4:30pm Next steps and Wrapping up
  • 1. Shared Task
  • 2. Committee formation
  • 3. Funding opportunities
  • 4. Other issues
6:00pm Dinner for those around