Oct 21: Ian Tenney: Probing for Structure in Sentence Representations

Please join us for another NLP Seminar at 4:00pm in 202 South Hall on Oct 21st.

Speaker: Ian Tenney (Google)

Title: Probing for Structure in Sentence Representations

Abstract:

With the development of ELMo, BERT, and successors, pre-trained sentence encoders have become nearly ubiquitous in NLP. But what makes these models so powerful? What are they learning? A flurry of recent work – cheekily dubbed “BERTology” – seeks to analyze and explain these models, treating the encoder as an object of scientific inquiry.

In this talk, I’ll discuss a few of these analyses, focusing on our own “edge probing” work which looks at how linguistic structure is represented in deep models. Using tasks like tagging, parsing, and coreference as analysis tools, we show that language models learn strong representations of syntax but are less adept at semantic phenomena. Moreover, we find evidence of sequential reasoning, reminiscent of traditional pipelined NLP systems.

This work was jointly conducted with Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick.

Biography:

Ian Tenney is a software engineer on the Language team at Google Research in Mountain View. His research focuses on understanding and analysis of deep NLP models, particularly on how they encode linguistic structure, and how unsupervised or weakly-supervised learning can give rise to complex representations and reasoning. He was a Senior Researcher on the sentence representation team for the 2018 JSALT workshop, and from 2016 to 2018 taught in the MIDS program at UC Berkeley School of Information. He holds an M.S. in Computer Science and a B.S. in Physics from Stanford.

Oct 8: Alexander Rush: Revisiting Grammar Induction

Please join us for another NLP Seminar at 11:00 am at Soda 380 on Tuesday, Oct 8.

Speaker: Alexander Rush (Cornell)

Title: Revisiting Grammar Induction

Abstract:

Deep learning for NLP has become synonymous with global models trained with unlimited data. These models are incredible; however, they seem unlikely to tell us much about the way they (or language) work. Less heralded has been the ways in which deep methods have helped with inference in classical factored models. In this talk, I revisit the problem of grammar induction, an important benchmark task in NLP, using a variety of variational methods. Recent work shows that these methods greatly increase the performance of unsupervised learning methods. I argue that these approaches can be used in conjunction with global models to provide control in modern systems.

Biography:

Alexander Sasha Rush is an Associate Professor at Cornell Tech. His group’s research is in the intersection of natural language processing, deep learning, and structured prediction with applications in machine translation, summarization, and text generation. He also supports open-source development including the OpenNMT project. His work has received several paper and demo awards at major NLP and visualization conferences, an NSF Career Award, and faculty awards.  He is currently the general chair of ICLR.

Sept 30: Jinfeng Rao: Structure-Aware Learning and Decoding for Neural NLG in Task-Oriented Dialog

Please join us for another NLP Seminar at 4:00pm in 202 South Hall on Sept 30th.

Speaker: Jinfeng Rao (Facebook)

Title: Structure-Aware Learning and Decoding for Neural NLG in Task-Oriented Dialog

Abstract:

Generating fluent natural language responses from structured semantic representations is a critical step in task-oriented conversational systems. Previous work primarily use Seq2Seq models on flat meaning representations (MR), e.g., in the E2E NLG Challenge, which lacks of controllability of generated texts. We propose a tree-structured MR for better discourse-level structuring and sentence-level planning. We propose a constrained decoding and a tree-to-sequence approach to add structure constraints into model learning and decoding. Our experiments show both approaches lead to better semantic correctness and combining them achieves the best performance.

In addition, I will also briefly talk about my recent work on bridging the gap between relevance matching and semantic matching for short text similarity modeling.

Biography:

Jinfeng Rao is currently a research scientist at Facebook Conversational AI. Before that, he was a visiting researcher at Stanford University. He obtained his PhD with Prof. Jimmy Lin from University of Maryland College Park. Jinfeng’s research interest lies at the intersection of natural language processing, information retrieval and deep learning. At Facebook, he focuses on building and shipping world-class NLG system in Assistant. He has published more than 20 articles in the major NLP/ML conferences, including ACL, EMNLP, KDD, etc. His past research helped Comcast build their XFINITY voice search system, where his proposed multi-task system has processed billions of voice queries from 20M+ voice remotes in 2019. His work also helped Comcast win the 69th Emmy Award (2017) for the technical contributions in advancing television technologies.

(Slides, available to those with @berkeley.edu email)

NLP Seminar Schedule Fall 2019

We usually meet on Mondays from 4:00-5:00pm, in  room 202 South Hall.

This is our calendar so far, and will be updated as more people are added:

Sept 23: Cristian Danescu-Niculescu-Mizil, Cornell. (talk details)

Sept 30: Jinfeng Rao, Facebook. (talk details)

Oct 8: Alexander Rush, Cornell. (talk details. time and location is different: 11 am, Tues, Soda 230)

Oct 21: Ian Tenney, Google. (talk details)

Nov 18: Urvashi Khandelwal & John Hewitt, Stanford

Dec 6: Yoav Artzi, Cornell. (time is different: 11 am, Fri, location TBD)

If you have a @berkeley.edu email and would like to be added to the NLP seminar mailing list, email lucy3_li@nullberkeley.edu.

Sep 23: Cristian Danescu-Niculescu-Mizil: Conversational markers of (anti)social dynamics

Please join us for the first NLP Seminar of the semester on Monday, September 23,  at 4:00pm in 202 South Hall.   All are welcome!

Speaker:  Cristian Danescu-Niculescu-Mizil

Title:  Towards an artificial intuition: Conversational markers of (anti)social dynamics

Abstract:  

Can conversational dynamics—the nature of the back and forth between people—predict outcomes of social interactions?  This talk will describe efforts on developing an artificial intuition about ongoing conversations, by modeling the subtle pragmatic and rhetorical choices of the participants.

The resulting framework distills emerging conversational patterns that can point to the nature of the social relation between interlocutors, as well as to the future trajectory of this relation.  For example, I will discuss how interactional dynamics can be used to foretell whether an online conversation will stay on track or eventually derail into personal attacks, providing community moderators several hours of prior notice before an anti-social event is likely to occur.

The data and code are available through the Cornell Conversational Analysis Toolkit (ConvoKit):
convokit.cornell.edu

This talk includes joint work with Jonathan P. Chang, Lucas Dixon, Liye Fu, Yiqing Hua, Dan Jurafsky,  Lillian Lee, Jure Leskovec, Vlad Niculae, Chris Potts, Arthur Spirling, Dario Taraborelli, Nithum Thain, and Justine Zhang.

Biography:

Cristian Danescu-Niculescu-Mizil is an assistant professor in the information science department at Cornell University.  His research aims at developing computational methods that can lead to a better understanding of our conversational practices, supporting tools that can improve the way we communicate online.  He is the recipient of several awards—including an NSF CAREER Award, the WWW 2013 Best Paper Award, a CSCW 2017 Best Paper Award, and two Google Faculty Research Awards—and his work has been featured in popular media outlets such as The Wall Street Journal, NBC’s The Today Show, NPR and the New York Times.

Fall 2019: NLP Seminar Resumes!

The NLP Seminar returns Fall semester; we will be meeting on Mondays from 4-5:30pm in 202 South Hall.

Watch this space for talk announcements.

Fall 2018: No Seminar Talks

The NLP Seminar is taking a break in Fall 2018; we won’t be scheduling speakers this semester.   See you in the Spring!

Apr 30: Marilyn Walker: Modeling Narrative Structure in Informal First-Person Narratives

Please join us for the last NLP Seminar of the semester on Monday, April 30,  at 4:00pm in 202 South Hall.   All are welcome!

Speaker:  Marilyn Walker (UCSC)

Title:  Modeling Narrative Structure in Informal First-Person Narratives

Abstract: 

Many genres of natural language text are narratively structured, reflecting the human bias towards organizing our experiences as narratives. Understanding narrative structure in full requires many discourse-level NLP components, including modeling the motivations, goals and desires of the protagonists, modelling the affect states of the protagonists and their transitions across story timepoints, and modelling the causal links between story events. This talk will focus on our recent work on modeling first-person participant goals and desires and their outcomes. I describe DesireDB, a collection of personal first-person stories from the Spinn3r corpus, which are annotated for statements of desire, textual evidence for desire fulfillment, and whether the stated desire is fulfilled given the evidence in the narrative context. I will describe experiments on tracking desire fulfillment using different methods, and show that a LSTM Skip-Thought model using the context both before and after the desire statement achieves an F-Measure of 0.7 on the corpus. I will also briefly discuss our work on modelling affect states and causal links between story events on the same corpus of informal stories.

The presented work was jointly conducted with Elahe Rahimtoroghi, Jiaqi Wu, Pranav Anand, Ruimin Wang, Lena Reed and Shereen Oraby.

Biography:

Marilyn Walker, is a Professor of Computer Science at UC Santa Cruz, and a fellow of the Association for Computational Linguistics (ACL), in recognition of her for fundamental contributions to statistical methods for dialog optimization, to centering theory, and to expressive generation for dialog. Her current research includes work on computational models of dialogue interaction and conversational agents, analysis of affect, sarcasm and other social phenomena in social media dialogue, acquiring causal knowledge from text, conversational summarization, interactive story and narrative generation, and statistical methods for training the dialogue manager and the language generation engine for dialogue systems.

Before coming to Santa Cruz in 2009, Walker was a professor of computer science at the University of Sheffield. From 1996 to 2003, she was a principal member of the research staff at AT&T Bell Labs and AT&T Research, where she worked on the AT&T Communicator project, developing a new architecture for spoken dialogue systems and statistical methods for dialogue management and generation. Walker has published more than 200 papers and has more than 10 U.S. patents granted. She earned an M.S. in computer science at Stanford University, and a Ph.D. in computer science at the University of Pennsylvania.

Mon Apr 16: Amber Boydstun: How Surges in Dominant Media Narratives Move Public Opinion

Please join us for our NLP Seminar next Monday, April 16, at 4:00pm in 202 South Hall.

Speaker: Amber Boydstun (Associate Professor of Political Science, UC Davis)

Title: How Surges in Dominant Media Narratives Move Public Opinion

Abstract:

Studies examining the potential effects of media coverage on public attitudes toward policy issues (e.g., abortion, capital punishment) have identified three variables that, depending on the issue, can wield significant influence: the tone of the coverage (positive/negative/neutral), the frames used (e.g., discussing the issue from an economic vs. a moral perspective), and the overall level of media attention to the issue.  Yet, to date, no study has examined all three variables in combination.  We fill this gap by building a theoretical argument for why, despite the important variance across different issues, in general a single measure should be able to predict significant shifts in public opinion: surges in media attention to “dominant media narratives,” or stories that consistently frame the issue the same way (e.g., economic) using the same tone (e.g., anti-immigration) relative to other competing narratives.  We test this hypothesis in U.S. newspaper coverage to three very different policy issues—immigration, same-sex marriage, and gun control—from 1992 to 2012.  We use manual content analysis linked with computational modeling, tracking tone (pro/anti/neutral), emphasis frames (e.g., economic, morality), and overall levels of attention. Using time series analysis of public opinion data, we show that, for all three issues, previous surges in dominant media narratives significantly shape opinion.  In short, when media coverage converges around a unified way of describing a policy issue, the public tends to follow.  Our study adds to the fields of political communication and public opinion and marks an advance in computational text analysis methods.  (Joint work with Dallas Card and Noah Smith)

Mar 12: Rob Voigt: Implicit Attitudes, NLP, and the “Real World”

Please join us for the NLP Seminar on Monday, March 12  at 4:00pm in 202 South Hall.   All are welcome!

Speaker: Rob Voigt (Stanford)

Title: Implicit Attitudes, NLP, and the “Real World”

Abstract:

While some forms of bias in language are explicit, such as overt references to stereotypes, much linguistic bias is far more subtle, where implicit attitudes towards social groups pervasively affect how we talk to and about members of those groups. As a result, such variation is often identifiable only in aggregate accounting for the contexts of language use. In this talk, I will present two projects from my dissertation which aim to complement NLP techniques with on-the-ground facts about the world to understand the joint linguistic and extralinguistic factors that contribute to social biases.

First, I’ll present the results of a study using body camera footage from the Oakland Police Department as interactional data for analyzing racial disparities in officer language. Applying a computational linguistic model of respect across a month of everyday traffic stops, we found that officers were less respectful to black than to white community members, even after controlling for social factors like officer race and contextual factors like the location of the stop and the severity of the offense. Second, I’ll present ongoing work exploring representations of immigrants in the US news media over historical time. Our results thus far suggest cyclic patterns of linguistic “othering” that recur with each immigrant group as they arrive and are directly connected to economic and demographic circumstances of those groups.

( Slides )

« Older posts