Title: “Benchmarking for Responsible AI”
Abstract: Acclaimed physicist Lord Kelvin once said that what cannot be measured cannot be improved. Though Generative AI — its evolution as a technology, what innovations it may enable, and how to govern it — has dominated the social zeitgeist for a number of years, our ability to actually measure its performance is woefully lacking. Many of the most prominent public benchmarks for these models fall into one of two categories: those (1) crafted by model-creators themselves or computer science experts; or (2) those that are broadly crowdsourced. Even where there are relevant and domain-expert-informed benchmarks, the overwhelming majority of them test final model outputs—making it immensely challenging to study how individual components of Generative AI models actually perform or what biases they may introduce. In this talk I will explore how measurements and their shortcomings limit the domains and mechanisms through which we can deploy AI responsibly. Focusing on the emerging legal landscape at the state level, I will break down which regulatory initiatives are gaining traction and outline where we might need more building blocks to get off the ground.
In doing so I will discuss the challenge of designing policy that can be operationalized and how cross-disciplinary collaboration can help fill the gap between publishing frameworks and promoting a more responsible AI landscape.
Bio: Tom Zick is an attorney and affiliate at the Berkman Klein Center for Internet and Society, where her research focuses on the intersection of AI ethics and law. She specializes in developing tools to support safe and equitable AI governance. Tom has collaborated with leading frontier model labs to build research partnerships with the Harvard academic community, with a particular focus on red teaming and model alignment. She has advised early-stage tech founders on responsible technology development and has worked with the City of Boston on deploying AI and
operationalizing its data governance framework. Tom holds a JD from Harvard Law School and a PhD in Astrophysics from UC Berkeley.
Zoom: https://wse.zoom.us/j/96307689353?pwd=5AfXHeSyETabThsauR7t7vzlXnIKwt.1
Meeting ID: 963 0768 9353
Passcode: 595531