Title: “Benchmarking for Responsible AI”
Abstract: Acclaimed physicist Lord Kelvin once said that what cannot be measured cannot be improved. Though Generative AI — its evolution as a technology, what innovations it may enable, and how to govern it — has dominated the social zeitgeist for a number of years, our ability to actually measure its performance is woefully lacking. Many of the most prominent public benchmarks for these models fall into one of two categories: those (1) crafted by model-creators themselves or computer science experts; or (2) those that are broadly crowdsourced. Even where there are relevant and domain-expert-informed benchmarks, the overwhelming majority of them test final model outputs—making it immensely challenging to study how individual components of Generative AI models actually perform or what biases they may introduce. In this talk I will explore how measurements and their shortcomings limit the domains and mechanisms through which we can deploy AI responsibly. Focusing on the emerging legal landscape at the state level, I will break down which regulatory initiatives are gaining traction and outline where we might need more building blocks to get off the ground.
In doing so I will discuss the challenge of designing policy that can be operationalized and how cross-disciplinary collaboration can help fill the gap between publishing frameworks and promoting a more responsible AI landscape.
Bio: Tom Zick is an attorney and affiliate at the Berkman Klein Center for Internet and Society, where her research focuses on the intersection of AI ethics and law. She specializes in developing tools to support safe and equitable AI governance. Tom has collaborated with leading frontier model labs to build research partnerships with the Harvard academic community, with a particular focus on red teaming and model alignment. She has advised early-stage tech founders on responsible technology development and has worked with the City of Boston on deploying AI and
operationalizing its data governance framework. Tom holds a JD from Harvard Law School and a PhD in Astrophysics from UC Berkeley.
Zoom: https://wse.zoom.us/j/96307689353?pwd=5AfXHeSyETabThsauR7t7vzlXnIKwt.1
Meeting ID: 963 0768 9353
Passcode: 595531
Title: Building Haystacks to Find Needles
Abstract: The internet is a big place, comprising billions of users and tens of billions of network devices. Discovering and remediating vulnerabilities in these devices is an imperative for a more secure internet. Unfortunately, vulnerabilities that affect millions of hosts represent only a small fraction of the overall internet. Finding these “needles” at internet scale requires collecting an exponentially larger “haystack.” In this talk, Erik Rye will describe two novel techniques he developed to collect unprecedentedly large network datasets. He will describe how he used these datasets to enable the discovery of new network security and privacy problems at internet scale. These include stark, real-world security and privacy vulnerabilities, such as revealing troop positions in Ukraine and exposing previously-unreachable Internet of Things devices like smart light bulbs in users’ homes. Rye’s findings have prompted design changes in systems run by Apple, SpaceX, and router manufacturers, and improved the security and privacy of millions of affected individuals.
Bio: Erik Rye is a final-year PhD candidate at the University of Maryland, where he focuses on solving large-scale network security and privacy problems. He regularly publishes in venues like the ACM Special Interest Group on Data Communications Conference and IEEE Security & Privacy, and he has shared his work at industry conventions like Black Hat USA and in popular media like KrebsOnSecurity.com. Rye contributes to the network security and measurement communities by running the IPv6 Observatory, which publishes weekly insights into the state of the internet. He holds master’s degrees in computer science and applied mathematics from the Naval Postgraduate School, and also likes dogs.
