Cyber NYC Initiative

• Is Defense Winning in Cyberspace?
  • PIs: Jason Healey, School of International and Public Affairs
  • Little academic work has explored the nature of failures in cyber markets, or how regulation can impact these markets. This project investigates and develops initial frameworks for cyber market failures and cybersecurity regulation. It also works on more basic research, based in such frameworks, for regulatory options rooted in technology and computing. 
• pdslib: Making Private Aggregation a First-Order API in Modern Operating Systems
  • PI: Roxana Geambasu, Computer Science; Asaf Cidon, Electrical Engineering & Computer Science
  • Building on success creating differential privacy architecture for web browsers, the research team is aiming to extend privacy-preserving aggregation to mobile operating systems. The research will help improve privacy protections for mobile device users by reducing reliance on data collection.
• Building Safe LLM Agents with Environment-Awareness
  • PI: Micah Goldblum, Electrical Engineering
  • While large language models are often employed as part of powerful agentic systems, they’re also vulnerable to correspondingly dangerous security and privacy breaches. This project introduces a method to make LLMs aware of the risks associated with the environments in which they operate, helping them make more appropriate and secure decisions.
• In-model Firewall: Flexible, Generalized Defense Against LLM Jailbreaks
  • PI: Junfeng Yang, Computer Science
  • Large Language Models are prone to jailbreak attacks, where prompts are designed to elicit harmful outputs. This project aims to make LLMs more robust to jailbreak attacks by creating dedicated firewall layers within the models that detect and respond to the core semantics of a prompt, while maintaining model utility. 
• Differentially Private Streaming Algorithms for Continual Observation in the Turnstile Model
  • PIs: Rachel Cummings, Industrial Engineering and Operations Research; Rebecca Wright, Barnard College, Computer Science
  • Differential privacy (DP) is a mathematically rigorous definition of privacy that has gained significant popularity since its formalization in 2006. However, it involves tradeoffs between privacy and accuracy that can be difficult to understand. This project builds a tool for visually explaining differential privacy to engineers, with a focus on the accuracy resulting from varying the privacy parameters. 
     
• KAgent: A Neuro-Symbolic Agent for Safely Resolving Kernel Vulnerabilities
  • PIs: Baishakhi Ray and Junfeng Yang, Computer Science
  • Effectively repairing bugs in software is a task that’s important for our society and infrastructure, but often complex and costly. This project develops AI agents that use multiple different types of information - like source code, bug reports, stack traces, and system logs - to effectively locate and patch software bugs. 
• Fuzzing with Tool-Assisted LLMs
  • PIs: Suman Jana and Junfeng Yang, Computer Science
  • Fuzzing has been highly successful at finding bugs in large real-world software. However, fuzzers often hit a performance plateau after their initial success, because they lack understanding of what types of inputs to use.  This project improves fuzzers’ abilities to accurately detect bugs by improving their statistical rigor. 
     
• Cryptographic Techniques for Safe AI
  • PI: Tal Malkin, Computer Science
  • This project builds on research investigating LLM watermarking, where AI-generated content is marked so it can be distinguished from human-generated content. The researchers will integrate techniques such as interactive proofs and adversarial noise models, among other cryptographic techniques, to make AI models safer.

2024-2025 Projects

• Bounding Measures of Inequality for Trustworthy Machine Learning
  • PI: Richard Zemel, Computer Science; with Toniann Pitassi, Computer Science
  • Learning-based predictive algorithms are widely used in real-world systems and have significantly impacted our daily lives. However, many algorithms are deployed without understanding of their potential for failure or how they perform across different populations. This project develops techniques that help algorithm designers ensure their models are fair both across and within groups.
• Constructive Responses to Harassment
  • PI: Susan McGregor, Data Science Institute; with Rachel Greenstadt and Damon McCoy, New York University
  • This project investigates a user-centric approach to mitigating online harassment. The researchers plan to assess how online reporting mechanisms can better meet user expectations and needs, thus mitigating the harms of online harassment and abuse. 
• Securing the Software Supply Chain via Effective Detection of Rogue Updates
  • PI: Junfeng Yang, Computer Science, with Yaniv David, Computer Science
  • This project works to detect malicious updates to benign software. This type of malware is known to be extremely dangerous; one prominent example is the famous SolarWinds hack that compromised thousands of organizations including nine US government agencies. The researchers have invented a novel algorithm and built a system for effective detection of rogue updates for packages in Javascript, one of the most widely used programming languages. 
• Mitigating Human Vulnerability to AI-Generated Misinformation
  • PI: Tamar Mitts, School of International and Public Affairs
  • AI-generated disinformation is a major concern across sectors, with LLM-generated content becoming more ubiquitous, persuasive, and targeted. This project cllects systematic data on the use of generative AI in disinformation campaigns, and tests mitigation strategies such as watermarking across contexts and information types.