AI/ML Lecture & Networking @ Intuit

A monthly ML research colloquium by the New York Machine Learning Research Guild — this session in collaboration with our partner, Intuit. Long Context and Retrieval Rohun Agrawal · PhD Candidate, Computer Science, Columbia University · NSF Graduate Research Fellow Leon Li · PhD Candidate, Computer Science, NYU Courant · Modal Topic: Advances in Architectures for Long Context and Retrieval Long-context inference is bottlenecked by the growing KV cache, in both memory and latency. However, existing KV cache compression methods either lose accuracy or must prefill the entire context before they can compress it. We introduce Latent Context Language Models (LCLMs), a family of encoder-decoder compressors trained end-to-end at scale that map long inputs to a short sequence of latent tokens before the decoder consumes them. LCLMs establish a new Pareto frontier in accuracy versus latency and memory over KV cache compression, while unlocking new capabilities such as agents that skim a compressed context and expand relevant segments on demand, and nearly lossless online compaction of the model's own outputs during generation. For grounding language models in large document corpora, Retrieval-Augmented Generation (RAG) is the ubiquitous solution, yet it relies on a brittle lookup that can't be trained end-to-end and slows generation by prefilling long contexts. We introduce Retrieval Layers, learnable sparse cross-attention layers that retrieve external text in latent space and Pareto-dominate RAG on accuracy versus latency. About Rohun Rohun Agrawal is a PhD candidate in computer science at Columbia University, co-advised by Micah Goldblum and Pavel Izmailov, and an NSF Graduate Research Fellow. Rohun's research pushes the temporal horizons of machine learning models: extending the future horizon through planning with world models, and the past horizon through efficient large-scale memory retrieval. Recent work includes Closing the train–test gap in world models for gradi