Oct 6, 2026 · 40 min · 11 segments
Jennifer Neville did not want to go into computer science—but that’s exactly where she landed. Neville discusses the starts and stops that led to her professional sweet spot and her work identifying…
Jennifer NevilleGuestChad AtullaHostWe noted that one of the themes of your research is how we can improve these systems to bring them better into alignment with what users actually want.
And you mentioned a case here of, hey, well, we know that users realistically are often not fully specifying what they want in their first turn.
And after you put this paper out, they expressed... resonance with this frustration about getting lost in multi-turn conversations.
How do you think about understanding the user's wants and needs and experience? And how does that relate back to what you do with, as you said, simulating a user, for example, to run these longer multi-turn evaluations?
Yeah, that's getting to see at scale what users are trying to do and what kind of failures and successes they have in the systems is one of the advantages I think you get if you're in an industry lab and you get to see data at scale like that.
So we have done a lot of work with the product groups here internally at Microsoft analyzing consumer logs at scale to see to figure out what are the main patterns of failures that that users are experiencing.
And that is the provides the kind of insight or motivation for a lot of the a lot of the work that we do.
And there's a lot of great product groups that are also analyzing the successes that users have, which would allow you to then even recommend to users how What are the kind of tasks that they're going to be able to do successfully in the systems that we have now? I think if you think back, when search engines first started, people had to learn how to interact with search engines in a way to effectively get the information that they're looking for.
We might have forgotten that that happened now because it was so much a part of what everybody would do before all these AI systems came out.
But now I think there's a transition away from the search engines to these chat based systems.
And I think there's going to have to be some learning from the users as well in this new environment.
And I think analyzing the things that are successful in terms of what users are doing is a great way to start recommending and tutoring or teaching the users in the same way.
But what does that actually mean? Are you genuinely looking at real data? How is privacy coming in here? What sort of processes do you have for responsibly leveraging data in the wild?
We are analyzing the data at scale to extract patterns of failures or successes in a privacy-preserving kind of way.
There's also a lot of restrictions that we can't actually see the data eyes on.
We can only sort of push methods to process the data in very restricted environments and then And then these sort of higher level insights that are privacy preserved are the things that then fuel like what we would do algorithmically later on.
So just to be clear, we're not fine tuning on your data or your responses, but the more details you can give in your responses, whether it's through donated information information when you give a thumbs down and you give a description of it or if it's in the sort of chat interaction you have with the models those can eventually get distilled into things that are going to improve the models not through people actually looking at what you've been doing
We noted that one of the themes of your research is how we can improve these systems to bring them better into alignment with what users actually want.
And you mentioned a case here of, hey, well, we know that users realistically are often not fully specifying what they want in their first turn.
And after you put this paper out, they expressed... resonance with this frustration about getting lost in multi-turn conversations.
How do you think about understanding the user's wants and needs and experience? And how does that relate back to what you do with, as you said, simulating a user, for example, to run these longer multi-turn evaluations?
Yeah, that's getting to see at scale what users are trying to do and what kind of failures and successes they have in the systems is one of the advantages I think you get if you're in an industry lab and you get to see data at scale like that.
So we have done a lot of work with the product groups here internally at Microsoft analyzing consumer logs at scale to see to figure out what are the main patterns of failures that that users are experiencing.
And that is the provides the kind of insight or motivation for a lot of the a lot of the work that we do.
And there's a lot of great product groups that are also analyzing the successes that users have, which would allow you to then even recommend to users how What are the kind of tasks that they're going to be able to do successfully in the systems that we have now? I think if you think back, when search engines first started, people had to learn how to interact with search engines in a way to effectively get the information that they're looking for.
We might have forgotten that that happened now because it was so much a part of what everybody would do before all these AI systems came out.
But now I think there's a transition away from the search engines to these chat based systems.
And I think there's going to have to be some learning from the users as well in this new environment.
And I think analyzing the things that are successful in terms of what users are doing is a great way to start recommending and tutoring or teaching the users in the same way.
But what does that actually mean? Are you genuinely looking at real data? How is privacy coming in here? What sort of processes do you have for responsibly leveraging data in the wild?
We are analyzing the data at scale to extract patterns of failures or successes in a privacy-preserving kind of way.
There's also a lot of restrictions that we can't actually see the data eyes on.
We can only sort of push methods to process the data in very restricted environments and then And then these sort of higher level insights that are privacy preserved are the things that then fuel like what we would do algorithmically later on.
So just to be clear, we're not fine tuning on your data or your responses, but the more details you can give in your responses, whether it's through donated information information when you give a thumbs down and you give a description of it or if it's in the sort of chat interaction you have with the models those can eventually get distilled into things that are going to improve the models not through people actually looking at what you've been doing
The rest of this transcript — segmented and speaker-labeled, so you land on the exact moment something was said
Search every transcript — by keyword, by phrase, or by meaning, across every show Radar indexes
Trends — what is surging across podcasts, measured against its own baseline
Alerts — when a name you follow appears in a newly indexed episode
No account is needed to search Radar.
