Skip to main content
Robert Nishihara

Robert Nishihara

Computer scientist

May 6, 2026

13:31
I'm just wondering if you can give some of the ways that you think about the juxtaposition of Ray versus something like an orchestrator like Airflow or Dagster or a distributed compute system like Spark, and some of the selection criteria for when somebody would use Ray for a particular use case versus reaching to some of the other tools that are more of these, I'm gonna say, pipeline native workflows.
13:56
Yeah, that's a great question.
13:58
So data preparation, training data preparation and pre-processing is something that has changed so much since, you know, the early days of deep learning.
14:07
If you remember early on when people were training deep learning models, you would pre-process your data, but the pre-processing you'd do was very limited.
14:18
You might, in the case of ImageNet, you know, a computer vision dataset, you might try to y- sort of take each data point and generate multiple data points to augment your dataset, and you could do that by taking each image and randomly cropping or scaling it to get another, uh, you know, to get some, some more versions of that same image.
14:38
And now people have, uh, you know...
14:42
And, and going back to that point in time, all of the research was on model architecture, right? The ImageNet dataset was a fixed dataset and the, you know, split into your training set and your test set, so it's a sort of this fixed benchmark.

9 MINS LATER

23:56
And so you might sometimes find yourself fighting with the Kubernetes orchestrator, where it's saying, "Hey, you're done over there," and you're saying, "No, I'm really not." And I'm just wondering if you could talk to some of those aspects of how teams are dealing with some of this question of resource optimization at the hardware level to be able to get the most out of their expensive compute.

We value your privacy

We use cookies to understand how you use our platform and to improve your experience. Click “Accept All” to consent, or “Decline non-essential” to opt out of non-essential cookies. Read our Privacy Policy.