Big data

Modern astrophysics is characterised by massive, hyper-centralised international collaborations, and an often tacit belief that fundamental physics sits at the top of the scientific pyramid, while the questions of real astrophysics are mere nuisances.

This ideological elevation of the fundamental over the particular is evident in the architecture of modern big data astronomy.

The past few decades have been characterised by experiments yielding massive volumes of data.

Yet these pipelines are not neutral data-gathering machines.

Tech is always about the new, shiny thing.

During Under Armour's second era, big data and IoT were flavors of the year.

Big data promised a world where companies could harness data and derive deep insights that would increase sales, lower costs, and identify consumer trends before they even appeared.

Silicon Valley warned that any company not leveraging big data and not making data-driven decisions would be at a permanent disadvantage.

Internet of Things, or IoT, promised a future where everything in the world would be connected to the internet and that would forever transform how humans work, live and play.

With IoT, the lights in your home would automatically turn on when people entered the room, your fridge could track how many eggs you had left, your bathroom mirror would show you the weather, your shirt would measure heart rate, and the gym mirror could teach you form and technique in real time.

Wearables like Jawbone, Pebble, and Fitbit jumped into mainstream attention, big data vendors like Hortonworks and Sumo Logic exploded, and startups who built their entire business on analytics like Stitch Fix got funded.

Big data and IoT spread through hype alone to every industry, even home security as we covered earlier this season with ADT.

and their lengths between the two types are not within a type.

So having all these big data, big data set, you will say, well, that's great, right? So a lot of opportunities, a lot of great things that you can do.

But are there also challenges too? So if I ask you that question, the answer would be yes.

And I would argue that a lot of the challenges come from due to one single number, number seven.

So this might be a good audience to ask because a lot of you have HCI backgrounds.

So anyone can tell you why that might be a challenge in dealing with big data set.

Yes.

In a human memory capacity, short-term memory capacity,

I mean, you can't run a deep neural net on a standard laptop.

You need big data and the cloud.

You really do.

And when we talk about big data, we have to understand the sheer physical strain it puts on hardware.

It's often defined by the five Vs.

You have volume, obviously, the petabytes of storage required.

Ryan calls me and he's met the astrophysicist at a bar, not really in a bar, I was actually on an escalator.

But they, you know, they were doing this work and they wanted to use LLMs to learn from Big Data.

And basically, Brian and I looked at each other and said, well, okay, if you're having this problem over here, you know, you're using LLMs on Big Data and like web traffic.

These people are thinking about an astrophysics.

It probably applies broadly.

So then we kind of zoomed out.

And what we recognize is like, in particular, in the world of people who are trying to save or improve a planet and make it a better planet for the future generations Thank you.

There is a large cluster of this same problem, big data, lots of intelligence baked inside of that data, challenging to get that intelligence out.

I'd say tactical positions.

You've got MSCI Taiwan and AI Big Data, a lot of thematic stuff.

I don't think they're broad enough for most investors to get really interested in it.

But I think more choice, as you said, is good.

And he spoke with the Space Association of Australia at one of our meetings.

And we talked about the Square Kilometre Array's big data.

He's speaking here with Peter Arwood, at the time the president of the association.

One of the things I want to emphasize tonight was the translation.

So a lot of the skills we developed for the SKA in engineering and data science and astronomy are translatable, are usable for other things.

And these are three things I want to talk about tonight.

One is a thing called space situational awareness, which you've probably heard of, the laser communication Sasha is going to talk about, and I'll talk a bit about big data.

So what's space situational awareness got to do with astronomy, or radio astronomy in particular? Remember I talked about the Murchison Wide Builder Ray? It's a telescope that sits up there in the Murchison.