Starting a Giant Online Data Platform Is the Easy Part

It's so powerful because the EIA was such a good data source.

We can talk about this.

There's so much there.

But even as a journalist who covers these topics and feels like I know the EIA data decently well, often my best path for accessing it would be you put in a command in Cloud Code.

And it like churns for five minutes.

And then it spits out an answer.

But even when it spits out an answer, it's not necessarily visualized.

I would have to take a few more steps to put it in the tool that we use at Heatmap.

That's all to say, this tool is extremely useful and I really appreciate it.

What did you learn from making this tool? Just in terms of the charts that you encountered while putting the platform together, are there...

Things you thought you believed about U.S. energy or things you thought to be the case that then upon working at the coalface of the data, so to speak, you discovered were not actually true.

Yeah, I mean, I think the first thing I learned is how deep the US EIA data is.

I think the point is that it's a really incredible data source.

The people collecting this data, publishing it, are doing amazing work.

I think where it falls flat is this last hurdle of making it accessible and usable for people unless you're a complete data nerd and know your way around the website and know exactly where to go.

So I think one is just appreciation for the depth and detail of the amount of data that's there.

I think in terms of... learnings from the data.

I think there are quite a lot of things that you would intuitively think to be true and actually when you start to look at the data they're often incorrect.

I mean I think one of the key ones is there's lots of discussion about you know electricity load growth and how that affects prices right and often I guess the conventional wisdom would be from supply and demand, that if your demand is really tightly squeezed, then you see an increase in prices.

And actually, when you look at the correlation there, there's very little, or if anything, it may be slightly points in the opposite direction, right? The states that have had the largest load growth have actually seen the smallest increase in prices, for example.

I think a very common question people have is how does the, in terms of the energy transition, how does the shift to a given energy source affect prices? And we're often looking for these correlations like against gas, if gas goes up and down, what happens to prices, solar and wind, if you generate more.

Solar and wind as your prices go up and down.

I found this in the international data previously when I looked at it, that often you'd find very little correlation for any of the sources, which again is quite counterintuitive.

And you also generally find that with the US state level data.

where my takeaway from that is like probably I do think the choice of sources does affect prices but I think there are actually other factors in this energy system that are probably overpowering that and therefore you just don't really see a very very clear signal so there's lots of stuff like that when you start to explore kind of challenges some of your previous assumptions.

It's so powerful because the EIA was such a good data source.

We can talk about this.

There's so much there.

But even as a journalist who covers these topics and feels like I know the EIA data decently well, often my best path for accessing it would be you put in a command in Cloud Code.

And it like churns for five minutes.

And then it spits out an answer.

But even when it spits out an answer, it's not necessarily visualized.

I would have to take a few more steps to put it in the tool that we use at Heatmap.

That's all to say, this tool is extremely useful and I really appreciate it.

What did you learn from making this tool? Just in terms of the charts that you encountered while putting the platform together, are there...

Things you thought you believed about U.S. energy or things you thought to be the case that then upon working at the coalface of the data, so to speak, you discovered were not actually true.

Yeah, I mean, I think the first thing I learned is how deep the US EIA data is.

I think the point is that it's a really incredible data source.

The people collecting this data, publishing it, are doing amazing work.

I think where it falls flat is this last hurdle of making it accessible and usable for people unless you're a complete data nerd and know your way around the website and know exactly where to go.

So I think one is just appreciation for the depth and detail of the amount of data that's there.

I think in terms of... learnings from the data.

I think there are quite a lot of things that you would intuitively think to be true and actually when you start to look at the data they're often incorrect.

I mean I think one of the key ones is there's lots of discussion about you know electricity load growth and how that affects prices right and often I guess the conventional wisdom would be from supply and demand, that if your demand is really tightly squeezed, then you see an increase in prices.

And actually, when you look at the correlation there, there's very little, or if anything, it may be slightly points in the opposite direction, right? The states that have had the largest load growth have actually seen the smallest increase in prices, for example.

I think a very common question people have is how does the, in terms of the energy transition, how does the shift to a given energy source affect prices? And we're often looking for these correlations like against gas, if gas goes up and down, what happens to prices, solar and wind, if you generate more.

Solar and wind as your prices go up and down.

I found this in the international data previously when I looked at it, that often you'd find very little correlation for any of the sources, which again is quite counterintuitive.

And you also generally find that with the US state level data.

where my takeaway from that is like probably I do think the choice of sources does affect prices but I think there are actually other factors in this energy system that are probably overpowering that and therefore you just don't really see a very very clear signal so there's lots of stuff like that when you start to explore kind of challenges some of your previous assumptions.