Ty Fenton  00:00

Software is becoming the defining feature of energy storage, more and more, as time goes on.

Nico Johnson  00:08

Energy storage has certainly become more than a fad. I often like to say that storage is the tail wagging the dog of the clean energy sector. Energy storage is so useful in so many ways, and it is scaling quickly. However, operating these batteries turns out is more complex than we initially imagined. Today, we're going to explore how data analytics and automation specifically are going to change how we think about the way storage fleets are managed. To talk about that more deeply, is Ty Fenton, the director of data analytics at Flexgen. Ty, thanks for taking time to swing by.

Ty Fenton  00:46

Thank you. Thanks for having me.

Nico Johnson  00:48

No, you started in remote operations, right here in our home state of North Carolina, at a company that many would know, Strata Clean Energy. Really, actually watching how these assets perform in real time. I'm curious, how that experience taught you the ways these systems operate and break, and how that eventually led you into storage.

Ty Fenton  01:09

Great. Yeah, so when I came out of UNC undergrad, I went to work for Strata, Strata Solar at the time, like you said, and I started in the control room, and at the time I was kind of a little bummed, didn't seem too exciting, just kind of work in a control room, but now I look back and I'm so grateful that I had the opportunity, because it gave me the opportunity to learn everything about how these energy assets operate. I was there taking calls from the site, someone saying it's got error code 1432 what does that mean, and reading the manual, figured out how to fix it, and during my time there, I eventually started to understand more about the data we collected, and using my background in statistics that I learned at UNC, I was able to then go to our CEO and say, "Hey, there's a lot more I could learn from this. Can I start doing the next level and start to try to stop these faults before they happen? Find underperforming systems on a batch, not like one at a time with manual intervention. And so, eventually, it kind of led me to this point of using the data coming off these solar sites for more and more impact to optimize Stratas fleet, and I did that in solar for, for a couple years. I did that at Strata, then I went to Engie, and then Goldman Sachs renewable power group, which is now Emanate.

Nico Johnson  02:11

Yeah, it

Ty Fenton  02:12

was actually at a meeting at Eminate when I was there, and I've been, you know, in solar for a while, you know, really got into the weeds of this, like finding, you know, all the different tracker angles, in a row shading and soiling and when to wash panels, all this interesting analytics, and I was in a meeting of about a dozen solar experts, you know, from asset managers, operators, the whole kind of like the works of experts there, and we were looking at this contract for an energy site, look at the data coming in from energy storage site, and no one knew what it was, we were all like, what is what is this field, how do we track this metric, what is this, and I was just.. I could tell at that point. This teleported you back like 10 years exactly. This was.. this feels.. this was 2021 I think. And I could just tell, I was like, this is the next thing. Obviously, you could tell that anyone could tell. Yeah, energy storage was.. was becoming more and more common, for sure. And I was like, if I'm going to learn and sort of be the leader in the energy storage space, I'm not going to do it from a solar perspective. I need to go into energy storage, and so that was really the impetus for me to jump and end up with with flex gen, and I was really excited to get directly into the storage space and try to, you know, bring some of the data insights that I learned from it from solar into storage.

Nico Johnson  03:18

Yeah, I want to point out to one of the pieces of advice that I give folks there is this in the, in the sort of the TikTok culture that is so prevalent now. I'm going to say kids, like young professionals, want to jump over steps, and they miss the reality that, like, really learning the fundamentals is fundamental to success, long-term success. If you go to a big consulting firm, they don't let you like immediately jump in and lead without having learned literally how to do a PowerPoint, right? Like, your job is make this PowerPoint, which can feel rather banal when you are the top performer at your whatever Ivy League school, but jobs like what you had at Strata, jobs at I tell folks, like, go to a utility. I don't want to work at utility. Utilities are boring, they're the enemy. Oh, well, then the silt. Then tell me, how you're going to learn the way the power industry works have any empathy for the operators of the grid, or really understand the tension between firm power, resource adequacy, capacity, natural gas generators, etc. Like, can you learn all that at Solar Startup A?

Ty Fenton  04:37

It's, it's fundamental to know that, and I've been learning that piecewise through my experience, it would have been great to have that time. Actually, I did intern at Southern California, since I learned a little bit of it. Nice,

Ty Fenton  04:46

didn't go there long enough.

Nico Johnson  04:48

And yet, when you went to Strata, because you actually were able to watch at the operations level how the plants perform, yeah, you got effectively like the solar industry version of that. You got this. Indoctrination in the the the how and why these assets are creating the value they do. It

Ty Fenton  05:08

was fundamental, and actually, while I was the director of data analytics, my prior role, I told all the data scientists, data analysts to spend at least a week shadowing the the rock text, yeah, sit in the control room, see what they do, help out where you can, it's not enough, but that was something kind of on that path of building those fundamentals. It's essential.

Nico Johnson  05:26

You and I were talking about how I've been having conversations lately with operators around storage assets that confirm the suspicion that these assets really do require different level of data and sort of operational attention than maybe traditional solar assets. Could you unpack that a bit for me? What you've learned since your time at Flex Jen.

Ty Fenton  05:48

So energy storage assets are far more complex than solar, and really what that comes down to is the distributed nature of those storage assets. And so with energy storage, you have people might think it's, you know, one big container, you got a container of batteries, and it's just going to work like your phone does, and start to discharge evenly, but inside of those big containers you might see out in the field are individual cells, and you'll have hundreds of those cells within each container, 1000s across the entire site, and what makes that really crucial is you have 1000s of cells that can be wired together electrically, and so you have any one of those cells be the limiting factor on that site, and so what you see in practice is you might be discharging your site, the operators say you have 50% state of charge, you want to discharge it down to zero to meet some energy arbitrage opportunity, and during that cycle you'll suddenly go from 20% state of charge down to zero. Yeah, and what that is, is you might have 20% state of charge in those cells, but you have one cell in series with the others that has hit the low cell voltage threshold and cannot discharge anymore, and how it works also in operation is once you've stopped the discharge, it'll recover back to the same voltage that this pier, so you've sent someone out to site and say, "Hey, go find that bad cell, I know it's in this rack. Yeah, they'll go out and won't see anything wrong. You need to be able to process and analyze the data in that temporal element of when you have that fault happen in order to catch what's going on in the site, okay. And so there's a huge data element to

Nico Johnson  07:07

it, okay. So I want to pull a thread here that I hear what you and Leonard, who I've had on the show, are saying by way of giving this very specific detailed example, I don't hear how it's different, for example, from an actual solar cell on a solar module underperforming. The thing that occurs to me is under voltage of a cell because a bird poops on that square of a solar panel used to be a big problem, because all the cells are wired, yep, exactly the same way in series, yeah, and we fixed it more or less with bypass diodes. What you're telling me is that in the battery architecture, there's not a bypass diode equivalent, there's not a bypass diode equivalent at all, everything's wired in series, and, and also with with the the modules, you'll have somewhere in the realm of a dozen, couple dozen, maybe in, in the string, I guess, yeah, 150 modules, string

Ty Fenton  08:10

limited your string, and so, but the worst case, you have one really bad, bad module that's slowing down the current through that whole DC string, that's that's going to be a smaller fraction of your site than you get in energy storage, where you'll have really, you'll have hundreds of cells together, and it can be for some of these sites, can be, you know, a third of your site that all sudden is cut off because of that one cell, exactly. Okay, it's far more that feels

Nico Johnson  08:32

like the problem that needs to be fall solved, rather than like, yes, step one is like, well, let's understand the data,

Ty Fenton  08:38

yeah,

Nico Johnson  08:38

so that we can actually have this kind of a conversation, but what is the maybe you don't have the answer for this, but is the industry working on by like the equivalent of a bypass diode, so that we don't have this critical limiter?

Ty Fenton  08:49

There, there is, there are some applications of that. I haven't seen it in large scale. One solution to that is string inverters, where you don't have, because it is, it's the electrical control is the inverter PCS. Yeah, and so if you are able to distribute that more and have string inverters at each rack, that means that now you have only, you know, 100 200 cells that will, that will be operated together instead of 1000 or more. Yeah, so that's that's one solution. You could theoretically have some sort of bypass diet, but it hasn't been done in operation yet. That's a hardware problem that hasn't fixed yet,

Nico Johnson  09:23

you know, at Flexgen, you guys talk about moving towards closed loop battery operations. First, I'm not sure what that means, so I'd love for you to sort of disambiguate that term for me. But what does it look like in practice to have closed closed loop? What problems does automating solve that human operators struggle with?

Ty Fenton  09:39

Yeah, so I've been at Flex Gen for about four years now, and when I first joined, my main task was, I mean, a collect the data in a usable format and get it into a central database, that was a key thing, because there's actually, there's far more data that comes off these energy storage sites than solar. Solar collects data every five minutes, or even 15 minutes is normal, with battery new storage, you're sampling certain, certain signals. Sub second, and at the very most, a second level sample is normal from far more data points as well, because you have all every one of those cells will collect current voltage data from, and so you have a lot of data. Just that data management problem is a huge problem that a lot of companies haven't solved yet, and it really impacts their operations. But beyond that, my first kind of three years there was, how do I get the expert knowledge that is in just a few people's head, the commissioning tech, the, you know, the operator, the software engineer. How do I get that, and to put it into algorithms, so that we can extract that knowledge and distribute it to all of the remote operations center technicians, to all the customer owners that own these sites? How can we get that out to them? And now the next step is, well, okay. So we have a reoccurring problem that we know is going to happen. We have a known solution that we know how to fix it, and we know the basically the signal that this is going to be a problem. Now we've able to determine some predictive diagnostics analytics that will come in and say, hey, this hasn't failed yet, but in about two weeks you're going to see a failure on this one cell. We can tell that one's degrading, it's going to be a problem, and so what we do is close that loop, and rather than having a human has to go there and say, you know, here's a breakthrough diagnostics that says we're a problem, I'm going to go ahead and dispatch someone here, or recalibration is another problem, you have to recalibrate your state of charge due to coulomb counting drift on these cells, we're able to close that loop and say we know this will be a problem. Here are the signs that it's becoming a problem. We're going to go ahead and just do that for you overnight when you're not using the battery. That's the next step, and it's actually a really crucial part of what human operators do right now in these assets is that that sort of manual intervention. We're trying to cut that out, so they can focus on the true optimization of how to trade better and higher strategy decisions they need to make.

Nico Johnson  11:42

Okay, if we are pulling the humans out of that, because we can sort of algorithmically address it now that we've had enough human sort of observation and analysis, aren't we also automating trading? What I heard you say is like we're moving humans to this other area where we're already automating and using AI, I think a question that comes up for me is, okay, well, Where are the humans irreplaceable in the process?

Ty Fenton  12:08

These are energy-critical assets and critical to our entire infrastructure, and, and so you always want to have a human in the loop to oversee anything that's going to be, you know, a major charge, discharge that has a chance to back the grid, and there are narc regulations around that too, where we're not saying, you know, hey, we're gonna put a computer in charge of your site. Yeah, so humans are really critical to oversee these systems in most cases. Yeah, where we're putting taking humans out of the loop are things like these automated maintenance actions, where you need to recalibrate your state of charge. Yeah, you know, that's a manual process that if you're gonna do it and have a computer human sit there, it's really manual and tedious. Let's just take that out, totally out of your loop. Let's predict failures where they happen and dispatch a tech automatically. That's the stuff that we're trying to cut humans out on it. Humans should stay in charge the

Nico Johnson  12:49

best use. Humans should stay

Ty Fenton  12:50

in charge of the actual charging and discharging of the battery for the time being, and we're going to keep evaluating options there,

Nico Johnson  12:56

and at the moment also of the process of writing the instructions, right, like really deciding the decision theory around how the automations work

Ty Fenton  13:06

exactly. It

Nico Johnson  13:07

does feel like we're in a moment right now where folks on in the AI realm would be like, "Oh, you're wrong, like our agents can do that. We're running to be, to be, to be determined.

Ty Fenton  13:16

We're running some shadow tests where we're looking at internal algorithms compared to our our manually written control loops, and see if we can share improvements, but it's not anything that we are ready to take to production yet.

Nico Johnson  13:28

While we're on the topic of AI, obviously hyperscalers, data centers are both super popular, certainly from a power perspective. Our industry is all you can talk about these days, and very disliked in most of the communities. Trigger was saying to me recently that there's a poll that shows that data centers are now poll as worst liked than ICE.

Ty Fenton  13:52

Wow, you

Nico Johnson  13:53

like that statistic, that's impressive. So, if we're talking about AI and data centers, it's because we're trying to figure out how to power them, and if we're trying to figure out how to power them, then naturally, like speed to power matters. We know we can deploy solar and batteries, but I'm hearing conflicting data around what batteries can actually do for data centers.

Ty Fenton  14:13

Yeah,

Nico Johnson  14:13

what's your position on that, or your take on where batteries work and where they don't make sense?

Ty Fenton  14:19

Yeah, so flex 10 has really been leaning into the data center space a lot, because there is space for batteries, sort of everyone sort of knows that batteries, being such a an integral tech to the grid, they must have a place in data centers, and so we've been trying to really explore what works and what customers want, and we came in first with the idea of well, we can replace your diesel generators that are your backup, your unadopted power supply, UPS, we thought we can replace that with the battery, we can put that there, and you can ride through any small grid outages with the battery. Turns out that even though that is a fair application, and you're seeing some applications of that, it's not something that most data center operators want to take yet, because their backup is tried and true. It's gotten them through, you know, hundreds, 1000s of utility outages, they know it works, technology works, they. Diesel generators are familiar with, and so they're not ready to take that step yet to replace them with batteries. I expect that will come later, but what they are looking at in the meantime is actually very specific to the new technology coming up, this AI transient load from the training of the AI models, and so how that works is when you're training an AI model, all the chips in the entire data center, so every single chip has to work in synchrony with the others during the training cycle, and so you have these data, these energy spikes, where you'll have an intensive part of the training load and the energy spike use, energy usage spikes, and then you'll have this trough where you're communicating to the other chips to kind of update on everything that happened, and so you got this really variable energy usage, and what that means is that whether you connect it to the grid or a data center has brought their own power, their own turbines that are that are on site, it it's too much variability in the energy load for those generators to take. Yeah, and so you need some way to stabilize it, and there are, you know, a couple ways to do that, but really, by far the best way, and most energy efficient way, which is, you know, really my goal here is, let's not waste energy is is batteries, because you can just simply, you know, charge and discharge from the battery to to meet those spikes and troughs, where, and keep a stable load on the grid. Where

Nico Johnson  16:11

does the battery sit in the overall architecture? Then, so, so, because what you're saying is it's acting as a shock absorber.

Ty Fenton  16:16

Yep, so the battery is going to sit between the data center and the generator, whether that's the grid at the

Nico Johnson  16:21

medium median voltage.

Ty Fenton  16:22

Yes, not

Nico Johnson  16:22

at a low voltage.

Ty Fenton  16:23

Well, you might have it on a lower.. it would depend, and on the setup, there's there's options for both. Either way, it's going to be between the data center and the grid, because you need to stabilize that.

Nico Johnson  16:33

Okay, but not between the generating source, like the backup power source, and the data center, mostly between the grid and the data center,

Ty Fenton  16:41

it's going to be between the data center and, and the grid. Yeah, and so it's not.. it depends. So, if you have solar on site as well, and you want to charge your batteries with solar, that means you're going to be at at a lower voltage, you're not going to be at grid voltage. That's right. And so, it will depend a little bit on the application. Yeah, either way, batteries can provide that dampen. I mean, one

Nico Johnson  16:58

of the things that I'm.. what I'm drilling down on this with use like flex gen has a long history of like first in oil and gas, like working as with offsite power or off grid power, like micro grids working in hybrid scenarios with natural gas plants, etc. And I'm trying to figure out what the architecture is supposed to look like, because I've been having conversations with, like, long duration storage folks, and everybody sort of has their approach to the mousetrap of what's going to serve the AI data center best, and I don't presume that you have the answer per se, but I'm curious if you can help eliminate sort of a version that you're seeing data centers say this is what we want, rather than then solution providers saying this is what you should want.

Ty Fenton  17:46

Well, right now we're being approached by data center companies that are coming with very specific needs, and so everyone is a little different, and that's something we've actually had to avoid, is going in saying we've got your solution, it looks like this, here you go, because that's not what these very unique setups have everyone's bringing up some, some bring in their own generation, some have very limited, great interconnections, and don't want to go above that, or we have all these different needs that are that data center operators are coming to us with, and so we're trying to say, you know, we can work with whatever, like you said, Flexion does have a background with controlling fossil resources, right, spinning turbines, solar, you know, every all these different resources we can control, and so we sort of have done all of it and have the control algorithms to do so, and it's a bit of what do we need, and or what do you need the customer to the data centers, and how can we provide that to you.

Nico Johnson  18:34

Cool, if you're watching this and you're thinking, I wonder what all what this all means, we've got, we'll link to some other episodes that we're doing around data centers, in particular different approaches to battery storage, so that you can become better informed. This is really a fascinating conversation for me, and I think that it's one that, as we saw with solar, like, we'll be able to look back with perfect insight, because hindsight is 2020 and recognize where we made errors, but I wonder if we fast forward now 10 years, sort of if you put your thinking cap on, of like, oh, maybe you're running a battery company at that point, does the energy storage industry still look like a hardware business, because that's how, when I was watching sort of Fluence and others sort of begin to spin up even flex gen, it very much was a hardware business, or does it become more of a software defined energy system?

Ty Fenton  19:28

Yeah, that's that mean it's a great question. It's really forward looking, and we're starting to see the emergence of the software defined business becoming more and more prevalent. That's really where we're going as we get more advanced use cases, whether that's you're wanting to layer multiple revenue streams on a battery in a grid, like ERCOT, or if you're trying to be a really critical grid infrastructure, or run a data center, software is more and more the differentiator than just the hardware, and that goes also to software. FlexGen integrates with any hardware provider, you can buy your batteries wherever you want and bring them in. And and that what that also does is it means that we can optimize any hardware too, and that software piece is really important. If you want to go out and buy different, different hardware streams, as they, you know, become more economical than a different setup, you need to have software that's able to handle all that. And so software is becoming the defining feature of energy storage more and more as time goes on.

Nico Johnson  20:18

It's interesting to see how the business models evolve, where you know where companies have to make this decision. Am I an integrator? Am I a software provider? It's clear that the decision at Flexion has been, we have to figure out how to be an operating layer rather than an integrator committed to one architecture. I think that's admirable, given the current state of the industry. Ty Fenton, thank you for taking the time to swing by and help us better understand the state of the battery economy at the moment.

Ty Fenton  20:47

Thank you so much. It was a pleasure.

Nico Johnson  20:48

Look forward to having you back sometime.

Ty Fenton  20:49

Absolutely, thank you.

Nico Johnson  20:51

Hey, thank you for tuning in. And if you are going to one of our other upcoming events, please swing by our live stage and say hello. I do love meeting you in person. Twice weekly, we deliver conversations with founders and leaders on the front lines of the clean energy transition, and we're here bringing this content to you each and every week, free of charge. Of course, it's not free; our sponsors help pay the bills and keep the lights on, so that we can help you build your legacy in the clean energy transition. And so, if you'd like to say thank you to them, or learn more about what it is that they bring into the world, or perhaps see how you could reach 1000s of listeners twice a week, just like they do, check it out at Sun Cast dot media forward slash sponsor. Remember, you are what you listen to. Thanks again for showing up, Solar Warrior. It's half the battle.

‍