Forecasting tomorrow's sunshine (to better use the solar)

graememk

https://www.boards.ie/discussion/comment/118759561#Comment_118759561

Prediction 21.7kwh, Generation was 19.9, but by the graph it looked like it was scattered cloud. Id call that close enough!

Tomorrows is 8 something. ( I havent done anything on angle correction or anything like that, as my panels are fairly flat.)

I take the sum of the daylight hours solar irradiance, multiply it by my panel area (each of my panels are 1.7*1, actually a bit more but taking some off for the frame etc) and then multiplied by 0.18 which is a conservative efficiency of the panel. Mine are fairly flat so that's how I'm getting away with it.

bullit_dodger

Yeah - I think that was a success. Forecasted 16.82Kwhr and actually generated 15.2Kwhr. So a difference of 1.6kwhr or within 10%. I think that's a "win". Some fairly decent enough curves to for the day.

Ultimately what you want is some reasonably decent range to operate in so you can decide roughly what to with setting your charge levels and/or moving loads like washing/drying to daytime if you think you'll have surplus. Sure it'd be nice if you could nail it down to within 1Kwhr, but weather is weather and it doesn't behave as nice as the numbers.

I think I'll be automating that algorithm into my app, and programming the SOC via the Web API. Good thing is that the model should work for anyone else too fairly accurately....and it's relatively cheap to operate. Tomorrows forecast = 7.15Kwhr

That's pretty shocking from forecast.solar. I mean it's ok to be out, but a factor of 100%.....that's kinda pointless.

connesha

Today was one of those fluke days where it was bang-on: Estimate 28.554 kwh. Generated 28.400. (I can't imagine it'll be that close again)

The far bottom-right "NULL" will be updated to 28400 when the script runs at 1:45am tonight. (At this same time the script will populate the next days row, with the exception of the "actual_wh", and so on each night)

Today is the first day the script is running, hence today having an individually calculated estimate_factor in the table.

The estimate_factor is calculated as: For each of the previous N days, divide actual_wh by radiation. Then get the average of these.

You can ignore some of the values in the other columns, as its just setup as an automatic script from today ,so the non-critical values were loaded with approx data (the radiation and actual_wh are accurate for all days, and that's all the matters for calculating today onwards..)

For completeness: the script sets these ones at 1:45 am on the day of:

• radiation - pulled from Met Eireann
• estimate_factor - calculated from the previous individual days actual_wh/radiation
• estimate_wh - radiation * estimate_factor
• battery_initial_wh - the battery soc (in watt hours, not %). This is so I know how much to add to the battery
• charge_on - did it set the inverter to charge from grid (based on the estimate_wh and battery_initial_wh)
• charge_wh - if charging is on, how many wattHours will be added to the battery between 2am and 5am
• charge_da - the deciAmps needed to reach this charge_wh (the inverter register uses deciAmps as unit)

This one is filled in on the next day, when script runs again at 1:45

• actual_wh - the generation for day, in wattHours

bullit_dodger

I like that. In fact I like the whole community chucking ideas out there and seeing what sticks.

So connesha how many days were you thinking of computing the average for. I used 10 in my own model. Like you though, I had one day in my spreadsheet above where it was uncannily accurate. March 5th was Forecast 13.08Kwhr, actual 13.07Kwhr. As you said, I think it's a fluke, but a nice fluke. :-)

connesha

I think somewhere between 1 and 2 weeks is probably the ideal. I guess you want it long enough that the average is meaningful, but not too long as the factor will change over time as the sun gets higher (and more perpendicular to the panels). I'll probably go with 10 too. As I've only populated 7 days of data so far, the number of days hasn't come into play yet.

bullit_dodger

Yup, I was thinking the same about the factor changing as the sun changes the "efficiency" of the panels as it gets higher and higher, or in my case with east/west orientation that I have...

Because that change happens in a sinusoidal waveform, around now (mid march) is where you probably want it down around 10 days, as the daily change is highest......but in June and Dec where the change is minimal we could probably shift that out to 20 days. Shouldn't really matter too much, but it's worth keeping in mind that right now in March is where it's changing at the fastest rate.

bullit_dodger

Just fyi - forecast this morning at 6am was 1407, which equates to a forecast of 6.15Kwhr for me. I'll report in end of day today with the actual

connesha

Forecast at 1:45AM was 1031 for me, giving an estimate of 9.724 kwh.

Generation has been fairly poor for the day so far (3kwh so far), but looks to be brightening up now, so there's still time to try to approach the level...

Since 9.7kwh is less than my expected usage for the day, the charging from grid was turned on and filled the battery overnight

graememk

https://www.boards.ie/discussion/comment/118765105#Comment_118765105

More or less close of business today.. I'm at 8.1kwh

bullit_dodger

Well, forecast was 6.2kwhr, and I rocked in with 10.1Kwhr. So it was a good bit off today. However, I do think that this was more that the raw data going in (the forecast) wasn't great over the model itself. I think the forecasting model is fine, but if the input is off.....Well, as you know how the saying goes..."garbage in - garbage out".

Cleared up pretty nicely weather wise from 1:30pm onwards today. Charting the differences between the forecast and the actuals observed, I'm sort of seeing a trend, although it's a bit early to tell for sure.

A lot of the times I seem to be generating 1.5-2.0Kwhr more than the forecast, so I may have to have some kinds of multiplier.

I do feel though that while today was a bit off, that there's a algorithm there to give you a decent forecast for 6 out of 7 days of the week.

graememk

I think the met.ie data is one of the best sources we have.

And a few hours of sun in the afternoon can make a bad day into a good day.

Would the "surprise" good day skew your prediction much going forward?

bullit_dodger

It shouldn't skew it mate. In fact if anything with the way that it's setup by taking the moving point averages from both the forecast and the forecasts, given enough time, it would sort of "self learn" and adjust itself, as the moving averages incorporate more data and dump out the older less accurate forecasts.

One thing to remember of course is that while weather forecasting has come on a lot, it's still not an exact science. While I am pulling for my lat/long, Met Eireann work on "grid cells" and if you happen to fall into an adjacent cell, you might have slightly a different forecast. Normally adjacent cells are 95% the same, but if you were near the coast or a mountain, you may see a bigger change. Shouldn't really matter to be fair, but I'll mention it.

Still I've been pretty happy with the forecasts so far to be honest. Yeah, today was a bit off, but generally if we can get it within 2-3Kwhr, that's kind of a win I think. it'll allow us to set charge levels and/or move loads.

graememk

https://www.boards.ie/discussion/comment/118770092#Comment_118770092

Yep I agree it's close enough.

And I'm shocked on how close I'm getting by putting in the m2 of my panels * total irradiance * 0.18 😆

As you have the data, care to humour me? Try it on yours and see how a higher angle treats it.

Where I would gain in the mid day I'd lose out in the morning/evening. Higher angle won't do as much during the day but more in the morning and evening

bullit_dodger

Well, my panels are 1840mm x 1088mm = 1.84m x 1.08m = 1.98m^2 and I've 14 of them, so that's 27.72 m^2.

So if I take today's forecast of 1407, that's

27.72 x 1407 x 0.18 = 7020 = 7.02 Kwhr

and my other model came to 6.15Kwhr. I just think the raw forecast was "off" for me today. That's ok, just have to accept that it's not a perfect value everyday. It's a fun challenge.....

connesha

Yesterday was another good prediction. Estimate 9.7kwh. Actual 9.2kwh.

Today will come in under the estimate of 26.4kwh. It'll probably finish up somewhere around 20 or 21kwh by the end of the day.

Automatically setting the charge rate based on this has been successful so far.

For yesterday, given the estimate was 9.7kwh, it fully charged the battery overnight.

For today, given the estimate was large, it didn't need to charge to get through the day. However, the script has another check: it sees what the battery level is at 1:45AM, predicts what the level it will have depleted to by 8am, and does a small topup if necessary such that for breakfast the battery is 30% soc (no point having completely empty battery at breakfast time, when will have some usage then). That's why you can see a small charge this morning.

connesha

I have the feeling that the ratio of radiation to generation won't be linear for all radiation levels. And for sure when the inverter clips at its max 6kw, this will throw off the ratio. While it won't be an issue for the days that it does clip, as these days will normally have so much generation that it won't matter if the estimate is a good bit out, it will have an effect on the calculated estimate_factor, and therefore may skew the estimates for lower radiation dates.

Am thinking of updating the way estimate_factor is calculated to basically use days with a more similar radiation to today. For instance:

Query the db for the last 14 days. From those 14 days, take the 7 days that had a radiation level most similar to todays radiation. Calculate the estimate_factor using these 7 days only.

At least this continues to be self adjusting over the year, no intervention needed, and uses "recent days like today" to calculate...

Thoughts?

graememk

I forgot to declare what my prediction was for today.. and i dont have it recorded lol but it was 20 kwh.. i think.

Got 21kwh. Its been close enough for me.. to not worry about sun position and angle of sun :P

17.9 for tomorrow, and 19 for the next day.

https://www.boards.ie/discussion/comment/118770533#Comment_118770533

I dare say your way is more accurate, but requires historical data. Would love to see how my quick and dirty calculation performs on different aspects too

connesha

@graememk my panels are 1,029mm * 1,855mm = 1.91m^2.

12 panels east, 12 west = 45.84m^2 total

Roof pitch is approx 20deg.

Panels are 390W, 20.43% efficient

For today, 45.84 * 2822 * 0.18 = 23.28kwh

Actual generation for today was 19.5kwh

bullit_dodger

So another one today which was a fair chunk off. I'd a figure of 2891 forecasted by Met Eireann which should have yielded 13.3kwhr. Sadly not even close. Actuals today was 8.2Kwhr.

It was real nice this morning, blue skies until 11-12am but then there was a massive rain storm that came through when the six nations game was on about 4pm. I was giving this a bit of thought, you see my East array is only 1.5kwp and my West Array is 3.8Kwp.

You can see the nice smooth(ish) line on the east array up until 11am. If my East and west arrays were reversed, I'd say I'd have been closer to the forecast as it was very much a chalk and cheese difference between morning and afternoon, and when the big arrary would be yielding it's max.....it was lashing rain.

It's a bit of a challenge that, but I might be able to code up a solution (going the hourly route and then summing it up) - have to think about it.

By the way, tomorrow (so far) looks awesome and setup for best day of the year. Which is odd as it's forecasting rain for Dublin. (3660 at the moment, which equates to 16.1Kwhr for me - I'm skeptical :-)

connesha

Last 10 days, incl today, since started forecasting.

Past 2 days were less accurate.

connesha

@graememk , I can easily add another line into the graph for your calc with my data if that's any use to you?

bullit_dodger

Actually that chart you have there connesha is something I should have thought of myself.....here's my own actual verses forecast. Shows it much better this way.

I'm pretty happy with that to be fair. Yeah, the 12th (yesterday) I was a bit out, as per the reasons I gave above that the "good" sunshine was in the morning on my smaller array and then the wet afternoon was on the west array. Had I a 50-50 split, I reckon I'd have been closer to the forecast. Course summing the radiation forecast into 1x figure for the day doesn't take "time of day" into consideration - but overall, I think it's good enough to use it in decision making.

Aside: I worked out the money side before for optimizing your batteries SOC off a forecast. You might (might) save about a fiver in march. Outside of this in April/May most people won't charge batteries of night rate anyway, but this isn't why I do it. I do it cause I like to see the fun in getting it to work, not to save a fiver:-)

connesha

Very nice!

Yea, the money side may not always add up huge, but it is a very interesting project. Same motivation here.

My ultimate goal is to see how close I can get to using only generated electricity, or 5 cent grid electricity (2am to 5am EV plan), for the full 12 months of the year. Where as much of the generated electricity as possible is used. 20kwh DIY battery cells ordered a couple of weeks ago to support this effort... While these forecasts dont make a huge difference for me now, with current 5kwh battery (4kwh usable), the same scripts figuring out whether to charge 3wkh or 12kwh at night into the larger battery will definitely make a difference (as will automatically triggering an Eddi boost at night based on forecasts, which is also on the list...) I'm guessing all this together could mean some financial savings, but haven't calculated any of the financials out at all. (I'd still be doing it for fun whether it makes financial sense or not)

bullit_dodger

(I'd still be doing it for fun whether it makes financial sense or not)

That's admirable. Too many people only do stuff because there's a "fiscal payback", when in truth there's value in the actual doing. The experimentation, the thinking, the joy in getting something to work etc The DIY batteries are a good example of that. I'm on the fence myself on that one, but ultimately would like to do something similar to you and get myself 20Kwhr for €4-5K and keep my existing 8.2kwhr operational somehow. Can be tricky to get two storage systems to run, but it's doable I believe. So I'm curious to see how yours works out. DrPhil did a great job on his 6 months ago.

Why so much storage? Well similar to yourself I'd like to get to 100% self-consumption. Being able to "ride through" a bad day in March is part of that. Longer term I will have an EV , as will we all ......and I have no issues in charging an EV 10-15Kwhr from a house battery, if the car was elsewhere for the daytime. I get there's losses etc. So that's the plan. Course, longer term the EV's might be the store if they standardize on a protocol, we'll see.

Winter - that's just not possible to so 100% self consumption with solar unless your making hydrogen or something to do longer term storage in the summer. Not so sure my neighbors would like the Hindenburg out my shed in the back yard. LOL

connesha

Yea winter will be challenging some days. But what I mean is that the sum of charging a big battery at night, plus the miserable production for the day, should get through a lot of days and evenings

I was on the same tarrif a few weeks before getting solar, so have some data on how much, and at what time, we're consuming, and how habits can sway that. During that period, I set just about everything that had a timer on it to run between 2am and 5am: Immersion, dishwasher, washing machine, (dryer sometimes, although ours is a bit too loud for nighttime, and the dehumidifier does most of our clothes drying very economically anyways). It wasnt any extra effort, and TBH actually found it more convenient for some appliances to run at night - like having the default 3h dishwasher program be done when we get up, and having the hot water boiling in the morning, besides needing to wait for a bit of sun to get from warm to hot. The appliance start times were staggered a bit, and some appliances use the heavier load earlier in the cycle

By doing this, and not trying to reduce other load, we averaged in the region of 40% of consumption coming in between 2am and 5am (out of some 16kwh or 18kwh per 24 hours total).

This is all based on the smart meter readings, 30 min granularity, downloaded from the bordgais website.

So adding that all up: having e.g. 12kwh usable available in the battery, maybe generate 2 or 3kwh, taking off X% for the conversion inefficiencies, and its right in the ballpark.

Of course need to watch that don't overload the main fuse, and how hard want to drive the battery. Charging at the max 100a would get it almost full at night in 3 hours, so that'd bring the battery to 15kwh+ usable. I guess can see how hard the panels drive it during the summer, and if they put 100a into it for a few hours, I don't see why not at night time. I know its advisable to go easier on them for prolonging their life, but I'm learning towards "I have it, so I'm going to use it as much as I need it"

Well, that's a lot of theory... the proof will have to wait...

connesha

Todays estimate in chart below.

Note, the algorithm to forecast was updated before this estimate along the lines of what I posted a couple of days ago, to "use days more similar to today". I found the previous model was over-influenced by 1 very good/bad day, and "think" its probably useful to try to compare more like-to-like days.

Since last night it is:

• Get the last 12 days data.
• From that 12, take the 7 days where the radiation is most similar to today
• Sum the radiation for those 7 days, sum the actual_wh
• estimate_factor = sum_of_actual_wh / sum_of_radiation

12 and 7 are not really based on anything more scientific than having enough data to average, and not going back too far.

I believe this is somewhat similar to yours bullit_dodger, with the addition of the "days like today"?

bullit_dodger

I like the "days like today" modification. A rough filter would be say +/- 1000 away from the Met Eireann forecast. I'll run that algorithm through my histroical data and see what it looks like to me.

E.g if Met Eireann is saying 2850, then you will accept previous days where forecast was in the 1850 -> 3850 range.

I do know that sadly in my own case that wouldn't have helped me for last Saturday estimate, as it was more a case of "when in the day" the solar radiation fell rather than how much. I think the forecast from Met Eireann was sort of ok, but the timing screwed me up.

Looking at your chart above though, you're (for the most part) getting pretty nice forecasts. The main outliers being March 7th and March 12th. Have you pulled the daily historical charts for those and seen if there's anything similar to what I noticed on Saturday?

graememk

hows tomorrows forecast looking?

Mines not great at all! although 24kwh for wed.

connesha

Its this where you pull the historical data?

https://www.met.ie/climate/available-data/historical-data

I'm struggling to find March GlobalRadiation numbers at any stations. Local stations don't seem to have historic GlobalRadiation, and even the airport only has for Feb and earlier. What station do you use?

Todays final here BTW:

Too early to say if todays estimate calculation change is a success, or if today was just lucky... I guess will see in another week or 2...

Tomorrow is currently looking exactly the same as today. Thursday onwards look to be picking up a lot. But each forecast is locked in in the middle of each night, so will change ...

Separately, was thinking about how to make more use of older data; specifically, we rely on ratios between radiation and actual generation to make an estimate, but are only using a small set of recent days to predict. So, toying with using the suns angle compared to panel angle today vs each older_date. Maybe the Zenith Angle at Solar Noon compared to panel angle on both days, which may allow an adjustment to be made for each older day to make it usable.

Maybe this could allow for going back further in time to use a bigger data set to average? (Or maybe its just not worth it, for one, because the met data can change 10 minutes after we use it...)

Not fully formed yet, but am currently logging out the Time of SolarNoon and its Zenith Angle for a range of days. If that data, or the script that produced it sparks anything, I can send it on.

It may then allow us to really zone in on using previous days with similar radiation. For instance, get the last 20 days that are within 250 radiation of today, even if some of them are 2 months old. Adjust the each older one for sun angles, and get your estimate from that.

Currently, I'm prioritising recent days over similar days, in that its only looking back 12 days, and takes the most-similar ones from that (even though they could all be 2000 radiation out, for example)

bullit_dodger

So the historic data I've stored for my latt/long going back to last August when I setup that VM in Azure to host the tool. I did it that way as I wanted it to poll every 1 hr and stay on 24x7. As a result I've not been looking at the Met Eireann history site as i have most of what I want already. (I actually didn't know it existed - so thanks)

That said, since I have all the data myself.... I extracted back towards the start of Jan and computed what the forecasts would have been with the same algorithm.

That's a decent enough correlation to be fair.

I've been giving some thought connesha to the idea that you have of the similar days (but outside of the window) verses dis-similar days (but inside the window). I think you actually need the dissimilar ones as ultimately what your doing with the 10 point moving average is to "smooth" the radiation to actuals ratio.

I'm speculating: You panels will behave differently with different angles (esp with not direct sunlight). I'm not sure that say 1000 radiation on a bright day in Jan, would be the same as 1000 radiation on a murky day in March, even taking the sun's angle to the panels into consideration. The panels generate some power from diffuse light, and or inverter inefficiencies would be different between Jan and (say) April. I've zero data to support that theory by the way, just speculating. So I think taking the last 10-14 days and computing off that (dissimilar and all) gives you a more rounded figure to base your forecast off.