Forecasting tomorrow's sunshine (to better use the solar)

connesha · 14-03-2022 10:11PM

That graph is awesome.

Which exact estimation formula is the graph?

The +/- 1000 you mentioned in previous post?
Or
The 7 closest radiations out of last 12 days?

bullit_dodger · 15-03-2022 12:53AM

No, it's original algorithm where I sum up the radiation for the past 10 days and get the average. Then work out the similar 10 day moving average for the "actuals", work out what the Kwhr/1000 units of solar is.......and then compute the forecast based off the forecast for the day. No filtering.

Tomorrow if I get time, I'll have a crack at writing some code to do the other algorithms you mentioned above such as filtering out +/- 1000 figures along with the closest 7 in the last 12-14 days.

I do think they will yield different results. You may even see some days becoming more exact along with other days on the same chart diverging more. Should be able to chart them and then compare the deltas in the various algorithms to the actuals and see which algorithm is more exact. Course I realize that even with the data points that I have, I'm still on the low side. You'd need 100's (1000's) of points to see which algorithm historically works best.

But, I think if you get it "fairly right" 6 out of 7 days a week, then that's a win.....and it's kinds fun for a geek like me :-)

connesha · 15-03-2022 11:38AM

Yea, I'd be interested to to see how the closest 7 in the last 12-14 days looks across your data-set

Agree, each way we do it will probably throw up some positives and negatives, and the weather forecast data is only a forecast after all... not an actual radiation at your exact home...

I do think we're very close now, and are certainly well into the region of having "a pretty accurate estimate, most of the time",

I did make one further change last night to the way the estimate is done. While it is live now, it won't have any impact on estimates until about a month from now.

The background:

The linear ratio we assume between radiation and generation will likely be problematic for some setups, especially in summer. Taking my own setup as an example to illustrate:

9.4kwp, split evenly East-West, into a 6kw inverter, on a low pitch roof (around 20 deg). Because of the low pitch, both strings can generate a lot when the sun is high, and I'm expecting clipping on good summer days (even this week I saw a few peaks of 6kw production).

If the inverter is greater than or equal to what panels will practically produce, then no problem. But if the inverter is less, then there is a cap, and the radiation to generation ratio will no longer be linear.

So, if have a bunch of recent days that were clipped, and now want to make an estimate using todays low radiation, then todays estimate will be off. Using our original algorithm of taking the last 10 days, this will be a problem. The 7 out of 12 was trying to help, and should be a little better (considering it will exclude the more severely clipped ones), but I think we can do better again.

The change:

Conveniently, this effect will happen mostly when the sun angle is not changing so much day-to-day, so, perhaps we can use a wider window to take our previous days data from.

The algorithm now no longer hard-codes 12 as the number of days to go back (the window).

Instead, it walks backwards from today, adding up the difference in noon sun angles per day, and when the difference reaches 4.7 deg, that's how many days it goes back.

Why 4.7? Right now is the time of the year with greatest rate of change. 12 days back from today is a 4.7deg difference, so that's the max allowed.

Come April 20th, the window will rise to 13 days, and by full summer it would be over a couple of months wide if left unbounded. (I've imposed an upper cap of 21 days, as there are potentially other factors besides just angle)

By choosing 7 out of 21 previous days, it'll be able to avoid more dissimilar days, and in particular the low radiation days will be less influenced by the high radiation (clipped) days, and vice-versa.

So, in summary:

Summer (and mid-winter): pick the 7 days with the closest radiation out of the last 21 days
Spring/Autumn (more changeable sun angle): pick the 7 days with the closest radiation out of the last 12 days
And scale between those

Todays forecast 19.985 kwh

bullit_dodger · 15-03-2022 01:36PM

Ahh, that's a good point about the inverter clipping and it creating a "false ceiling" on the actuals. Hadn't thought of that. For me, it's not going to be an issue. I've a 5Kw inverter and I have (currently) 5.3 Kwp in panels, but they are split in an east/west config so it's unlikely that I'll ever get to max inverter load. Now I do have some tentative plans to add 2x more panels to the east array, bringing me up to 6.1Kwp, so when that happens there's a slight chance that I might hit that 5Kw limit, but it'll be rare I reckon.

The one good thing that we have going for us in this situation is that for the actual day in question where you have clipping, your production is going to be well above any decision making that you will be doing with setting battery charging levels or whatever. If you forecast is 40Kwhr and you "only" get 35kwhr due to inverter clipping.....I think we'll still be happy :-) Maybe not with the forecast, but with the production. Your right though, if that is then built into the averages going forward, it would take a few days for it to filter out of the history , so it would negatively affect your forecasts until 10-12 days or however many you have in the look back range is passed.

I like that idea you have about the 4.7 degrees change. That's quite clever. Wish I'd thought of that! Pretty slick......so now just need to come up with a way of computing the current date window. Should be able to work it out with sine/cosine and then the date difference from March 22nd/Sept21st.

graememk · 15-03-2022 01:49PM

And you were saying I'm overthinking it looking to calculate the angle of the sun and panels!

Although what I'm running with is close enough though. I think I can calculate the sun position for any given hour, just have to port it from excel.

I'll see if the mood take me there

connesha · 15-03-2022 02:13PM

😁 It all seems so simple before we start... and then get sucked in bit by bit...

bullit_dodger · 15-03-2022 02:16PM

https://www.boards.ie/discussion/comment/118787995#Comment_118787995

LOL - yeah, it's easy to get sucked into stuff mate. To be fair though, it's a lot easier to work out a date window than it is to compute the hourly sun angles from the Met Eireann forecast....twice (as I have two arrays east & west) :-)

connesha · 15-03-2022 02:26PM

This'll do it.

import pytz
from utils.utils import Utils
from astropy.coordinates import get_sun, AltAz, EarthLocation
from astropy.time import Time
from suntime import Sun

class SunAngles:

  def __init__(self, longitude, latitude):
     self.longitude = longitude
     self.latitude = latitude
     return

  def get_bounded_num_days_back_within_angle(self, angle, lower_limit, upper_limit):
     num_days = 0

     location = EarthLocation.from_geodetic(self.longitude, self.latitude)
     sun = Sun(self.latitude, self.longitude)

     solar_noon_time_today = self.get_solar_noon_for_date(sun, Utils.date_today())
     zen_angle_today = self.zen_angle_at_date(solar_noon_time_today, location)

     angle_sum = 0
     last_angle = zen_angle_today
     while num_days <= upper_limit:
        date = Utils.date_from_offset(-(num_days + 1))
        solar_noon_time = self.get_solar_noon_for_date(sun, date)
        zen_angle = self.zen_angle_at_date(solar_noon_time, location)
        diff_from_last = abs(last_angle - zen_angle)
        if angle_sum + diff_from_last > angle:
           break
        angle_sum += abs(last_angle - zen_angle)
        last_angle = zen_angle
        num_days += 1

     num_days = Utils.clamp_value(num_days, lower_limit, upper_limit)
     return num_days

  def get_solar_noon_for_date(self, sun, date):
     tz_ireland = pytz.timezone('Europe/Dublin')
     sunrise_time = sun.get_sunrise_time(date)
     sunset_time = sun.get_sunset_time(date)
     sunrise_time_local_tz = sunrise_time.astimezone(tz_ireland)
     sunset_time_local_tz = sunset_time.astimezone(tz_ireland)

     # midpoint is technically not 100% accurate for SolarNoon, but it's very close, and plenty good enough for this purpose
     solar_noon = sunrise_time_local_tz + (sunset_time_local_tz - sunrise_time_local_tz) / 2
     return solar_noon


  def zen_angle_at_date(self, date_time, location):
     altaz = AltAz(obstime=date_time, location=location)
     astropy_time = Time(date_time)
     zen_ang = get_sun(astropy_time).transform_to(altaz).zen
     degrees = zen_ang.degree
     return degrees

ESTIMATE_NUM_DAYS_HISTORY_MIN = 12
ESTIMATE_NUM_DAYS_HISTORY_MAX = 21
ESTIMATE_NUM_DAYS_USE = 7
ESTIMATE_SUN_ANGLE_MAX = 4.7

  sun_angles = SunAngles(config.LONGITUDE, config.LATITUDE)
  num_days_history = sun_angles.get_bounded_num_days_back_within_angle(ESTIMATE_SUN_ANGLE_MAX, ESTIMATE_NUM_DAYS_HISTORY_MIN, ESTIMATE_NUM_DAYS_HISTORY_MAX)

import datetime

class Utils:

  def __init__(self):
     return

  @staticmethod
  def date_from_offset(offset=0):
     return datetime.date.today() + datetime.timedelta(days=offset)

  @staticmethod
  def date_today():
     return datetime.date.today()

  @staticmethod
  def date_tomorrow():
     return Utils.date_from_offset(1)

  @staticmethod
  def date_yesterday():
     return Utils.date_from_offset(-1)

  @staticmethod
  def clamp_value(value, min_value, max_value):
     if value > max_value:
        return max_value
     elif value < min_value:
        return min_value
     else:
        return value

connesha · 16-03-2022 08:49PM

Happy St Patricks Day for tomorrow all! ☘️

Todays graph below. The last 3 days are when the "7 out of 12" change was made, and I've managed to stop tinkering with the algorithm since (which was no small feat!)

I think I'm at the point of not changing it for a while now, let it run, and see how it goes for a few weeks/months... Its been a lot fun getting here (can we call our predictors Alpha stage?), definitely learned a lot, and lots of good ideas with @bullit_dodger and @graememk especially.

Would be kinda interesting to see how the predictors that come with home assistant fare compared to these over time. Not sure if anyone watching here has past data, or is gathering future data on these. I hope we haven't wasted our time ;-) .. I don't think so.

And if anything from mine is of any use to anyone (theory/logic/code/mistakes), just give me a shout.

Time to move on to the next mini-project. Seriously, since got solar just a few weeks ago, my time has vanished, and the todo list is falling off the end of the page.

Now, to try to figure out home assistant... Installed it a couple of weeks back, spent a couple of evenings playing with it, but can't for the life of me figure out how it hangs together or how to do stuff with it... maybe it'll click eventually.

As first mini-integration, may try to get a home assistant screen/button/toggle/UI to apply an "away mode" to the script. Something simple, e.g. when HA sets AwayMode, the script reduces its daily_predicted_electricity_usage to a flat BASE_LOAD x 24. So then it won't charge at all at night time, if the forecast is even giving 5 or 6kwh (this last part should work as is once the predicted usage is reduced). Anyway, that's a different one...

graememk · 16-03-2022 10:38PM

That's my comparison, top 2 is forecast.solar, which usually is optimistic to say the least.

My met prediction is close, within a kwh or 2.

Will be interesting how temperature will come into play.

Eg in may and June my inverter clips.

But on the hot days in Aug it doesn't clip.

championc · 18-03-2022 10:38AM

Maybe time to start incorporating some automation type intelligence into your setup ?

Nelbert · 18-03-2022 12:37PM

I'm seeing familiar style graphs here. I've just purchased an energy monitor and incorporated it in to home assistant.... Background elec usage is about 5-600 watts at any given time but anytime the heat pump kicks in it jumps a huge amount.

Has anyone used the solar forecast integration in home assistant (familiar graphs so thinking there is some HA users)? If so how accurate are the forecasts Vs your production? I'm trying to gauge it's usefulness as a tool to work out potential impact.

championc · 18-03-2022 01:24PM

https://www.boards.ie/discussion/comment/118801324#Comment_118801324

The use of the forecast for me is really to gauge the level of night time charging I need to do, to ensure that I get to 23:00 that evening without needing to import any daytime units, while also not over-charging and ending up exporting to the grid.

From when the clocks go forward until the end of BST, I really don't see the need to be worrying about night charging, except when a really crap day is forecast.

But as for the most accurate source of data, I really haven't got to the point of really looking into things in an absolute way

mp3guy · 18-03-2022 01:43PM

https://www.boards.ie/discussion/comment/118801324#Comment_118801324

I find solcast much more accurate than the service built into HA's energy management system. Been using it for over a year now to do load shifting like championc mentioned and also decide on the method for heating hot water.

System · 18-03-2022 02:09PM

This discussion was created from comments split from: Daily pv production.

Jonathan · 18-03-2022 02:10PM

https://www.boards.ie/discussion/comment/118801324#Comment_118801324

Merged your comments into this thread where they are better suited.

connesha · 08-04-2022 11:23AM

As its past end of month, here's an update on how the met.ie forecasts are working out following the changes we talked through a few weeks back: using 7 of the last 12-day-window with similar radiation, and adjusting the window based on the sun angle (the sun angle hasn't kicked in yet; the "12d" in the chart shows the current window size)

And using these forecasts allowed the script to charge the battery automatically. I'm finding the readings for import as seen by the inverter differs somewhat from the actual. At the end of the day its what the meter/supplier sees that matters, so am using the monthly bill to judge success.

The script uses the EV period (2am to 5am) only for charging, i.e. its purpose is to avoid grid usage at all other times. So it will use this EV period to charge for (1) a poor day of generation, or (2) if the battery has gone low by 2am and needs small charge to get through to generation time in the morning.

There is about a 1kwh/day "leakage" in my system: while the inverter is trying to match house load on the fly, it misses it by approx 1kwh imported per day. So, this 1kwh per day spread across the non-EV period seems unavoidable.

March has been a great month.

This is with a 5kwh PureDrive battery. A bigger battery (20kwh coming soon I hope) would have reduced the EV period usage significantly for months like this past month, probably to approaching 3kwh. Although that wouldn't save much this month, it should have a decent effect on winter months: i.e. only EV period usage should rise in winter, and the usage in the more expensive times of the day should remain much the same. Lets see...

SD_DRACULA · 08-04-2022 11:37AM

I wonder if the EV tariff will still be worth it with the considerable hikes coming in one week time

+100 to the standing charge and quite a bit on the others too, even with full discount.

Peak rate is a 🤬 joke.

connesha · 08-04-2022 11:40AM

Yea, wondering the same...

My guess is that if you have tight enough automation to really put almost everything into the EV period, then it probably is. But I haven't done the calcs on it yet...

Are the other non-EV smart tariff standing charges rising by 100e also?

Jonathan · 08-04-2022 01:20PM

https://www.boards.ie/discussion/comment/118897717#Comment_118897717

Looks like it is working out well for you @connesha. Have you any code you can share with others on Github etc? Might help someone get up and running quicker.

SD_DRACULA · 08-04-2022 01:21PM

About 70 odd. My plan was to use this tariff alongside a 20kwh battery for winter mainly and load shift as much as I can but coming up to €500 now for basically no usage at all if you include the gov charge.

Anyone come across any other similar tariff with better rates?

connesha · 09-04-2022 01:11PM

https://www.boards.ie/discussion/comment/118898290#Comment_118898290

Sure, will do.

Will need to see what the easiest way to share it will be. There are quite a few things there in total, and most won't be of interest... will probably confuse more than help. I may just write it up here later this week, and post some of the key code, that should be easier for anyone getting started. In the mean-time, if anyone wants anything, just shout.

connesha · 10-04-2022 06:01PM

https://www.boards.ie/discussion/comment/118902568#Comment_118902568

Code for forecasting, battery charging and a bunch of other random stuff is posted over on the Automation thread now. #275

This is the output of all my ramblings on this thread several weeks ago. Hope its useful to someone...

graememk · 10-04-2022 06:16PM

Ive found a nodeRed Node that will give me the sun position now and I think ive got my head around the cosine calculations.

Just need to implement them now in javascript.

That being said, my rough and ready prediction for flat panels is working well. But I would like it to be more versatile.

connesha · 10-04-2022 06:30PM

https://www.boards.ie/discussion/comment/118907652#Comment_118907652

If its of any use to you, I dumped the angles to Excel a month back and still have it. Was just for eye-balling a the time. Let me know if you want. Covers that angle at Solar Noon for each day from 27/08/2021 to 29/09/2022. (the lib I used only went so far into the future, so I went back a bit as well as forward to cover over a full year)

graememk · 10-04-2022 06:36PM

https://www.boards.ie/discussion/comment/118907742#Comment_118907742

I found this, https://flows.nodered.org/node/node-red-contrib-sunpos I just push a datetime into it and it spits out the exact position for the sun at the set location & time

SD_DRACULA · 13-04-2022 09:53AM

Guys is there an easy way to hook this into home assistant?

Forecast.solar is overestimating by a lot on the dark days.

Jonathan · 13-04-2022 10:58AM

@connesha Out of interest, what was your prediction vs actual generation for yesterday?

Yesterday morning Forecast.Solar and Solcast were predicting 10.4kWh and 9.9kWh respectively. A really dull day where I am in Fingal and ended up with only 1.9kWh for the day.

SD_DRACULA · 13-04-2022 11:41AM

https://www.boards.ie/discussion/comment/118920278#Comment_118920278

Same here predicted 15kwh and real was 3.5kwh

connesha · 13-04-2022 01:53PM

Yesterday was a real outlier for me too. Predicted 13kwh, actual 5.5kwh. Biggest deviation I since started doing this.

My met.ie weather forecast was locked in at 1:45 AM, and at that point met.ie was showing radiation in the middle of the day at around 170 to 190 per hour. (The solar generation forecast was based around those levels)

I manually looked at the met.ie weather data at around 10 am yesterday, just out of interest, and they had changed to show levels of 80 to 90 radiation for the same period.

So, I think yesterday was just one of the days where the weather forecast changed significantly over the course of the morning. Don't think there's really anything can do with days like this, given that you got to make a charge-decision in the middle of the night... I guess the up-side is that they these days are outliers...

Forecasting tomorrow's sunshine (to better use the solar)

Comments