Advertisement
If you have a new account but are having problems posting or verifying your account, please email Niamh on [email protected] for help. Thanks :)
New AMA with a US police officer (he's back!). You can ask your questions here

Solar Monitoring + Home Automation (My Setup)

  • 05-03-2021 1:19am
    #1
    Registered Users Posts: 2,744 ✭✭✭ graememk


    I've talked about my monitoring system and I've said I'd do a post on it that I can refer back to/ update as things change

    Background
    I monitor 2 systems, my parents solar and my own. We are neighbours and the 2 houses are networked together. Up in the north west.

    Parents Array - 5.6kwp, SE, No battery, HW divert, Low angle (15°).
    CT on grid, solar and immersion.

    Mine - 7.2kwp, SW/NE 50:50 Low angle (12°), 10kwh DIY battery
    CT on grid and battery, Solar is currently pulled via a serial connection.

    Both are on steel roofs, made the install easier. Worked with a local installer for both.
    Using OpenEnergyMonitor hardware and their emon software

    Hardware:
    Everything is ran from a raspberry pi running the EmonHub(EmonCMS) SD card image, Really easy to setup and get running. This also runs Node Red.

    Actual monitoring is done with 2 EmonTx's but the wifi version of it.
    Needs 2 power inputs, one to power the system(5v usb) and the second is a 9v AC/AC adapter, for a voltage reference to be able to measure accurately.
    emontx has 4 ct inputs and a few other sensors can be connected to it.


    These are configured to send their data over MQTT (a way of devices to communicate) back to the emonHub, (which by default runs a mqtt broker)
    Both systems are run from the same pi.

    This raspberrypi also connects to my sofar me3000sp to pull data from that. - currently I'm using that to also monitor my solar generation, I still need to order another ct for that task. It makes some of my graphs a little bit fuzzy if there is scattered clouds.

    My EVSE is an EmonEVSE, It can do solar divert, It just needs a MQTT feed of the available excess amount and has smoothing algorithms to try and account for less than perfect days!

    I have a solid fuel stove that has a radiator feed and a separate heating coil for the hot water tank. (2 pipes in and 2 out). This is monitored by a second raspberry pi. I have the temperatures of the flow and return of all the pipes.
    These are logged every 10 seconds and sent over mqtt to the emonhub.
    It can also control the pumps on the system - from a script in node red & Mqtt.

    Smartplugs
    I have got some Ener-J smart plugs that also have power monitoring.
    These particular ones are based on an ESP chip too, and are easily reflashed to run custom firmware - Tasmota, This can then be monitored and controlled by MQTT. - Sidenote, a lot of the wifi smart devices are made by Tuya, - they can supply the products, you just have to do the branding and marketing. Some use ESP chips, some do not.

    Automation

    I run my automation from node red. Its a flow based development tool, and the functions in it are javascript. You can import nodes to interface with Api's, pull stuff from MQTT, etc etc

    Things I am doing now are:
    Office smart plug:
    We have an outside office, Its insulated but a bit of heat in it is good for it. - This is connected to my parents house. Have an oil heater, set to minimum, 800w
    I have a flow that takes the Excess value from the solar (excluding HW divert) and if it is greater than 800w, turn on the radiator. It does that by sending an MQTT command addressed to the relevant topic. The HW divert takes whatever is left.
    Plan is to do something similar with a heated towel rack.


    Modbus Serial connection: Did not write this one but this pulls a lot of information over rs485 from the me3000sp and pushes it out over mqtt.

    Excess Value for EVSE.
    I am monitoring the battery SOC, how much is going into the battery and the current excess value.

    I have it set if my battery is greater than 90% it will start to send out the full excess value to my EmonEVSE, excluding any battery charging.
    I can adjust this to still send some to the battery or the battery can take whats left over. I don't have to wait until the battery is completely full and exporting until putting something into the car.

    My solar prediction flow
    Been poking around the met.ie api, and in what you can pull via an xml file, they have a solar radiation value. a bit of processing to pull out what i want from it and now at a glance i can see if tomorrow is going to be a Hit, Miss or Maybe!

    There is an api using bluelinky for the Kia Uvo app, Need to look into that more. There is a demand shaper, in the EmonCMS
    When configured correctly, you could have it putting excess solar into the car but still making sure you have say 50% charge in the car every morning.

    Remote Access:
    So everything is stored locally, except for the stuff i chose to send to the emoncms.org website. This is not free, its about 1 pound per feed per year, and you get credit towards it when you buy from their shop. Not sure what I'm going to do when I run out of free credit
    My monitor uses 11 feeds to run the graphs.

    Recent addition
    Tado, smart thermostats and Radiator valves. Still configuring it and Id say its a no brainer for someone with a simpler system (ie straight oil/gas) - sounds like you have little robots about the house.

    I need to figure out a way to be able to not let it turn on the oil when there is enough heat coming from the stove. Have a few ways in mind, Most likely will be using the Raspberry pi that's there.

    I think thats all.. so far anyway!


«1

Comments

  • Registered Users Posts: 3,913 ✭✭✭ mp3guy


    That's some serious DIY. I do similar stuff but with home-assistant and regular old REST calls via Python, plus some MODBUS over TCP.

    Nice to have it all automated like this, but I think this is way over the heads of a lot of folks who'd be getting solar! Ultimately the black box products that do this for you cost more because you don't have to know how to do it (or maintain it) yourself.

    Did you look at home-assistant before you started?

    How are things backed up? You mentioned a few RPis, if they're on SD cards they're going to die within a couple years. This is the thing that I hate about DIY IoT stuff like this, stuff just randomly breaks and you have to fix it yourself. I'm running a single i7 NUC with an M.2 drive that has nightly differential backups on all my custom scripting so it'll be as painless as possible when it inevitably dies.

    How many hours would you say you've honestly put into this monitoring and automation setup?


  • Moderators, Society & Culture Moderators Posts: 31,304 Mod ✭✭✭✭ Gumbo


    I have Alexa she can boost my heating.

    No idea what the OP is posting about but known enough that it’s beyond my capabilities:D


  • Moderators, Sports Moderators Posts: 16,099 Mod ✭✭✭✭ slave1


    Yeah, lovely to read it all but whoooosshhhhh


  • Registered Users Posts: 2,744 ✭✭✭ graememk


    mp3guy wrote: »
    That's some serious DIY. I do similar stuff but with home-assistant and regular old REST calls via Python, plus some MODBUS over TCP.

    Nice to have it all automated like this, but I think this is way over the heads of a lot of folks who'd be getting solar! Ultimately the black box products that do this for you cost more because you don't have to know how to do it (or maintain it) yourself.

    Did you look at home-assistant before you started?

    Home assistant I'd heard of but haven't really looked into it. I liked the emon system for a nice interface, and built for energy monitoring.

    I actually have very little of smartplugs or smart lights, Dabbled with the ikea ones, but ended up still just using the switches on the wall.

    Totally way over the top for a normal solar install. Wouldn't recommend it to anyone that isnt interested in the finer control of things, or the additional information.

    mp3guy wrote: »
    How are things backed up? You mentioned a few RPis, if they're on SD cards they're going to die within a couple years. This is the thing that I hate about DIY IoT stuff like this, stuff just randomly breaks and you have to fix it yourself. I'm running a single i7 NUC with an M.2 drive that has nightly differential backups on all my custom scripting so it'll be as painless as possible when it inevitably dies.

    Oh.. I've been burned by that before, Original Pi that ran the circulation pumps. killed the sd with no backups. Ended up sitting disconnected for a long time before I resurrected it recently. - Still have the manual pipe stats as a fail-safe.. it crashes sometimes and needs a reboot.

    I have that python script well backed up now, although nothing is saved on that now, its just reading the sensors and spewing out the data.

    As for the emonHub, I need to setup regular backups on that. It does run a low write database. But there is backups!

    flows in node red are backed up too, they don't change much now.
    mp3guy wrote: »
    How many hours would you say you've honestly put into this monitoring and automation setup?

    Too many, but its a hobby and would have been playing video games anyway!

    The actual solar monitoring end of it, Not that long, got the pi up and running one sunday afternoon. then it was just a matter of pointing the emontx's to it and configuring the feeds, so prob another evening tinkering, with youtube in the background! If the emonpi was bought off the shelf it would be an even easier setup.. provided you can get the ct's around the relevant cables

    The other stuff.. figuring out how I wanted to do it took far longer than actually doing it in node red!


  • Registered Users Posts: 227 ✭✭ slayer91


    Hi,

    Would you recommend the EMONPI, I am looking at this to monitor my house load and solar (to be installed when construction is allowed again)
    The MQTT is very handy for connecting into my Homeseer system

    Regards

    Liam


  • Advertisement
  • Registered Users Posts: 5,176 ✭✭✭ Ubbquittious


    Interesting. What sensors are you using on the flow and return pipes? I put a load of MC9700s on mine but never rigged the thing up.

    Ireland's premier property crash prediction site




  • Registered Users Posts: 2,744 ✭✭✭ graememk


    Interesting. What sensors are you using on the flow and return pipes? I put a load of MC9700s on mine but never rigged the thing up.

    I'm only measuring temperature, but it's just DS1820 temperature sensors (encapsulated ones) zip tied to the pipes with a thermal pad between it and the pipe.

    It's natively supported on the raspberry pi as a file you just have to read.


  • Registered Users Posts: 5,176 ✭✭✭ Ubbquittious


    graememk wrote: »
    I'm only measuring temperature, but it's just DS1820 temperature sensors (encapsulated ones) zip tied to the pipes with a thermal pad between it and the pipe.

    It's natively supported on the raspberry pi as a file you just have to read.




    Ah yes I came across that one before but never tried them. Do you find them good? I suppose you'd need something like that given the Pi's lack of ADC.



    Was going to go with Sensirion I2C sensors but then I'd need a little PCB for each one which is a pain

    Ireland's premier property crash prediction site




  • Registered Users Posts: 2,744 ✭✭✭ graememk


    Ah yes I came across that one before but never tried them. Do you find them good? I suppose you'd need something like that given the Pi's lack of ADC.



    Was going to go with Sensirion I2C sensors but then I'd need a little PCB for each one which is a pain

    For the temperature range I'm dealing with they are perfect, had one get a bit noisy for a while, but a full power cycle sorted it, thats the only issues I've had.

    Simple and accurate.

    Its almost as easy as just plugging it into the Rpi, but i think there is a resistor needed. would have to look it up, exact setup I've forgotten. Basically, join all the power, ground and signal wires together and off you go.

    I think they have been there since 2014, would have been used for about 2-3 years, then the SD died.. along with my code and weird database/webpage control. Revived in Sept.

    Now it just reports and acts on information from mqtt.


  • Registered Users Posts: 5,176 ✭✭✭ Ubbquittious


    What kind of stove is this that has separate outputs for the dihew and the rads? I have a stove with 2 outputs as well but it all comes from the same water jacket. Does yours have a feature to disable the rads with a lever or something like you'd find on a range?

    Ireland's premier property crash prediction site




  • Advertisement
  • Registered Users Posts: 2,744 ✭✭✭ graememk


    What kind of stove is this that has separate outputs for the dihew and the rads? I have a stove with 2 outputs as well but it all comes from the same water jacket. Does yours have a feature to disable the rads with a lever or something like you'd find on a range?

    Comes from the same water jacket. The way mine is plumbed is unconventional.
    Hot water tank has 3 coils, 1 for solar at the bottom and 2 standard ones. Normally one would be piped into the stove and the other the oil. (Have solar hw too)

    On mine both are to the stove. I have the ability of switching it back to oil without much difficulty if needed. Well, would need some plumbing but the connection is there to the oil but just capped off.

    So one pair of boiler fittings go only to a coil in DHW (I have a pump on this too) - if I force this pump on it practically prioritises the hot water. The rad feed won't get hot enough to trigger the radiators.

    And the other pair goes to the second coil and radiators.
    It comes from the stove where there is a T and that coil is done by gravity.

    After the T the circulation pump for the rad and an one way valve before it connects to the radiator lines.

    If I wanted to, I could pull heat out of the HW tank in the morning and push it around the house to give a kickstart to the heat in the morning.


  • Registered Users Posts: 791 ✭✭✭ reklamos


    Nice setup you have there.
    I have been automating for a while now all based on HomeAssistant
    Here is my setup:
    HA, nodered, esphome, influxdb, grafana and many more running on docker. I have used RaspberryPi before but it did not have grunt at that time and SD cards were not reliable.
    PV monitoring:
    I have Solis inverter and it is sending it's data over wifi to cloud. I have found online how to talk to API and wrote simple python app that is running in container. It is scraping that data and pulling it back and sending over mqtt to HA, this puts it into influxdb for longer term storage. This also allows me to use grafana to make fancy dashboard(see attached).
    I have another small 12V PV on the shed that is charging leisure battery. This one has basic monitoring(Voltage, Amps)

    House monitoring:
    Temperature and humidity is monitored in all rooms and bathrooms. Water temperature for immersion tank.
    PIR sensors and door/window sensors
    Multiple sockets have sonoff and teckin switches to monitor power usage.
    Internet traffic monitoring

    Automations:
    Automatic excess electricity diversion to heat water. I did not have to anything here :)
    All automation is done through node-red
    House heating. Originally I had single thermostat on ground floor and mechanical timer for boiler. These were replaced with sonoff minis. All rads have electronic trvs that are connected to sonoff. This allows me to control temp for each room individually.
    Immersion tank trv was also replaced with electronic one. I can make sure that the water temperature is kept at certain level when needed.
    PIR is used to detect motion and also turn on/off lights in hallways. The light level and turn on time tracks the sunset. I use zigbee bulbs.
    LED strip under cabinets in kitchen that also turn on based on PIR and time of the day.
    Door/window sensors are self explanatory, except they are also tight to heating. If there is a window opened in the room the heating for that room will be disabled.
    Extractor fans in bathrooms turn on/off based on humidity level.
    Standby devices are automatically turned off after certain amount of time.
    The dishwasher can be automatically turned on if it is sunny.
    The automation in the shed is controlling air pump for greenhouse plants depending on the amount of sun.
    Hyperion(ambilight) for TV only turns on in the evenings.
    Fish tank lighting

    Controls:
    Initially all was controled over PC or phones. Recently I have installed old tabled on the wall and that now also used for control.
    In the living room I have Aqara cube that is used to control lights and also the brightness. I did not expect much but now got really used to it.
    I can access and control all remotely via app through VPN.

    Future plans:
    Detailed battery monitoring, waiting for the parts to arrive
    Pull data directly from inverter over modbus
    Replace influxdb with Prometheus as I use it for other things and do like it more
    Automate blinds
    Smart mirror

    My house is a mixture of wifi, BLE and zigbee devices. I choose devices that can be flashed with tasmota or esphome etc. If required I do write my own code. I keep everything in house and try to avoid Cloud as much as possible, so no Alexa or Google.
    Probably there is something else that I forgot about.


  • Registered Users Posts: 3,913 ✭✭✭ mp3guy


    reklamos wrote: »
    Pull data directly from inverter over modbus

    Nice! How many hours have you put into this? How often does something go on the blink and need to be looked at? That's the killer for me, something randomly stops responding or you have to restore from backup (I presume with all this you have an automated offsite backup and restore facility?).

    I'm already pulling data from inverters over modbus, far far superior to relying on external flaky cloud APIs. Also the update rate is fantastic (order of seconds rather than minutes!).


  • Registered Users Posts: 849 ✭✭✭ tails_naf


    reklamos wrote: »
    Nice setup you have there.
    I have been automating for a while now all based on HomeAssistant
    Here is my setup:
    HA, nodered, esphome, influxdb, grafana and many more running on docker. I have used RaspberryPi before but it did not have grunt at that time and SD cards were not reliable.
    PV monitoring:
    I have Solis inverter and it is sending it's data over wifi to cloud. I have found online how to talk to API and wrote simple python app that is running in container. It is scraping that data and pulling it back and sending over mqtt to HA, this puts it into influxdb for longer term storage. This also allows me to use grafana to make fancy dashboard(see attached).
    I have another small 12V PV on the shed that is charging leisure battery. This one has basic monitoring(Voltage, Amps)

    Could you share that python script? I've a solis inverter also and would like to have my own copy of thw data. That grafana dashboard looks lovely. Man... I'm jealous these setups!!


  • Registered Users Posts: 6,400 ✭✭✭ Alkers


    How much did all the monitoring kit set you back? Did you have either of system setup basically beforehand and have you any handle of how much a difference it is making in terms of your utilisation of the power you generate? Are there any off the shelf kits available or why did you go such s DIY route?


  • Registered Users Posts: 791 ✭✭✭ reklamos


    mp3guy wrote: »
    Nice! How many hours have you put into this? How often does something go on the blink and need to be looked at? That's the killer for me, something randomly stops responding or you have to restore from backup (I presume with all this you have an automated offsite backup and restore facility?).

    I'm already pulling data from inverters over modbus, far far superior to relying on external flaky cloud APIs. Also the update rate is fantastic (order of seconds rather than minutes!).
    I do not know how much time I put into this as I keep adding stuff. It took some time to fine tune the automation parts and all the flows. There were wth moments when the automation would kick in but it always was doing what I told it to. I just have not thought of the specific scenario and these are normal teething problems. Now it works pretty well and stable, otherwise it would not be there. Wife and kids also know how to use it, which I consider a success :)
    Because everything is conteinerized the updates are easy, if something does not work you just rollback. Also containers look after themselves and restart if crashes. Never had to restore the data. I do make backups but only ones a month as there is not much config changes happening. If I lose monitoring data so be it. I do not concider it critical. The only blips that happen time to time is that Solis cloud stops responding/updating but that is out of my control hence I want to monitor locally.

    There are plenty automation kits on the market but here are my reasons to avoid them:
    1. Majority want to talk to cloud
    2. Each vendors have their apps/controls
    3. Initial cost and in some cases monthly subscription cost
    4. Manufacture impose limitations


    I like DIY but I do understand that this is not for everyone.


  • Registered Users Posts: 791 ✭✭✭ reklamos


    tails_naf wrote: »
    Could you share that python script? I've a solis inverter also and would like to have my own copy of thw data. That grafana dashboard looks lovely. Man... I'm jealous these setups!!
    Here is better one https://github.com/dkruyt/ginlong-scraper mine has not been updated for a long time. This one looks pretty good will probably convert mine to it also. This is also the dashboard that I based mine on, except mine is heavily modified. Feel free to PM if you have questions.


  • Registered Users Posts: 791 ✭✭✭ reklamos


    Alkers wrote: »
    How much did all the monitoring kit set you back? Did you have either of system setup basically beforehand and have you any handle of how much a difference it is making in terms of your utilisation of the power you generate? Are there any off the shelf kits available or why did you go such s DIY route?
    It did made impact how/when I use my electricity. I was able to greatly improve my solar utilization with the help of monitoring and automation. This allows me to keep my export to minimum.
    See my earlier post regarding DIY


  • Registered Users Posts: 1,303 ✭✭✭ SD_DRACULA


    reklamos wrote: »
    Nice setup you have there.
    PV monitoring:
    I have Solis inverter and it is sending it's data over wifi to cloud. I have found online how to talk to API and wrote simple python app that is running in container. It is scraping that data and pulling it back and sending over mqtt to HA, this puts it into influxdb for longer term storage. This also allows me to use grafana to make fancy dashboard(see attached).
    I have another small 12V PV on the shed that is charging leisure battery. This one has basic monitoring(Voltage, Amps)

    I will need to do something very similar as I will be getting one of the Solis inverters too and unfortunately they don't have native integration with Home Assistant which is too bad.
    Might give you a shout when it comes to configuring.
    I assume you cannot control it also this way? i.e. Change the times when the battery can be charged from the mains when topping it up with night rate.


  • Registered Users Posts: 849 ✭✭✭ tails_naf


    I had a cut off doing this over the easter break...it's not for the faint hearted - long story short:

    My HW:
    1) Solis inverter
    2) KNX for lighting control
    3) don't have energy monitoring just yet
    4) Old laptop running linux as the 'server'
    5) NAS drive that I used for backups. It's old and is mounted as a samba share

    I started with some guides online for using home-assistant+influxdb+grafana, and the ginlong-scraper script.

    Murphy's law was well and truly in play as I ran into the following issues:
    1) HomeAssistant can run in docker, but it's not as full featured (no plugins) - so after messing with the docker version, threw that away and went for the a full virtual-machine via virtualbox...may be overkill, but it works

    2) The docker home-assistant uses a database, but it would not update - always 0 bytes. After much digging found it's to do with drive permissions on cifs mounted network shares. Jsut doesn't work. Had to move to my local laptop drive, then it was grand. Took a detour into using postgresql to see if that would work (I like that the NAS is a RAID backup) - but that also fails to run on a samba/cifs share

    3) Could not get influxdb to work with docker. Found out after it was the same cifs/samba issue - so moved that to the local disk. Then it worked - and could see data from home-assistant being logged into influxdb (yippee!)

    4) The ginlong-scraper was designed for influxdb v1.x, but v2.0 really changed everything. Had to re-write some of it to make it work with the new python API.

    5) Installed grafana - and again got bitten by the database on the NAS - so basically now all of these tools are running off my laptop drive.

    6) Grafana is not 100% compatible with v2 influxdb, and the query language has totally changed, so couldn't use the default dashboard that comes with ginlong-scraper, so spent some time enabling legacy mode in v2.0 to allow v1.x access methods...

    whew....so now I have a rudimentary setup going, with home-assistant able to talk to KNX and log things in influxdb, and have it displayed on a grafana dashboard. Need to do a lot of tidy up to see how I can get all of these services auto started if necessary, as well as backing up databases to the NAS... busy weekend, would like to see what other things I can add to these dashboards, seems like a powerful system


  • Advertisement
  • Registered Users Posts: 791 ✭✭✭ reklamos


    tails_naf wrote: »
    I had a cut off doing this over the easter break...it's not for the faint hearted - long story short:

    My HW:
    1) Solis inverter
    2) KNX for lighting control
    3) don't have energy monitoring just yet
    4) Old laptop running linux as the 'server'
    5) NAS drive that I used for backups. It's old and is mounted as a samba share

    I started with some guides online for using home-assistant+influxdb+grafana, and the ginlong-scraper script.

    Murphy's law was well and truly in play as I ran into the following issues:
    1) HomeAssistant can run in docker, but it's not as full featured (no plugins) - so after messing with the docker version, threw that away and went for the a full virtual-machine via virtualbox...may be overkill, but it works

    2) The docker home-assistant uses a database, but it would not update - always 0 bytes. After much digging found it's to do with drive permissions on cifs mounted network shares. Jsut doesn't work. Had to move to my local laptop drive, then it was grand. Took a detour into using postgresql to see if that would work (I like that the NAS is a RAID backup) - but that also fails to run on a samba/cifs share

    3) Could not get influxdb to work with docker. Found out after it was the same cifs/samba issue - so moved that to the local disk. Then it worked - and could see data from home-assistant being logged into influxdb (yippee!)

    4) The ginlong-scraper was designed for influxdb v1.x, but v2.0 really changed everything. Had to re-write some of it to make it work with the new python API.

    5) Installed grafana - and again got bitten by the database on the NAS - so basically now all of these tools are running off my laptop drive.

    6) Grafana is not 100% compatible with v2 influxdb, and the query language has totally changed, so couldn't use the default dashboard that comes with ginlong-scraper, so spent some time enabling legacy mode in v2.0 to allow v1.x access methods...

    whew....so now I have a rudimentary setup going, with home-assistant able to talk to KNX and log things in influxdb, and have it displayed on a grafana dashboard. Need to do a lot of tidy up to see how I can get all of these services auto started if necessary, as well as backing up databases to the NAS... busy weekend, would like to see what other things I can add to these dashboards, seems like a powerful system
    Looks like you went through a lot of pain :)
    To me seems like most of your problems related to docker volume mounts. I do not use cifs but NFS volumes work pretty well.
    Not sure what plugins do not work in docker for you. I have all integrations running fine and also use HACS. If you talking about addons and addons store then afaik it only works in hass.io.
    You can use influx 1.8, do not have to go to 2.x. As mentioned before I will be moving away from influx myself.

    I also had some time over the long weekend and was able to pull some stats from my pylon batteries. Interesting thing I found that they only had 287 cycles and that is after 21 month of usage. I had thought that it will be way more than that. This means that I sue 150 cycles a year. They are rated at 6000 cycles officially. Even if I take 3000 cycles it would take ~20years to reach it. Of course they may lose capacity and die for other reasons. Now just need to get that data into HA :)


  • Registered Users Posts: 3,913 ✭✭✭ mp3guy


    tails_naf wrote: »
    I had a cut off doing this over the easter break...it's not for the faint hearted - long story short:

    My HW:
    1) Solis inverter
    2) KNX for lighting control
    3) don't have energy monitoring just yet
    4) Old laptop running linux as the 'server'
    5) NAS drive that I used for backups. It's old and is mounted as a samba share

    I started with some guides online for using home-assistant+influxdb+grafana, and the ginlong-scraper script.

    Murphy's law was well and truly in play as I ran into the following issues:
    1) HomeAssistant can run in docker, but it's not as full featured (no plugins) - so after messing with the docker version, threw that away and went for the a full virtual-machine via virtualbox...may be overkill, but it works

    2) The docker home-assistant uses a database, but it would not update - always 0 bytes. After much digging found it's to do with drive permissions on cifs mounted network shares. Jsut doesn't work. Had to move to my local laptop drive, then it was grand. Took a detour into using postgresql to see if that would work (I like that the NAS is a RAID backup) - but that also fails to run on a samba/cifs share

    3) Could not get influxdb to work with docker. Found out after it was the same cifs/samba issue - so moved that to the local disk. Then it worked - and could see data from home-assistant being logged into influxdb (yippee!)

    4) The ginlong-scraper was designed for influxdb v1.x, but v2.0 really changed everything. Had to re-write some of it to make it work with the new python API.

    5) Installed grafana - and again got bitten by the database on the NAS - so basically now all of these tools are running off my laptop drive.

    6) Grafana is not 100% compatible with v2 influxdb, and the query language has totally changed, so couldn't use the default dashboard that comes with ginlong-scraper, so spent some time enabling legacy mode in v2.0 to allow v1.x access methods...

    whew....so now I have a rudimentary setup going, with home-assistant able to talk to KNX and log things in influxdb, and have it displayed on a grafana dashboard. Need to do a lot of tidy up to see how I can get all of these services auto started if necessary, as well as backing up databases to the NAS... busy weekend, would like to see what other things I can add to these dashboards, seems like a powerful system

    Why do you need Home Assistant and Grafana? What's wrong with just Home Assistant and its own db / frontend?


  • Registered Users Posts: 791 ✭✭✭ reklamos


    SD_DRACULA wrote: »
    I assume you cannot control it also this way? i.e. Change the times when the battery can be charged from the mains when topping it up with night rate.


    I've read in other forums that people can make changes remotely but I am not there yet and I would say that will depend a lot on the inverter used.


  • Registered Users Posts: 849 ✭✭✭ tails_naf


    mp3guy wrote: »
    Why do you need Home Assistant and Grafana? What's wrong with just Home Assistant and its own db / frontend?

    I went this route as a week ago I knew zero about any of this and it seemed to be the done thing from what I saw online - however now I agree grafana might not be necessary as HA can also plot things, just not sure if it can do it as nicely as grafana can - might play around with it.


  • Registered Users Posts: 849 ✭✭✭ tails_naf


    reklamos wrote: »
    Looks like you went through a lot of pain :)
    To me seems like most of your problems related to docker volume mounts. I do not use cifs but NFS volumes work pretty well.
    Not sure what plugins do not work in docker for you. I have all integrations running fine and also use HACS. If you talking about addons and addons store then afaik it only works in hass.io.
    You can use influx 1.8, do not have to go to 2.x. As mentioned before I will be moving away from influx myself.

    I also had some time over the long weekend and was able to pull some stats from my pylon batteries. Interesting thing I found that they only had 287 cycles and that is after 21 month of usage. I had thought that it will be way more than that. This means that I sue 150 cycles a year. They are rated at 6000 cycles officially. Even if I take 3000 cycles it would take ~20years to reach it. Of course they may lose capacity and die for other reasons. Now just need to get that data into HA :)

    Nice - how did you get into that dashboard for the pylon batteries? I have them too...


  • Registered Users Posts: 849 ✭✭✭ tails_naf


    reklamos wrote: »
    Looks like you went through a lot of pain :)
    To me seems like most of your problems related to docker volume mounts. I do not use cifs but NFS volumes work pretty well.
    Not sure what plugins do not work in docker for you. I have all integrations running fine and also use HACS. If you talking about addons and addons store then afaik it only works in hass.io.
    You can use influx 1.8, do not have to go to 2.x. As mentioned before I will be moving away from influx myself.

    I also had some time over the long weekend and was able to pull some stats from my pylon batteries. Interesting thing I found that they only had 287 cycles and that is after 21 month of usage. I had thought that it will be way more than that. This means that I sue 150 cycles a year. They are rated at 6000 cycles officially. Even if I take 3000 cycles it would take ~20years to reach it. Of course they may lose capacity and die for other reasons. Now just need to get that data into HA :)

    The 'full' version of HA supported addons, which I think gives you things like MQTT via mosquito (yet to set that up), as well as an interface for backing up the database, and a function that ensures addons are running, and will re-start them if they die for some reason. I guess these things can be done with the docker version, though more manually - and to be honest running a VM seems a bit much.

    In terms of HA - one thing that confuses me is a 'view' - all I have right now is the default dashboard that has each entity there - but I'd like to be able to create perhaps a tree of dashboards (by room, function, etc). Is that possible?


  • Registered Users Posts: 791 ✭✭✭ reklamos


    tails_naf wrote: »
    I went this route as a week ago I knew zero about any of this and it seemed to be the done thing from what I saw online - however now I agree grafana might not be necessary as HA can also plot things, just not sure if it can do it as nicely as grafana can - might play around with it.
    Grafana is much more powerful with dashboards compared to HA and is useful if you want to store the data for longer term and want to do something with it. Otherwise HA is enough
    tails_naf wrote: »
    Nice - how did you get into that dashboard for the pylon batteries? I have them too...
    You need to connect serial(rs232) cable to the master battery and from there you can interface with them. There are free apps online that can pull data. This was only for me to test if the connection working. I am waiting for some part to arrive and will be using esp8266 to send data to HA.
    tails_naf wrote: »
    The 'full' version of HA supported addons, which I think gives you things like MQTT via mosquito (yet to set that up), as well as an interface for backing up the database, and a function that ensures addons are running, and will re-start them if they die for some reason. I guess these things can be done with the docker version, though more manually - and to be honest running a VM seems a bit much.

    In terms of HA - one thing that confuses me is a 'view' - all I have right now is the default dashboard that has each entity there - but I'd like to be able to create perhaps a tree of dashboards (by room, function, etc). Is that possible?

    Docker version and 'full' should be no different, they just packaged differently. I use mosquito that is also running in docker as this is what I use. It does not take much effort I would say once you understand how containers work it is far less work and faster compared to VMs. With VMs you virtualize OS with all the apps, with container you virtualize the apps.
    Hass.io is a more user friendly version of HA. It is more geared to users who just want to point and click with very little manual config but it has its limitations.


    You can have as multiple dashboards in HA. Click on 3 dots(see image) in the corner and select edit dashboard. There you can add more tabs and edit/add/remove cards


  • Registered Users Posts: 849 ✭✭✭ tails_naf


    reklamos wrote: »

    Docker version and 'full' should be no different, they just packaged differently. I use mosquito that is also running in docker as this is what I use. It does not take much effort I would say once you understand how containers work it is far less work and faster compared to VMs. With VMs you virtualize OS with all the apps, with container you virtualize the apps.
    Hass.io is a more user friendly version of HA. It is more geared to users who just want to point and click with very little manual config but it has its limitations.


    You can have as multiple dashboards in HA. Click on 3 dots(see image) in the corner and select edit dashboard. There you can add more tabs and edit/add/remove cards

    I was just looking through installing the supervisor version - it seems to be really discouraged - I've never seen more warnings that you must be a linux expert, yadda, yadda... though in the end it seems to be installing a docker image ... it also says you need debian 10 exactly, no iffs, buts or derivative OSes - what OS are you using? I'm using linux mint 20, ubuntu based.


  • Registered Users Posts: 791 ✭✭✭ reklamos


    tails_naf wrote: »
    I was just looking through installing the supervisor version - it seems to be really discouraged - I've never seen more warnings that you must be a linux expert, yadda, yadda... though in the end it seems to be installing a docker image ... it also says you need debian 10 exactly, no iffs, buts or derivative OSes - what OS are you using? I'm using linux mint 20, ubuntu based.


    Have not tried Supervisor but it looks to me like hass.io which I did not like as it was locked down and like to know what is happening under the hood and be able to make my own choices :) I have looked in the features and it would not provide me anything I would need at this point.
    I am still using coreOS as when I started it suited me best by being lightweight and this is where I run most of my containers. I will be migrating to Kubernetes soon, since I outgrew docker.


    I had some time this week and now I can pull data directly from my Solis inverter. Not only that I can also make changes also remotely. It is early days but I can make automate when and by how much to charge the batteries. Potentially there are other automatons that I have not thought off.


  • Advertisement
  • Registered Users Posts: 849 ✭✭✭ tails_naf


    reklamos wrote: »
    .
    I had some time this week and now I can pull data directly from my Solis inverter. Not only that I can also make changes also remotely. It is early days but I can make automate when and by how much to charge the batteries. Potentially there are other automatons that I have not thought off.

    Nice one, woild much prefer to talk to the inverter directly rather than scraping from the ginlong site. What do you use to login to the inverter? Any references/links you can share ?


Advertisement