Advertisement
If you have a new account but are having problems posting or verifying your account, please email us on hello@boards.ie for help. Thanks :)
Hello all! Please ensure that you are posting a new thread or question in the appropriate forum. The Feedback forum is overwhelmed with questions that are having to be moved elsewhere. If you need help to verify your account contact hello@boards.ie
Please note that it is not permitted to have referral links posted in your signature. Keep these links contained in the appropriate forum. Thank you.

https://www.boards.ie/discussion/2055940817/signature-rules
Hi all,
Vanilla are planning an update to the site on April 24th (next Wednesday). It is a major PHP8 update which is expected to boost performance across the site. The site will be down from 7pm and it is expected to take about an hour to complete. We appreciate your patience during the update.
Thanks all.

Ecars stats - a work in progress

  • 27-10-2020 4:35pm
    #1
    Black_Knight
    Moderators Posts: 12,363 ✭✭✭✭


    As some may know i've been pulling stats from ecars this while now. Until now it's been running on my home mini-pc, but I always wanted to make it more public. I had/have planned to re architect it to run as cheaply as I can in the cloud, and to add a few more insights (like peak times per charge point) but time is against me. For now, i've just moved it to a VM with certainly not enough memory (trying to keep costs down). It's not finished (it never will be) but for now I said i'd open it up and see how it runs (or falls over) with some traffic.

    With that in mind, go gentle (on me and the queries). It's not fallen over on me yet (only have it running 24 hours now), but a few extra users throwing queries at it might be the tipping point. I'd like to keep the cost below a couple of hundred a year so i've a small margin to add more memory, but if it just can't handle the traffic it'll have to wait until I've time to re architect it entirely.

    It's all grafana, and I didn't setup anonymous read only access yet, so it's user/password for now.

    https://ecars-stats.com/d/eFhAZ1cMz/esb-charger-availablilty
    test:testPassword

    image.png

    image.png


«134

Comments

  • #2
    Kramer
    Registered Users Posts: 3,224 ✭✭✭Kramer


    ........awaits DDOS attack from ECars..........:D.


  • #3
    Kramer
    Registered Users Posts: 3,224 ✭✭✭Kramer


    Fair play, great work :).


  • #4
    innrain
    Registered Users Posts: 2,071 ✭✭✭innrain


    Looks very nice.
    If I understand correctly on the graphs y represents how many plugs are connected at any given moment. On you screen grab at 16:00 are 8 CHAdeMO, 5 CCS and 2 Type2 (in majority AC43). Or now 20:55 5 CHA and 4 CCS -> 10% of the network is in use. Cool


  • #5
    Black_Knight
    Moderators Posts: 12,363 ✭✭✭✭Black_Knight


    Kramer wrote: »
    ........awaits DDOS attack from ECars..........:D.

    I DOS them, they DDOS me. Checkmate


  • #6
    Kramer
    Registered Users Posts: 3,224 ✭✭✭Kramer


    Would you have the stats for the Thomond, Limerick rapid for those 13 days I posted overall throughput for, in this post?

    https://www.boards.ie/vbulletin/showpost.php?p=115076440&postcount=2405

    It might be interesting to see how many individual charges were recorded, the ratio between CCS & Chademo, the average kWh drawn, if anyone stayed beyond 45 minutes etc.

    No AC usage in that time so it might be easier, if you have the data for that period.


  • Advertisement
  • #7
    Black_Knight
    Moderators Posts: 12,363 ✭✭✭✭Black_Knight


    innrain wrote: »
    Looks very nice.
    If I understand correctly on the graphs y represents how many plugs are connected at any given moment. On you screen grab at 16:00 are 8 CHAdeMO, 5 CCS and 2 Type2 (in majority AC43). Or now 20:55 5 CHA and 4 CCS -> 10% of the network is in use. Cool

    Yeah. For those graphs the Y axis is a count. The graphs over time are accurate to every 5 minutes. FYI the TYPE_2_MENNEKES is "Maxol Centra Antrim Road Filling Station". Says it's AC43 on the app, but when it's queried it's returns as TYPE_2_MENNEKES. If you go to that unit and filter by AC43 you won't see any results. Glitch I guess.

    The single stat panels are a count of the type of socket used in that time period. eg. Filtering by "Today so far" I can see 49 CCS socket sites were in use* and 58 CHAdeMO socket sites were in use*. 20 AC43 were also in use during this time.


    *In use is a loose term. The actual number represents the number of unique sites which were fully occupied for that socket today.

    Under the "Units/Sites" row you can see that that 58/49/20 usage count sums up to 77 FCP sites being fully occupied today. I could probably break that down to DC FCP fully occupied and AC FCP fully occupied if wanted, though not sure how much use it is.


  • #8
    Black_Knight
    Moderators Posts: 12,363 ✭✭✭✭Black_Knight


    Kramer wrote: »
    Would you have the stats for the Thomond, Limerick rapid for those 13 days I posted overall throughput for, in this post?

    https://www.boards.ie/vbulletin/showpost.php?p=115076440&postcount=2405

    It might be interesting to see how many individual charges were recorded, the ratio between CCS & Chademo, the average kWh drawn, if anyone stayed beyond 45 minutes etc.

    No AC usage in that time so it might be easier, if you have the data for that period.

    You can do that.... in a somewhat roundabout way. Find the siteID for thomond (ctrl F really). site ID is 22

    Pop 22 in the input at the top, and open the "general" row down the very bottom. It'll be messy, but it'll show you what the site was doing.

    https://ecars-stats.com/d/eFhAZ1cMz/esb-charger-availablilty?orgId=1&from=1602611400000&to=1603828200000&var-SiteID=499&var-SocketType=All

    Search Thomondgate. 14.17hours. Lines up pretty well with your data. 33kW average.

    I could get a graph showing charging against time over that time period, but when I tried it there I think I OOM killed my DB.
    EDIT: Got it:
    https://ecars-stats.com/d/eFhAZ1cMz/esb-charger-availablilty?orgId=1&from=1602611400000&to=1603828200000&var-SiteID=22&var-SocketType=TYPE_COMBO_GERMANY

    Hard to see the period of charing given the scale, but you can see how often CCS was used VS CHAdeMO. Just use the drop down up the top to change socket type.
    From the Socket section I can see for that time period CCS:
    2020-10-13 18:50:00 Circle K -Topaz Service Station, Sexton Street North, Thomondgate, Limerick City 22 8.25 hour
    CHAdeMO:
    2020-10-13 18:50:00 Circle K -Topaz Service Station, Sexton Street North, Thomondgate, Limerick City 22 5.92 hour


  • #9
    Black_Knight
    Moderators Posts: 12,363 ✭✭✭✭Black_Knight


    Totally jumping to conclusions here, but I'm gonna say a Zoe broke the Frankfield hill charge point the other day. Spotted it red today when I drove past.
    https://ecars-stats.com/d/eFhAZ1cMz/esb-charger-availablilty?orgId=1&from=1603890093588&to=1603898042909&var-SiteID=499&var-SocketType=All&var-Name=Circle%20K%20-%20Topaz%20Frankfield%20Service%20Station,%20Frankfield%20Road,%20Douglas,%20Cork

    Goes from AC43 charging to unknown and unknown ever since.

    30 hours out of use now. Good job ECars. Highly dependable.


  • #10
    Black_Knight
    Moderators Posts: 12,363 ✭✭✭✭Black_Knight


    Added a heatmap for usage per unit over the last week (select the unit from the dropdown at the top). Not exactly what I'd want but for now it's all i've time for.

    Ideally i'd aggregate 4 weeks of data into a week (split by days of the week) to give a rounder impression of typical usage per day. Needs thinking though. In the meantime, any feedback is welcome.

    image.png


  • #11
    innrain
    Registered Users Posts: 2,071 ✭✭✭innrain


    I think the idea to align the data sets based on weekdays it is very good. Not sure how difficult would be but having an average to compare to it is helpful. Say you want to compare today's usage pattern for a unit you need a reference and averaging all weekdays together might loose some insights. By comparing Mondays with Mondays you have a better picture.

    I noticed that the graphs in the units/sites panels have an offset of 1? They never drop to zero.


  • Advertisement
  • #12
    Black_Knight
    Moderators Posts: 12,363 ✭✭✭✭Black_Knight


    innrain wrote: »
    I think the idea to align the data sets based on weekdays it is very good. Not sure how difficult would be but having an average to compare to it is helpful. Say you want to compare today's usage pattern for a unit you need a reference and averaging all weekdays together might loose some insights. By comparing Mondays with Mondays you have a better picture.

    I noticed that the graphs in the units/sites panels have an offset of 1? They never drop to zero.

    Yeah, I did that because some of the graphs were jumping to 0 (when metrics were missed in a 5 minute period). And I only store data on the current active state of the site, not every state (ie. if you're charging, then available=0, occupied=0, preparing=0 etc). It looked a bit janky.

    I've updated it now so a lack of a metric will be a zero value. Let me know what you think


    Also. 99 fast charger sites! Galway will likely be the 100th.


  • #13
    Black_Knight
    Moderators Posts: 12,363 ✭✭✭✭Black_Knight


    Added anonymous access. No need for a login anymore


  • #14
    jusmeig
    Registered Users Posts: 1,252 ✭✭✭jusmeig


    Great work!!!!


  • #15
    Black_Knight
    Moderators Posts: 12,363 ✭✭✭✭Black_Knight


    Can see the rush on Cashel on the 18th (inter county travel permitted) after work.

    image.png

    Does everyone flock there because Ionity is a backup, but then not want to use it because it's so expensive. Other surrounding DC units seemed relatively quiet.

    Created a slim down dashboard to just check individual sites.
    https://ecars-stats.com/d/1CqPEh1Gk/ecars-single-site-data?orgId=1

    Wanted to have aggregated view of the week for each site ready for the Christmas rush, but zero time to get it together.


  • #16
    zg3409
    Registered Users Posts: 6,548 ✭✭✭zg3409


    Slightly different but similar project from NI, to collect details from NI ESB and non ESB sites. The aim is to use as proof to lobby for improvement to public charging network.

    Read only manual data entered here:

    https://docs.google.com/spreadsheets/d/1TOSPH63cx8Ez3RAbuP3MDy-d5bbTpx3fo8J8fUitObs/edit?usp=sharing

    From NI EV owners Facebook
    Sam Johnston , Everybody :

    I have attempted to create an online database of Charge points in Northern Ireland. I've not done this with fancy API's, just a few late nights using cpinfo's last archive, cross referenced with plugshare and easygo to get the current online status. This is not complete, because cpinfo archive does not include newer( e.g. lidl ) chargers.

    The data is now in a google sheet so available for any of us to update ( or spam with fake news !)

    So here's state of the [partial] network as of this week

    Link below is the non-editable google sheet for review. If you have any more chargers you want me to add, let me know and I'll update it until it a) gets unmanageable or b) someone has webz skillz and volunteers to add a nice webfront end to automate updating.

    Why bother doing this? Because we can create proper stats on the overall network, something that is very much absent in the existing system, where you have to piecemeal enter individual chargepoints.

    Anyone with better google sheet skills than me can also use this data set to get some really good info, like geographic dead spots etc.

    obviously those postal regions without *any* chargers don't appear on the map at all, but from my reading of the pivot table, BT30,35,36,37,41,45,47 and 78 are particularly badly in need of repair.

    The numbers of chargers: I've been generous and counted every socket, not just the stations. This also allows us to mark every non-functional cable as many stations have one working and one offline.

    I've set anything as "Not working" where Easygo reports broken, or no communication. I've also been generous and assumed every station that doesn't explicitly mention power is outputting 22kW AC - if you know ones that should be 11 or 7 ( or 43 ) please let me know.

    Goog Sheet : https://docs.google.com/spreadsheets/d/1TOSPH63cx8Ez3RAbuP3MDy-d5bbTpx3fo8J8fUitObs/edit?usp=sharing


  • #17
    Black_Knight
    Moderators Posts: 12,363 ✭✭✭✭Black_Knight


    zg3409 wrote: »
    Slightly different but similar project from NI, to collect details from NI ESB and non ESB sites. The aim is to use as proof to lobby for improvement to public charging network.

    Read only manual data entered here:

    https://docs.google.com/spreadsheets/d/1TOSPH63cx8Ez3RAbuP3MDy-d5bbTpx3fo8J8fUitObs/edit?usp=sharing

    From NI EV owners Facebook

    Ugh. Manual will never work out. I might reach out and share my site. I've bumped the spec of the machine so it can query maybe a week, but sure you know yourself week on week everything changes. I've some logic to build report data for every unit per day, and then roll that up into a week, but I've no time to look into visualising that. It'd likely need something custom made and my front end skills ain't there.

    New baby hasn't afforded me much spare time.


  • #18
    Black_Knight
    Moderators Posts: 12,363 ✭✭✭✭Black_Knight


    After lots of twiddling my thumbs I finally did a bit of work on this. Nothing to do with "busy times" or anything like that, but moreso the "how's progress going" side of things.

    https://ecars-stats.com/d/N7D-5-VMk/ecars-marked-for-upgrade

    List of added and removed units (literally just finished now so no historical data):
    image.png


    I improved the charger lookup also, so if you search "Kilmartins" you now see the list of every unit on that site, ie. 2.

    Also made the single site data a bit more performant and robust.
    https://ecars-stats.com/d/1CqPEh1Gk/ecars-single-site-data


    I want to build a better "peak times" widget, but I don't think it's possible in Grafana, and certainly too heavy on system memory. I can calculate the peak times per day, and then roll that up into a 4 week period no problem, but displaying it is a different story, and i'm certainly not a front end developer.


  • #19
    Black_Knight
    Moderators Posts: 12,363 ✭✭✭✭Black_Knight


    I'd like to say i've been working on getting something a bit more useful out there for the last few months, but sadly time is very limited. I've been doing some backend work to build data on each sites chargers sockets usage so I can put that data into a heatmap, and that's gone pretty well, but my lack of web front end skills is really showing now.
    Had a few hours this weekend to try put something together. Not sure it's the right direction to be going, but i'm learning by doing so as I learn i'll hopefully improve it, or get frustrated, scrap it and start again.

    http://test.ecars-stats.com/plotly.html
    It's not dynamic yet, only shows CCS, and the numbers don't really mean anything (a 1 indicates that over the last 4 weeks on that day at that hour the site was in use for 5 minutes).

    Next steps are to:
    * get the numbers signifying minutes
    * Account for sites with multiple of the same socket
    * Allow selection of any site
    * Display all socket types at a site


  • #20
    innrain
    Registered Users Posts: 2,071 ✭✭✭innrain


    I think it is brilliant the fact you manage to align the data. I can read a color chart even without numbers. They could be useful when comparing two different chargers otherwise the color is enough. The sum of the minutes used divided to the bin size (60 mins) should do the trick.
    I would use the rainbow color scheme as it is more granular but that is something personal.
    Yesterday you had Lucan on the page and by comparison with N11 you could see the former was in the city with check ins @1am. EV drivers not risking being out of juice so late on the motorway :)


  • #21
    Black_Knight
    Moderators Posts: 12,363 ✭✭✭✭Black_Knight


    Updated now to have selectable locations. Nevermind duplicates, it's returning addresses for each unit rather than being reduced to 1 per site.

    The data shown is an accumulation of when a socket (CCS only for now) has been charging (or unavailable to use because the other plug, CHAdeMO, is in use).

    Multi unit sites (mayfield etc) will look busier, but that's just because there's more charging going on at those places. I'll scale that down so the numbers are percentage of use per hour divided by available sockets.

    eg. If only 1 CCS socket was in use for a full hour (3pm-4pm) at mayfield should only represent 16% usage for that hour, as there's 6 CCS plugs there. I'm trying to give a visualisation on the likelihood when you arrive at a site, there'll be an available socket for you.

    Also plan to map out the faulty sockets in a similar fashion.


  • Advertisement
  • #22
    innrain
    Registered Users Posts: 2,071 ✭✭✭innrain


    Talbot Hotel Stillorgan Saturday @6am close to the max bracket :) The other Stillorgan P&R utilized nearly 24h on Saturdays Who said Blanch and Lucan are bad.
    M9 Kilcullen used up 24h on Mon/Tue (probably bank holiday spur). Nice data. And this is CCS only.


  • #23
    Black_Knight
    Moderators Posts: 12,363 ✭✭✭✭Black_Knight


    innrain wrote: »
    Talbot Hotel Stillorgan Saturday @6am close to the max bracket :) The other Stillorgan P&R utilized nearly 24h on Saturdays Who said Blanch and Lucan are bad.
    M9 Kilcullen used up 24h on Mon/Tue (probably bank holiday spur). Nice data. And this is CCS only.

    All the more reason for me to handle sites with multiple chargers better, and get the values in % terms rather than what they currently are (I won't go into it, but at the moment 48 would mean for the last 4 weeks CCS has been in full use that hour).
    Where there's X CCS plugs at a site, divide the number in the box by X. Or wait for me to fix it so they're all uniform.

    Nice spot though. They're crazy busy!
    innrain wrote: »
    And this is CCS only.
    True, but it does take into account when CHAdeMO is in use and thus blocking CCS from being used.

    Data when CCS charging or CCS unavailable because CHAdeMO is in use:
    image.png

    Data when CCS charging only:
    image.png

    The combo would suggest a lot of late night charging... i'll look into that. That seems unlikely. Do I need to stick a bit BETA sticker all over the page?


    EDIT: It's not a bug. That's someone not charging but plugged in. Makes sense at a hotel. Unit has status "AVAILABLE" but when you look at the socket itself it's "CHARGING". Explains why I can see CHAdeMO socket being much more available than the CCS one at Talbot hotel - the CCS spends most its time plugged into cars not charging.


  • #24
    Calahonda52
    Registered Users Posts: 12,164 ✭✭✭✭Calahonda52


    Great project:)
    1: Can the week start on Monday, makes comparing busy times at weekend easier, for weekends away :)
    2: why just one Lidl: is it a pay one?
    3: Randels garage Tralee, is it open on Sundays?: I use it when down there 5 euro a fill: they closed the long weekend

    “I can’t pay my staff or mortgage with instagram likes”.



  • #25
    innrain
    Registered Users Posts: 2,071 ✭✭✭innrain


    Black_Knight wrote: »
    The combo would suggest a lot of late night charging... i'll look into that. That seems unlikely. Do I need to stick a bit BETA sticker all over the page?


    EDIT: It's not a bug. That's someone not charging but plugged in. Makes sense at a hotel. Unit has status "AVAILABLE" but when you look at the socket itself it's "CHARGING". Explains why I can see CHAdeMO socket being much more available than the CCS one at Talbot hotel - the CCS spends most its time plugged into cars not charging.


    No it does not need beta. The link is called test which is self-explanatory.



    Interesting. Maybe someone from the nearby apartments charging overnight?


  • #26
    innrain
    Registered Users Posts: 2,071 ✭✭✭innrain


    Calahonda52 wrote: »
    Great project:)
    1: Can the week start on Monday, makes comparing busy times at weekend easier, for weekends away :)
    2: why just one Lidl: is it a pay one?
    3: Randels garage Tralee, is it open on Sundays?: I use it when down there 5 euro a fill: they closed the long weekend
    @2 It is from the 3rd party list off the ecars map. No data from that unit anyway. I think it is AC now but ecars doesn't know

    @3 It is Killarney not Tralee is in the petrol station and it is accessible overnight.


  • #27
    Black_Knight
    Moderators Posts: 12,363 ✭✭✭✭Black_Knight


    Calahonda52 wrote: »
    Great project:)
    1: Can the week start on Monday, makes comparing busy times at weekend easier, for weekends away :)
    2: why just one Lidl: is it a pay one?
    3: Randels garage Tralee, is it open on Sundays?: I use it when down there 5 euro a fill: they closed the long weekend

    Sure. Gimme a sec


  • #28
    Black_Knight
    Moderators Posts: 12,363 ✭✭✭✭Black_Knight


    Calahonda52 wrote: »
    Great project:)
    1: Can the week start on Monday, makes comparing busy times at weekend easier, for weekends away :)
    2: why just one Lidl: is it a pay one?
    3: Randels garage Tralee, is it open on Sundays?: I use it when down there 5 euro a fill: they closed the long weekend
    Black_Knight wrote: »
    Sure. Gimme a sec

    Done.


  • #29
    Calahonda52
    Registered Users Posts: 12,164 ✭✭✭✭Calahonda52


    innrain wrote: »
    @2 It is from the 3rd party list off the ecars map. No data from that unit anyway. I think it is AC now but ecars doesn't know

    @3 It is Killarney not Tralee is in the petrol station and it is accessible overnight.
    :( specsavers1
    Thanks

    “I can’t pay my staff or mortgage with instagram likes”.



  • #30
    Black_Knight
    Moderators Posts: 12,363 ✭✭✭✭Black_Knight


    Numbers are now a % chance that socket is not going to be available when you land there (either in use or unavailable because DC is in use by the other plug).

    Numbers now also take into account how many sockets of a certain type there are at a site.

    Next:
    - Filterable dropdown :D
    - Add the other sockets to the page/ability to toggle between CCS/CHAdeMO etc. (I assume one would only really care about 1 socket type, rather than rendering both at the same time on screen.
    - Add the same view but for how often a unit is faulty.
    - Make it less crap looking... gonna struggle here for sure. :pac:


  • Advertisement
  • #31
    innrain
    Registered Users Posts: 2,071 ✭✭✭innrain


    That is pretty neat. Now you just need a mobile app where the user select the charger they want to go to and get advised if the charger is likely to be free @ arrival time. Get notified if the charger is getting used while driving towards it with an revised likelihood of being free @ arrival time. Considering our network will remain behind the EV take out for a very long time the app would be useful enough. Find a way to monetize this and you're flying with Bezoos to the moon :D


Advertisement