Advertisement
If you have a new account but are having problems posting or verifying your account, please email us on hello@boards.ie for help. Thanks :)
Hello all! Please ensure that you are posting a new thread or question in the appropriate forum. The Feedback forum is overwhelmed with questions that are having to be moved elsewhere. If you need help to verify your account contact hello@boards.ie

Two new long-term climate data bases (Toronto, NYC)

  • 19-03-2021 11:17pm
    #1
    Registered Users Posts: 14,279 ✭✭✭✭


    I have recently posted excel files sharing data bases for Toronto (1840-2021) and NYC (1869-2021) in a thread over on Net-weather. I am going to see if these excel files will upload directly here, if not, I will post the link to the thread where they are available.

    I would also be quite glad to share the tables posted in that same thread, except that on Net-weather I have no time limits on editing, so I can maintain the ranking tables and make corrections where necessary, whereas here I would need to cut a deal with a friendly moderator or (worse still) become one.

    Don't want that added workload, but could be a one-dimensional moderator just editing my own climate threads.

    Anyway, going ahead with the experiment to upload these excel files on boards, will go to plan (b) if they don't upload.

    So it's plan (b) unfortunately my files are too large for the boards limit.

    Here's where you will find the two data files plus all the tables of ranked data that I posted separately. The download files are in the last two posts made by me (exclusive of any chat that develops as I only just uploaded them).

    https://www.netweather.tv/forum/topic/93113-toronto-180-a-north-american-data-base-of-180-years-now-includes-nyc-1869-2020/page/3/?tab=comments#comment-4474159


Comments

  • Registered Users Posts: 6,235 ✭✭✭Oneiric 3


    NYC long term daily average temp profile from MT's data:

    rILCzSn.png

    Very similar trend to Ireland's, in that temps peak in the second half of July/early August and are at their coolest in January & February. Not similar to Ireland's though are the actual temps, which are far more extreme in the warmest and coldest seasons.

    25th/75th are percentiles and further edit: date on chart should read 2021 and not 2020.

    Unlike Ireland, however, there doesn't seem to be a notable 'drier' period in NYC long term data base. Rainfall evenly distributed throughout the year, with perhaps the winter months less prone to heavier rainfalls that the warmer ones:

    EHvEN2L.png



    Edit: Above temperature graph in a more digestible form:

    4vvO5ET.png


    Data Source © M.T. Cranium.

    New Moon



  • Registered Users Posts: 6,235 ✭✭✭Oneiric 3


    Here is the running ten year mean average temp at NYC.

    mBAHCff.png

    All in accordance with global trends, but... there was a very sharp rise in temps at NYC during the last decade of the 19th century (highlighted with yellow marker) which is on par with, or even greater, in terms of magnitude than the rise over the following 120 years. Very curious.

    New Moon



  • Registered Users Posts: 14,279 ✭✭✭✭M.T. Cranium


    Thanks for those samplers, lots to see in these two data bases.

    The NYC temp rise in the 1890s can also be seen at Toronto. Part of it may be a natural warming that followed the generally cold decades of the 1870s-1880s. Another factor is an increasing urban heat island, not sure how to estimate the actual values during the pre-automotive era but some other aspects of an urban heat island were valid even then. Also the numbers we are working with are adjusted means rather than the raw data base means, you can check out the differentials in the range G1869 to AS 2021 in the NYC data base. Those changes were made by the NWS based on (presumably) considerations of site exposure, climatological day, and other unknown (to me) factors.

    One factor that I thought might explain the fast rise in the 1890s would be that the north magnetic pole was then starting to move steadily north so if there were any correlation between magnetic field and circulation, the jet stream would probably tend to migrate north at that time. Once past a critical change, further temperature responses would be smaller since the main work would be to reduce arctic air mass frequency.

    The 1890s at both locations produced a lot of variations, the climate seemed to be in an unstable mode. For example, bitter record cold in Feb 1895 and record lows at times in May 1895 suddenly followed by record heat at the end of May and start of June 1895. A rather similar rebound from winter cold to spring heat in 1896.

    Oneiric, if you had some time to look, the graph you produced is presumably from the raw temperature averages, but there is a column of adjusted (for u.h.i.) means further over to the right of that one (something like BR 1869 to 2020). That would be a more "climate sensitive" graph of the rising temperature trend as it would eliminate the 1.1 C assigned to the urban heat island by 1980 (and from there to end of series). Alternatively, I might throw in a graph below of that same trend, if it works to copy the graph and post it. Let's see ...

    That procedure doesn't seem to work here, I have done that on other sites (copy a graph from an excel file to here), but I had a look at the relevant data and in the 1890s you'll see in the comparison table that the NWS have generally increased the raw data averages by 0.3 to 0.4 C each year in that decade. I also verified that the graph posted above is pre-urban heat island adjustment official data, the values in the next post down are the annual numbers and 11-yr running means for the adjusted data. This is what should be compared to CET or other data sets that are certified to be free of urban heat island influence.


  • Registered Users Posts: 14,279 ✭✭✭✭M.T. Cranium


    Mean annual temps (NYC) 1869 to 2020, adjusted for urban heat island

    The adjustments are 0.1 C for 1881-1890, 0.2 for 1891-1900, etc for each decade, ending with 1.1 for 1981-2020 when it is assumed that the urban heat island has reached a stable maximum level. So these numbers are lower than the official numbers by those amounts from 1881 to end of series (same for 1869 to 1880). The second value is the running 11-year mean.

    These data are in BR-BS 1869-2020 of the NYC file.

    1869 10.78
    1870 12.00
    1871 10.61
    1872 10.56
    1873 10.56
    1874 10.72 10.98
    1875 09.67 11.07
    1876 11.06 11.00
    1877 11.56 11.03
    1878 12.00 11.00
    1879 11.28 11.06
    1880 11.72 11.04
    1881 11.23 11.11
    1882 11.01 11.05
    1883 10.18 10.87
    1884 11.23 10.81
    1885 10.51 10.82
    1886 10.46 10.84
    1887 10.40 10.81
    1888 09.51 10.72
    1889 11.40 10.82
    1890 11.40 10.83
    1891 11.91 10.94
    1892 10.86 11.05
    1893 10.02 11.23
    1894 11.36 11.44
    1895 11.30 11.49
    1896 11.74 11.44
    1897 11.69 11.38
    1898 12.36 11.41
    1899 11.80 11.42
    1900 11.91 11.44
    1901 10.87 11.53
    1902 11.31 11.47
    1903 11.20 11.54
    1904 10.09 11.47
    1905 11.53 11.44
    1906 12.31 11.40
    1907 11.09 11.42
    1908 12.48 11.52
    1909 11.59 11.48
    1910 11.48 11.61
    1911 11.49 11.54
    1912 11.10 11.33
    1913 12.38 11.36
    1914 10.71 11.29
    1915 11.54 11.23
    1916 10.82 11.29
    1917 09.99 11.29
    1918 11.38 11.29
    1919 11.71 11.12
    1920 10.88 11.19
    1921 12.22 11.06
    1922 11.44 11.11
    1923 11.11 11.25
    1924 10.56 11.29
    1925 11.39 11.31
    1926 10.17 11.47
    1927 11.39 11.48
    1928 11.50 11.50
    1929 11.78 11.50
    1930 12.00 11.55
    1931 12.62 11.54
    1932 12.29 11.70
    1933 11.73 11.78
    1934 11.07 11.83
    1935 11.12 11.71
    1936 11.29 11.71
    1937 11.90 11.61
    1938 12.29 11.53
    1939 12.01 11.54
    1940 10.46 11.59
    1941 12.02 11.69
    1942 11.58 11.70
    1943 11.36 11.67
    1944 11.86 11.75
    1945 11.58 11.68
    1946 12.24 11.82
    1947 11.41 11.86
    1948 11.58 11.99
    1949 13.13 12.05
    1950 11.30 12.04
    1951 11.98 12.00
    1952 12.42 12.00
    1953 13.09 11.92
    1954 11.92 11.98
    1955 11.81 11.83
    1956 11.14 11.88
    1957 12.20 11.79
    1958 10.59 11.67
    1959 12.20 11.54
    1960 11.42 11.50
    1961 11.93 11.51
    1962 10.99 11.47
    1963 11.10 11.40
    1964 11.66 11.51
    1965 11.43 11.44
    1966 11.93 11.48
    1967 10.77 11.42
    1968 11.38 11.54
    1969 11.77 11.59
    1970 11.49 11.59
    1971 11.89 11.55
    1972 11.17 11.50
    1973 12.39 11.49
    1974 11.61 11.56
    1975 11.72 11.56
    1976 10.89 11.59
    1977 11.39 11.56
    1978 10.67 11.66
    1979 12.17 11.62
    1980 11.78 11.65
    1981 11.79 11.66
    1982 11.62 11.74
    1983 12.23 11.76
    1984 11.96 11.80
    1985 11.96 11.86
    1986 11.84 11.97
    1987 11.73 11.90
    1988 11.57 11.93
    1989 11.12 11.89
    1990 12.90 11.88
    1991 12.90 11.79
    1992 11.07 11.74
    1993 11.96 11.84
    1994 11.79 11.93
    1995 11.84 11.93
    1996 10.96 11.88
    1997 11.29 11.84
    1998 12.90 11.81
    1999 12.51 11.76
    2000 11.07 11.79
    2001 12.40 11.87
    2002 12.46 11.94
    2003 10.79 11.99
    2004 11.40 11.82
    2005 12.12 11.83
    2006 12.68 11.96
    2007 11.68 12.01
    2008 11.84 11.96
    2009 11.12 12.01
    2010 12.62 12.12
    2011 12.46 12.19
    2012 12.96 12.16
    2013 11.84 12.21
    2014 11.34 12.22
    2015 12.62 12.39
    2016 12.90
    2017 12.40
    2018 12.18
    2019 12.01
    2020 12.96
    _______________________________


  • Registered Users Posts: 6,235 ✭✭✭Oneiric 3


    Thanks for all that M.T. I'll take a further look at the data you produced later on this evening and perhaps even look at the CET data also to see if there was a similar trend here in NW Europe at the end of the 19th century (but no doubt you yourself have probably already investigated)

    After I posted last night, I was looking around on the internet to see if there was any long term datasets in the US from more rural regions. Came across this site and although text data seems to be lacking (unless I just cannot find them) a few nice graphs are available:

    http://climate.missouri.edu/charts.php

    Sample:

    do4XUBN.png

    Just 'eyeballing' that single graph from the heart of the N American continent, and it is clear that something cyclical is going on regarding longer term temp trends. Just a pity we don't have the actual data from this spot to do further work with.

    New Moon



  • Advertisement
  • Registered Users Posts: 14,279 ✭✭✭✭M.T. Cranium


    I'm just working on a small excel file with only the annual means for the three locations, and running means. That should be small enough to post here and maybe if I can post it I can also get the copy graph function to work. Anyway, that may take a while because I can't do it on this laptop (my excel files all on the old mainframe) and I have to manually override one line of data which was formula generated.

    As to the CET warming at end of 19th century, I don't think there was quite the same dip in the 1870s and 1880s as in North America, I guess with all that arctic air pouring south in those decades, it at least maintained some kind of trough over the central Atlantic even if the years were not overly warm. Even from Toronto to NYC there are a few occasions where this arctic dominated pattern seemed to weaken once into the northeastern states, June 1883 for example produced a much different ranking at the two locations with a storm track evident between the Great Lakes and the Atlantic coast.

    So without that dip in temperatures (which is evident in the Toronto data starting earlier than NYC) there was no low level from which to rebound in the 1890s, hence the difference in signal. The warmth from 1948 to 1955 seems to be a constant at all locations now including that Missouri data set. That may be in part a rebound from extreme cold in early 1947 since the warming probably began with the warm summer of 1947. This seems to be more and more of a feature, when there were extremely cold winters in the 19th century, it would usually be a longer and slower recovery, but any time we get very cold winters nowadays, the rebound is quick and substantial (thinking of the change from the polar vortex cold records in early 2015 to summer autumn and winter temperatures that followed, also winter 1994 that was very cold in N America had two very warm years right afterwards.


  • Registered Users Posts: 14,279 ✭✭✭✭M.T. Cranium


    Here's that simplified excel file showing only the annual means (with urban heat island adjustments for Toronto and NYC).

    Will see if I can copy the graph of the comparison of 11-yr running means, if not, this file opens up to that.

    Not getting the chart, only the type around the chart. Maybe somebody can p.m. me with the method for uploading a chart to a post here, I know how to do it on other forums.


  • Registered Users Posts: 6,235 ✭✭✭Oneiric 3


    This seems to be more and more of a feature, when there were extremely cold winters in the 19th century, it would usually be a longer and slower recovery, but any time we get very cold winters nowadays, the rebound is quick and substantial (thinking of the change from the polar vortex cold records in early 2015 to summer autumn and winter temperatures that followed, also winter 1994 that was very cold in N America had two very warm years right afterwards.

    This is an interesting observation M.T, because I have noticed this same trend with our own 'IMT' data in recent years, in that short, sharp cooler periods (summer 2015, late winter/early spring 2018 as just a couple of examples) are followed quite quickly by a period of great warmth (relatively speaking). This year has bucked that trend a little though. The longer cooler (rather than cold) Jan/Feb period has not, so far at least, been followed by a sustained period of warmth, and just looking at the series anomaly for this year (compared with the average of the last ten) we are still running about 0.5c below that normal for this part of the year.

    Thanks again for putting together all of that data and sharing it with us. I'll defo take a good look at it over the course of tomorrow as should have more time to actually do it.

    In the meantime, a shot of one of your graphs on the data sheet you posted, which I look forward to having a good look through tomorrow,

    3LPbS09.png


    Looks good. A way to upload a chart like that to a forum like this is to just use the 'Snip' tool in Windows (it can be found by just searching in the search bar on bottom taskbar) and then just highlight area you want to capture, such as the graph above, saving it as an image and then just upload it to a image hosting site like this:

    https://imgur.com/upload

    then just a matter of getting the uploaded image url (usually found by right clicking it) and sharing it on the likes of here.

    New Moon



  • Registered Users Posts: 14,279 ✭✭✭✭M.T. Cranium


    Thanks, will try that, as you may have noticed on Netweather, the simple copy function captures a graph there, here it just captures the text in the same range.

    That annualmeans microfile now has an interesting addition so I will repost it, the addition is a graph showing the average temperature of CET (weighted 1.0) and the adjusted for u.h.i. TOR and NYC (each weighted 0.5), so it's basically an average of comparable data sets with the two similar ones (for location) averaged out first.

    What this shows is (a) the 1890s warming is a general feature although stronger in North America, (b) 1921 was a relatively very warm year but had little impact on running means at that time, (c) there was a peak of relative warmth around the intervals 1944-54 or 1945-55, followed by a general decline to 1962-72 which ran about 0.4 C deg colder, then the much publicized modern warming which seemed to set in around the late 1980s and levelled off once the "modern" warming was established by about 2012. If you already downloaded "annualmeans.xlsx" that first version doesn't have this feature which I added in rows 41-42 with a graph.

    Another addition is a running 30-year average for the three locations (the graph posted in the range AM10 to AT24).

    I would add, that graph for Missouri is similar in most regards, except that the 1930s show up warmer, that summer heat while significant further east was more sustained in the central regions, it tended to come and go further east and when not present it could be quite cool reducing some of the summer month means to the 20-40 rank range, oddly the 1930s warming seemed to be mostly about summer and winter while springs and autumns were generally somewhat on the cool side of average after 1931 which did keep pace, just a somewhat different climatic episode, the late 1940s and early 1950s warming seemed to be more about sustained warmth and did particularly well with spring and autumn rankings.

    You do tend to see the same years in all these locations doing similar things sometimes, like 1917 being relatively cold.


  • Registered Users Posts: 6,235 ✭✭✭Oneiric 3


    MT, Attached (I hope) are the ECMWF monthly (from 1900 to 2010) reanalysis data from the Toronto region (not sure if the coordinates are exactly right for the actual station. Will take a look at this data myself later on to compare to the actual recorded data in your sheets, but for now, thought you might like to take a look:

    Data is generated from this site and edit: anomalies in the 'anom' sheet are based around the rather curious 79-10 averages that Climate Reanalyzer uses.

    https://climatereanalyzer.org/reanalysis/monthly_tseries/

    New Moon



  • Advertisement
  • Registered Users Posts: 14,279 ✭✭✭✭M.T. Cranium


    Thanks, will also take a look. Nearest lat-long point to Toronto downtown is 44N 79W ... would have to wonder if it's reanalysis if they factor in lake Ontario suppression in March through July, that's a cold body of water in the spring.

    Anyone who downloaded the Toronto file should go in and correct one cell, I was working on the area of rankings for UHI corrections just before posting and updated them for Jan 2021, which involved adding one to ranks higher than its rank. In that process, a number slipped (probably one of those instances where the computer freezes up and you think you just corrected cell A but the result ends up in cell B). So what we have to correct is cell HE1969 (Jan 2015), which shows as 3 (at least on my home version it did, maybe the change happened after I uploaded). If you see a 3 there, it needs to be changed to 136 and that will trigger one other cell change for an average of 2015 rankings.

    I went back through the chain and all other entries for Jan 2015 are unaffected. The rank in the raw data table is okay and the actual temps listed in all tables are correct. So that was the only place this error impacted and as there had been no graphics done for that table, nothing to note there.

    What I'm doing recently is to construct the same colour-coded display for the UHI corrected ranks as already exists for the raw data ranks. Then since some of those change (early ones show some increases, late ones some decreases) I outline the changed colour code cells to give an overview of how the urban heat island correction affects the look of the overall data. It certainly flattens out the initial "sharp increase" look without totally removing the overall increase.

    Anyone who has the file but finds the change more work than necessary can wait for the next seasonal update where this error will be corrected and the new graphic as mentioned will be available.

    I have a few quality control panels to try to avoid data contamination problems, as you'll know if you work on excel files, the odd time you can type some intended change in and not see it where you thought it should be, which means perhaps it fell into a cell you had been looking at earlier. That would contaminate your data. But with the quality control panels, I can check at regular intervals that nothing like that has happened. The ranking tables are going to be placed in a newly designed q.c. check system where out of place ranks will show up on a graphic display. Given that there's probably over a million data points in the file, the odd error is probably difficult to avoid, even the pros have errors in their files as we found out with the NYC data.

    That "3" cell must have shown up in the edit because the original work was done from tables with accurate ranking for Jan 2015 and I would have been typing numbers like 136 at that point, suspect that it was during a computer slowdown during the 2021 rank insertion updates. Have checked to see if any other obvious errors in the table, the new q.c. system should ensure that there are no significant errors (a lot of ties using the C data, and some random scatter of different ranks from the older F numbers although the higher and lower ranks tend to be immune to scrambling as they are separated out more).

    Speaking of NYC, if you downloaded that file, a very minor change can be made, the record high on 25 Jan (60, 1967) was not colour coded (red). The day before was also a record, so if you can't see how to achieve the colour code but want to make the change, just copy the cell for the 24th (68 with red highlight) down one cell, then change the number you see to 60. That minor error is located in cell DA25.

    There are some interesting month to month or season to season variations in the overall trends. One was that in the "climate warming" episode around the 1890s, June was off to a faster start than July or August in that warming. Another was that the more recent warming shows the opposite trend, July remaining closer to average than faster-warming June and August. The winters in the 1840s tended to be considerably milder in terms of rankings than the other three seasons, in fact after the UHI corrections the winters ended up looking fairly similar to the mid-20th century before the "modern warming" spike. Some of those observations are illustrated by graphs in the edited version that will be available after May ends.


  • Registered Users Posts: 14,279 ✭✭✭✭M.T. Cranium


    Oneiric 3 wrote: »
    MT, Attached (I hope) are the ECMWF monthly (from 1900 to 2010) reanalysis data from the Toronto region (not sure if the coordinates are exactly right for the actual station. Will take a look at this data myself later on to compare to the actual recorded data in your sheets, but for now, thought you might like to take a look:

    Data is generated from this site and edit: anomalies in the 'anom' sheet are based around the rather curious 79-10 averages that Climate Reanalyzer uses.

    https://climatereanalyzer.org/reanalysis/monthly_tseries/

    I had a look at the raw data, turns out to be somewhat colder in general for January, their mean for the 111 years sampled was -7.6, the data average -4.7 and adjust to -5.5 after urban heat island removed.

    The warmest January (1932) was 1.9 in raw data, 1.3 adjusted and -0.6 in their data set. A recent very cold January (1994) was -10.0 (-11.1 adjusted) and the data set gives -12.2.

    So they seem just about perfect for a non-urban point in the general area somewhat further inland from Lake Ontario, I would imagine all of those data points would fit actual readings made in a farmer's field 10 miles outside the suburban limits. I haven't looked at other months or their anomalies.

    I would say they are safe to use for any purpose, not sure what to say about a 1979-2010 base for anomalies, it can't be all that different from 1981-2010. Why do the data sets end in 2010?

    Meanwhile if anyone is correcting their downloaded data sheets, relatively good news on the ranking quality control, I did that task and found only one other suspect rank, cell EG1941 (which refers to raw data for Sept 1939) got a bad edit somewhere and went from t-86 to t-7 (to be clear it should be t-86 or 86 inside a grey highlight box). Probably the same rigamarole as with the other egregious error. So that one can be changed manually and it will result in a couple of other cells changing. Once again, all data in tables and other places in the excel file remain valid for that month. I suspect it's an edit error because the colour code was right in the template of temperature anomaly groups by thirds.

    No other monthly ranks are mismatched with either the temperature series or the other ranking table now.

    If this does not concern you enough to bother changing it, the next edition will of course have that error corrected, and I now have a q.c. table installed to monitor for any unseen changes in the future.

    In recent weather news, March 11 set a record at Toronto (66 F) and March 26 set a record at NYC (82 F) and there are several other new records at both locations for tied or new high mins. The month will end up around 20th warmest at both locations, raw data and 30th adjusted. There will be only traces of snow which is below the March average for both locations but not all that unusual for mild cases.


  • Registered Users Posts: 6,235 ✭✭✭Oneiric 3



    I would say they are safe to use for any purpose, not sure what to say about a 1979-2010 base for anomalies, it can't be all that different from 1981-2010. Why do the data sets end in 2010?

    .

    This was a typo on my part M.T, the climate based used by C.R is the 1979-00 and not the 1979-10 that I said. I have very awkward hands when typing (and when not). :o

    I will get that ECMWF database for the co-ordinates you gave for downtown Toronto to see if there is any difference between those of the raw data and the data I posted from the greater Toronto region. Not sure why this data set ends in 2010, but later editions of this reanalysis data go further out, which should be accessible in that CR link I posted.

    New Moon



  • Registered Users Posts: 14,279 ✭✭✭✭M.T. Cranium


    Those co-ordinates are not carved in stone, what's the grid tolerance, nearest degree or 0.1 degree? The exact lat long as given in the EC source is 43.67N and 79.4W.

    What exactly is re-analysis, estimates from maps? The maps from before 1945 would have no upper air measurements and any recreated maps showing that are based partly on actual temperatures; my data base has probably the only actual temperatures near the grid point before the airport opened in 1938, so really this is just a sort of estimate of what my data base applied to estimated maps comes out as after the fact. But I do think it correlates at a very high level from what I checked so far, mainly the January month by month and the more extreme values of other months, don't see any outliers. It's all a bit on the cold side of the actual or even urban-adjusted numbers.


  • Registered Users Posts: 6,235 ✭✭✭Oneiric 3


    Those co-ordinates are not carved in stone, what's the grid tolerance, nearest degree or 0.1 degree? The exact lat long as given in the EC source is 43.67N and 79.4W.

    What exactly is re-analysis, estimates from maps? The maps from before 1945 would have no upper air measurements and any recreated maps showing that are based partly on actual temperatures; my data base has probably the only actual temperatures near the grid point before the airport opened in 1938, so really this is just a sort of estimate of what my data base applied to estimated maps comes out as after the fact. But I do think it correlates at a very high level from what I checked so far, mainly the January month by month and the more extreme values of other months, don't see any outliers. It's all a bit on the cold side of the actual or even urban-adjusted numbers.

    I've attached data from the 'ERA 4th Generation' for the coordinates you specified for Toronto in an early post. The data only goes from 1950 up to the present year (on a monthly basis) and the resolution is stated as being 0.5 by 0.5 degrees.

    And I really wish this platform would sort out its idioitic attachment system. Far too fiddly and unnecessarily complicated for my already very limited patience to endure.

    New Moon



  • Registered Users Posts: 6,235 ✭✭✭Oneiric 3


    M.T. I was looking through your Toronto data sheet and just want to confirm something with you, and that is, is this section here there actual 'raw' daily readings?

    3KUygUS.png

    I just want to be sure before I tidy it up (those God foresaken leap days should be completely abolished imo) and run it through some code to compare it with the data I posted up above just earlier.

    New Moon



  • Registered Users Posts: 14,279 ✭✭✭✭M.T. Cranium


    Yes those are raw daily means, but the monthly averages derived from them can be found in another part of the spreadsheet, they were averaged by EC and should all be within 0.1 rounded of what you would get from the work on those daily numbers. Also my tables for Toronto run top to bottom without a space for missing 29 Feb (the NYC tables have that feature) so that formulae for some columns would be different than others. Here's the grid references (range) for the monthly averages in the spreadsheet.

    They are located in rows 1840 to 2021 to make it easier to figure out to which year the data belong. The columns in play are HW to II. That gives you Jan to Dec and annual means. Unfortunately I did not block the data to convert to one decimal for round numbers, e.g. you'll see April 1842 says 7 which is 7.0.

    Let me know if you find this data section and have any questions. The website data are derived from their (EC) averages of monthly mean max and mean min rather than directly from averages of mean daily. It won't make much difference but some of these data would be 0.1 different from what you would get by taking the average of daily means. Once again, my table of daily means is derived from their conversions of the original F data whereas the tables above (the ones that are in F deg) are calculated by me from the entries in the original grids.

    If you still wanted to average my data rather than using this table, be aware then that each column runs unbroken from 1 Jan to 31 Dec so that leap years will run from (a) days 1232-1260 for February and (b) days 1261-1291 for March compared to 1260-1290 for non-leap years (incl 1900), etc to end of columns.

    My averages at the end of each row for dates have the formulae worked out for that. I suppose I should standardize this work with the NYC tables which I altered for missing leap year dates so that all similar dates of the year after Feb 28 are in the same row. Maybe the next version will have that built in. I'm nervous about throwing off formulae although they should convert along with the shifts.

    Also minor point but the -7 and -3 for Jan and Feb 1840 are just rough guesses based on data from Providence RI, there was no weather station at Toronto until 1st of March 1840.

    By the way, if you're transcribing these numbers into an excel file, then they also exist in print in the net-weather thread, in the second post of the thread.

    You asked if they were raw data and the answer is yes, including those monthly means I mentioned.

    For comparison with other data sets, I would recommend applying the urban heat island correction as follows: data 1840 to 1880 is okay as is. Data 1881 to 1890 reduce by 0.1 (to colder values), data 1891 to 1900 reduce 0.2, etc until you reach 1971 to 1980 which would reduce by 1.0 (C deg). After that 1981 to present all reduce by 1.1 C.

    If it would save you some work, I could post an excel file of those conversions, that does not yet exist in the excel file but it does exist in print in the net-weather thread, this time on page two after the data on dry spells is tabulated (go to page two of the thread, second post).

    If the purpose is to compare with CET or Irish data, I would prefer that these adjusted values would be compared, otherwise there will appear to be a spurious warming trend in the Toronto data over the 20th century into recent decades. If it's to compare with the re-analysis, I don't know what to recommend because I don't know if they factored into their numbers any result of the change in the environment from small city (in 1900) to metropolis (after 1960). By 1900 Toronto was already about a quarter million people with the weather station near the centre of the core, so an urban heat island would already be in place then. It would have grown approximately as suggested by my corrections thereafter. By about 1960 Toronto had surpassed two million population and had spread out a good 20-25 miles beyond its 1900 extent. Changes since then have been relatively minor with most of the growth out beyond the 1960 suburban belt, and tending to cluster rather than run out unbroken, due to greenbelt and river valley intrusions. So I don't think the urban heat island grew very much after about 1950-1960 and the data suggest that also. I used to run my own backyard weather station in a perfect spot for urban-rural comparisons (30 miles west of the downtown station) so I had about ten years of data for comparison, although the location where Toronto downtown is located would be a bit warmer than my location anyway due to lower elevation and closer proximity in winter to Lake Ontario. A sample of some very cold clear night readings would average about 6-8 F deg colder at the MTC-jr location. And that was in a small heat island of a smaller town, really rural settings might be another 4-6 F deg colder again. Most daytime readings were 1-2 deg lower there. However that was complicated by cooling lake breezes that would be just about extinct at my location inland, so some days in March to June in particular would run cooler in the city. It looked to me like that season extended into July more frequently in the 1840s and 1850s before there was any real urban development to weaken the lake breezes (and probably after some of those winters the lake was pretty cold into the middle of summer too).


  • Registered Users Posts: 14,279 ✭✭✭✭M.T. Cranium


    By the way the main reason why I didn't convert daily data to the same system as the monthly/annual data would be that I don't believe the differentials would apply unbroken to all days, the cool, cloudy and wet days would be reduced less than the warm, dry days especially those in modified polar air masses in warm seasons. So it would be a complicated process to change all the daily data. It would probably result in a more variable adjustment with the cooler, wetter months adjusting less than warm, dry ones.

    It's unfortunate that no particular weather station in the nearby rural areas has an unbroken data set for any long interval, they just came and went with various decades covered. So there's nothing absolute that I can use for a comparison. There is a long record at Kingston, Ontario at the east end of the lake, but it's probably too far from Toronto to provide the right kind of corrections, with the chance of anomaly regime differences; also that's a very lake-influenced location and despite being on the lake, Toronto only has limited intrusions of cooler air when winds are southeast to east. Kingston is open to southwest winds (which prevail in these climates) off the lake.


  • Registered Users Posts: 14,279 ✭✭✭✭M.T. Cranium


    Maybe then to save you some work just specify what time interval you want to have monthly means, and whether or not they should be raw data or adjusted to urban heat island reductions. I could post the file of just that information probably within an hour as it would not require much work here. Presumably if the data are in a dedicated excel file, you could then import said block of data into another file where you have the other data for comparison?

    (later developments ...)

    I created this file for you and anyone else who wants to use it, only the Celsius monthly means for Toronto are in this one, two sets of data, first the raw data as recorded, and second set, adjusted for urban heat island (so progressively larger amounts subtracted as explained).

    This should save you some work. If you do happen to average out my daily C numbers, you'll likely find them similar but occasionally off by 0.1 for the reasons outlined above. I am making up a table of that parameter to compare.


  • Registered Users Posts: 6,235 ✭✭✭Oneiric 3


    Yes those are raw daily means, but the monthly averages derived from them can be found in another part of the spreadsheet, they were averaged by EC and should all be within 0.1 rounded of what you would get from the work on those daily numbers. Also my tables for Toronto run top to bottom without a space for missing 29 Feb (the NYC tables have that feature) so that formulae for some columns would be different than others. Here's the grid references (range) for the monthly averages in the spreadsheet.

    They are located in rows 1840 to 2021 to make it easier to figure out to which year the data belong. The columns in play are HW to II. That gives you Jan to Dec and annual means. Unfortunately I did not block the data to convert to one decimal for round numbers, e.g. you'll see April 1842 says 7 which is 7.0.

    Let me know if you find this data section and have any questions. The website data are derived from their (EC) averages of monthly mean max and mean min rather than directly from averages of mean daily. It won't make much difference but some of these data would be 0.1 different from what you would get by taking the average of daily means. Once again, my table of daily means is derived from their conversions of the original F data whereas the tables above (the ones that are in F deg) are calculated by me from the entries in the original grids.

    If you still wanted to average my data rather than using this table, be aware then that each column runs unbroken from 1 Jan to 31 Dec so that leap years will run from (a) days 1232-1260 for February and (b) days 1261-1291 for March compared to 1260-1290 for non-leap years (incl 1900), etc to end of columns.

    My averages at the end of each row for dates have the formulae worked out for that. I suppose I should standardize this work with the NYC tables which I altered for missing leap year dates so that all similar dates of the year after Feb 28 are in the same row. Maybe the next version will have that built in. I'm nervous about throwing off formulae although they should convert along with the shifts.

    Also minor point but the -7 and -3 for Jan and Feb 1840 are just rough guesses based on data from Providence RI, there was no weather station at Toronto until 1st of March 1840.

    By the way, if you're transcribing these numbers into an excel file, then they also exist in print in the net-weather thread, in the second post of the thread.

    You asked if they were raw data and the answer is yes, including those monthly means I mentioned.

    For comparison with other data sets, I would recommend applying the urban heat island correction as follows: data 1840 to 1880 is okay as is. Data 1881 to 1890 reduce by 0.1 (to colder values), data 1891 to 1900 reduce 0.2, etc until you reach 1971 to 1980 which would reduce by 1.0 (C deg). After that 1981 to present all reduce by 1.1 C.

    If it would save you some work, I could post an excel file of those conversions, that does not yet exist in the excel file but it does exist in print in the net-weather thread, this time on page two after the data on dry spells is tabulated (go to page two of the thread, second post).

    If the purpose is to compare with CET or Irish data, I would prefer that these adjusted values would be compared, otherwise there will appear to be a spurious warming trend in the Toronto data over the 20th century into recent decades. If it's to compare with the re-analysis, I don't know what to recommend because I don't know if they factored into their numbers any result of the change in the environment from small city (in 1900) to metropolis (after 1960). By 1900 Toronto was already about a quarter million people with the weather station near the centre of the core, so an urban heat island would already be in place then. It would have grown approximately as suggested by my corrections thereafter. By about 1960 Toronto had surpassed two million population and had spread out a good 20-25 miles beyond its 1900 extent. Changes since then have been relatively minor with most of the growth out beyond the 1960 suburban belt, and tending to cluster rather than run out unbroken, due to greenbelt and river valley intrusions. So I don't think the urban heat island grew very much after about 1950-1960 and the data suggest that also. I used to run my own backyard weather station in a perfect spot for urban-rural comparisons (30 miles west of the downtown station) so I had about ten years of data for comparison, although the location where Toronto downtown is located would be a bit warmer than my location anyway due to lower elevation and closer proximity in winter to Lake Ontario. A sample of some very cold clear night readings would average about 6-8 F deg colder at the MTC-jr location. And that was in a small heat island of a smaller town, really rural settings might be another 4-6 F deg colder again. Most daytime readings were 1-2 deg lower there. However that was complicated by cooling lake breezes that would be just about extinct at my location inland, so some days in March to June in particular would run cooler in the city. It looked to me like that season extended into July more frequently in the 1840s and 1850s before there was any real urban development to weaken the lake breezes (and probably after some of those winters the lake was pretty cold into the middle of summer too).

    Thanks for this and the others well detailed info MT. I was eventually going to compare this Toronto data with that of the CET, but my priority for now is to compare it with its model reanalysis equivalent just to see how well the reanalysis (EC 4th Gen) has done in capturing the relatively recent (since 1950) historic temperature trends in that region. I did, a number of years ago, do similar for the 'IMT' series and while general broad scale trends from the EC grib data were more or less similar with the raw data, actual values rarely were, particularly regarding daily and monthly extremes (Max/Min/wind/pressure etc)

    New Moon



  • Advertisement
  • Registered Users Posts: 14,279 ✭✭✭✭M.T. Cranium


    Okay, so in that case I would suggest comparing the second data table generated (urban heat island adjusted) with the CET, and both with the reanalysis to see what you think of each of them.

    I did construct a table of my file means in C and found the expected slight scatter within 0.1 from the official means, with about 80% overlap. However, there were no outliers at all 1991-2010 which made me wonder if they switched their methodology in 1991 (as they would get the same values as me assuming I had no typos from their tables which are the sources of the C data).

    On the annual scale there is already a comparison graph available in the excel file for annual means and CET annuals, GL produced a graphic from that. Since it compares raw data it shows the Toronto means catching up to the CET, nowadays there is little difference, back before the city was very large, it was often 1 to 2 C deg colder than CET. The urban adjusted data (on a 5 yr rm basis) follow the CET fairly closely and remain in that range of 1 to 2 deg colder.

    The peaks of warmth around 1921 and 1948-53 are similar, as is the warming from about 1987 to 2012. A few minor differences can be seen also, the rise in the 1890s is a bit more prominent for Toronto.

    I have not gotten around to comparing months yet, that might reveal relationships which we don't see from just the annual data.


  • Registered Users Posts: 14,279 ✭✭✭✭M.T. Cranium


    I am adding some interesting data to the netweather thread, basically it is a continuous updating of record highs from a set of "starter highs" for 1869-1900, year by year, to put one into the mind set of the person living in those past years, what they considered to be record breaking, and how it all fits together in terms of frequency ... interesting in several ways, including the comparison with UK/Ireland weather which was often similar and sometimes opposite.

    This is complete through 1940 and under development for the rest of the period.

    From this I have already made a new discovery (new to me anyway) about pulses of warming in the past.

    It will take a while to complete the study but I have the preliminary numbers which get adjusted as I check through the data. What that shows is that for the whole period since the late 19th century, there have been regular pulses of warmer weather at intervals of about 15-16 years until the 1970s, then this pulse seemed to accelerate and its period since then has been 11-12 years.

    While it sounds like it should have a connection to solar variation, the actual plot of the curves shows that basically there was no strong correlation, the longer period was hitting at various times in solar cycles that peaked between 1905 and 1947, then it began to phase with them after the 1979 cycle, but was totally out of sync during the strong solar cycles of 1947-49, 57-59 and 68-72.

    Not sure what to make of this pulse, it seems to have an origin in either the Pacific or the Gulf of Mexico, or both acting together. Those are source regions for warmth impacting the NYC region, the Atlantic is closer but record breaking warmth reaching NYC usually tracks from around Oklahoma and Kansas east-northeast through the Ohio valley. And two times out of three, a warm pulse hitting this region will show up 3-5 days later over western Europe. This is not a guaranteed 1:1 result, some warm peaks in eastern North America never show up further east. The 1936 heat wave for example did not have any later response feature in Europe, it was cool and unsettled for weeks after that.

    None of this really tells us much about the human component of climate change. It could range from any conclusion between no connection to major driving factor, but I suspect that the climate of NYC since 1890 looks quite similar to what it would have been on a planet without human advanced civilization, and then if we accept that the greenhouse effect and urban heat islands are two different outcomes of human activity, and take away the urban heat island from the temperature record, that leaves us with a base climate signal that may or may not have been modified by human activity. I suspect it has been modified by some fraction of a degree (the urban heat island is larger). But these pulses may be an entirely natural phenomenon -- they seem to correspond to larger El Nino events.

    Despite all the research that has been done, I don't think anyone would be prepared to offer an "advanced" theory of how human activity drives a stronger Pacific oscillation (or how it leads to stronger El Ninos). If anyone thinks otherwise, I'd be interested in hearing the reasoning, and I am quite open minded about all these issues, my main position is that I don't know as much as I would like to know, and you can't make predictions without a knowledge of predictive theory (not on a regular sustainable basis anyway).

    Now here's a finding that will surprise some. The ten-year period with the most sustained daily records (the ones that exist as of now) is not that recent, it is 1944 to 1953. The frequency was 4.5 per year (random would be 2.5 over the 152 year record being studied). A secondary peak was 1993-2002 (4.1). A slight tertiary peak (2.9) was 2006-15. The pace in 2019-2020 and first third of 2021 is about equal to random expectation.


  • Registered Users Posts: 14,279 ✭✭✭✭M.T. Cranium


    That study is getting into the recent past now, have pretty much finished the 1960s and everything before 1960 is done (apart from any editing that may be needed). For NYC, the period 1944 to 1955 was probably just about as warm as the "modern warming" especially if we accept that the urban heat island was only at about 80% of its eventual power to warm overnight temperatures. About 15% of all eventual record highs fall into this twelve year window (which is 8% of the time available). It's interesting now for me because I'm old enough to recall the weather of the 1960s, in fact I was in NYC on New Years eve of 1965 which produced the daily record, we had family there and sometimes visited from our home in Ontario, travel was by bus as I wasn't yet a licensed driver (later in 1966 I was and that was the last of the bus trips).

    The summer of 1966 was a scorcher both in NYC and back home in Ontario, very low humidity compared to some of the heat waves we got over time. You'll see reference to a late August early September 1973 heat wave, that one I experienced from many different locations as I was on a road trip with some visiting UK relations, coming back from the western U.S. through Arkansas and Tennessee where it was not only that hot but way more humid than what I was used to seeing, like 82-84 F dewpoints. We drove through Washington DC maybe first or second of September on the way to visit those same NYC relations, and it was about 102 F there. That heat is worse than 117 F experienced at Las Vegas in August 2011 (it's a dry heat and fine as long as you don't touch anything metallic). And it cools off to about 90F by morning (in Las Vegas, almost tolerable for the first hour or so of daylight).

    Will be coming to some epic warm spring weather in the late 1970s in the next day or two, and hoping to have this whole historical "deep dive" finished by about Monday or Tuesday with the more recent stuff that I missed out on by moving west in 1995. Looks like that was a very good idea from some of these data.


  • Registered Users Posts: 14,279 ✭✭✭✭M.T. Cranium


    Will be posting the updated versions of the excel files on Sunday night or Monday morning.

    May 2021 turned out to be a slightly warmer than average month at both locations, fairly close to long term average after applying the u.h.i. corrections.

    It was running quite warm to the 27th then much colder air arrived along with a coastal low which brought heavy rain to NYC, a moderate amount to Toronto where it mixed with snow to provide the fourth latest trace of snow on record at the downtown location (1859, 1945, 1856 later in the range of 31 May to 4 June). The suburban airport location had its latest measurable snow on 28th (0.2 cm).

    Record highs were set in the period 20th to 25th at Toronto although they tended to be weak records barely edged out (not every day set one). NYC came within one degree of a high min record in this period.

    The spring of 2021 turned out to be the 11th warmest for Toronto, settling at 15th warmest after u.h.i. adjustments. Even so it was more than 2 deg cooler than 2010 or 2012.

    Updated excel files will be available as soon as I finish tidying up some new sections created between 1st of March and today. These track the records for NYC and most of that information is also in the thread now. You can see what the records looked like at the turn of the 20th century, and various points in between, as well as getting the overview of all records set.

    For record highs at NYC, I found that the interval 1944-53 had the highest production rate of eventual record highs, quite a bit higher than any ten years since, with a secondary maximum 2002 to 2011.


Advertisement