[M.T. Cranium]

I have recently posted excel files sharing data bases for Toronto (1840-2021) and NYC (1869-2021) in a thread over on Net-weather. I am going to see if these excel files will upload directly here, if not, I will post the link to the thread where they are available.

I would also be quite glad to share the tables posted in that same thread, except that on Net-weather I have no time limits on editing, so I can maintain the ranking tables and make corrections where necessary, whereas here I would need to cut a deal with a friendly moderator or (worse still) become one.

Don't want that added workload, but could be a one-dimensional moderator just editing my own climate threads.

Anyway, going ahead with the experiment to upload these excel files on boards, will go to plan (b) if they don't upload.

So it's plan (b) unfortunately my files are too large for the boards limit.

Here's where you will find the two data files plus all the tables of ranked data that I posted separately. The download files are in the last two posts made by me (exclusive of any chat that develops as I only just uploaded them).

https://www.netweather.tv/forum/topi...omment-4474159

[Oneiric 3]

NYC long term daily average temp profile from MT's data:



Very similar trend to Ireland's, in that temps peak in the second half of July/early August and are at their coolest in January & February. Not similar to Ireland's though are the actual temps, which are far more extreme in the warmest and coldest seasons.

25th/75th are percentiles and further edit: date on chart should read 2021 and not 2020.

Unlike Ireland, however, there doesn't seem to be a notable 'drier' period in NYC long term data base. Rainfall evenly distributed throughout the year, with perhaps the winter months less prone to heavier rainfalls that the warmer ones:





Edit: Above temperature graph in a more digestible form:




Data Source © M.T. Cranium.

[Oneiric 3]

Here is the running ten year mean average temp at NYC.



All in accordance with global trends, but... there was a very sharp rise in temps at NYC during the last decade of the 19th century (highlighted with yellow marker) which is on par with, or even greater, in terms of magnitude than the rise over the following 120 years. Very curious.

[M.T. Cranium]

Thanks for those samplers, lots to see in these two data bases.

The NYC temp rise in the 1890s can also be seen at Toronto. Part of it may be a natural warming that followed the generally cold decades of the 1870s-1880s. Another factor is an increasing urban heat island, not sure how to estimate the actual values during the pre-automotive era but some other aspects of an urban heat island were valid even then. Also the numbers we are working with are adjusted means rather than the raw data base means, you can check out the differentials in the range G1869 to AS 2021 in the NYC data base. Those changes were made by the NWS based on (presumably) considerations of site exposure, climatological day, and other unknown (to me) factors.

One factor that I thought might explain the fast rise in the 1890s would be that the north magnetic pole was then starting to move steadily north so if there were any correlation between magnetic field and circulation, the jet stream would probably tend to migrate north at that time. Once past a critical change, further temperature responses would be smaller since the main work would be to reduce arctic air mass frequency.

The 1890s at both locations produced a lot of variations, the climate seemed to be in an unstable mode. For example, bitter record cold in Feb 1895 and record lows at times in May 1895 suddenly followed by record heat at the end of May and start of June 1895. A rather similar rebound from winter cold to spring heat in 1896.

Oneiric, if you had some time to look, the graph you produced is presumably from the raw temperature averages, but there is a column of adjusted (for u.h.i.) means further over to the right of that one (something like BR 1869 to 2020). That would be a more "climate sensitive" graph of the rising temperature trend as it would eliminate the 1.1 C assigned to the urban heat island by 1980 (and from there to end of series). Alternatively, I might throw in a graph below of that same trend, if it works to copy the graph and post it. Let's see ...

That procedure doesn't seem to work here, I have done that on other sites (copy a graph from an excel file to here), but I had a look at the relevant data and in the 1890s you'll see in the comparison table that the NWS have generally increased the raw data averages by 0.3 to 0.4 C each year in that decade. I also verified that the graph posted above is pre-urban heat island adjustment official data, the values in the next post down are the annual numbers and 11-yr running means for the adjusted data. This is what should be compared to CET or other data sets that are certified to be free of urban heat island influence.

[M.T. Cranium]

Mean annual temps (NYC) 1869 to 2020, adjusted for urban heat island

The adjustments are 0.1 C for 1881-1890, 0.2 for 1891-1900, etc for each decade, ending with 1.1 for 1981-2020 when it is assumed that the urban heat island has reached a stable maximum level. So these numbers are lower than the official numbers by those amounts from 1881 to end of series (same for 1869 to 1880). The second value is the running 11-year mean.

These data are in BR-BS 1869-2020 of the NYC file.

1869 10.78
1870 12.00
1871 10.61
1872 10.56
1873 10.56
1874 10.72 10.98
1875 09.67 11.07
1876 11.06 11.00
1877 11.56 11.03
1878 12.00 11.00
1879 11.28 11.06
1880 11.72 11.04
1881 11.23 11.11
1882 11.01 11.05
1883 10.18 10.87
1884 11.23 10.81
1885 10.51 10.82
1886 10.46 10.84
1887 10.40 10.81
1888 09.51 10.72
1889 11.40 10.82
1890 11.40 10.83
1891 11.91 10.94
1892 10.86 11.05
1893 10.02 11.23
1894 11.36 11.44
1895 11.30 11.49
1896 11.74 11.44
1897 11.69 11.38
1898 12.36 11.41
1899 11.80 11.42
1900 11.91 11.44
1901 10.87 11.53
1902 11.31 11.47
1903 11.20 11.54
1904 10.09 11.47
1905 11.53 11.44
1906 12.31 11.40
1907 11.09 11.42
1908 12.48 11.52
1909 11.59 11.48
1910 11.48 11.61
1911 11.49 11.54
1912 11.10 11.33
1913 12.38 11.36
1914 10.71 11.29
1915 11.54 11.23
1916 10.82 11.29
1917 09.99 11.29
1918 11.38 11.29
1919 11.71 11.12
1920 10.88 11.19
1921 12.22 11.06
1922 11.44 11.11
1923 11.11 11.25
1924 10.56 11.29
1925 11.39 11.31
1926 10.17 11.47
1927 11.39 11.48
1928 11.50 11.50
1929 11.78 11.50
1930 12.00 11.55
1931 12.62 11.54
1932 12.29 11.70
1933 11.73 11.78
1934 11.07 11.83
1935 11.12 11.71
1936 11.29 11.71
1937 11.90 11.61
1938 12.29 11.53
1939 12.01 11.54
1940 10.46 11.59
1941 12.02 11.69
1942 11.58 11.70
1943 11.36 11.67
1944 11.86 11.75
1945 11.58 11.68
1946 12.24 11.82
1947 11.41 11.86
1948 11.58 11.99
1949 13.13 12.05
1950 11.30 12.04
1951 11.98 12.00
1952 12.42 12.00
1953 13.09 11.92
1954 11.92 11.98
1955 11.81 11.83
1956 11.14 11.88
1957 12.20 11.79
1958 10.59 11.67
1959 12.20 11.54
1960 11.42 11.50
1961 11.93 11.51
1962 10.99 11.47
1963 11.10 11.40
1964 11.66 11.51
1965 11.43 11.44
1966 11.93 11.48
1967 10.77 11.42
1968 11.38 11.54
1969 11.77 11.59
1970 11.49 11.59
1971 11.89 11.55
1972 11.17 11.50
1973 12.39 11.49
1974 11.61 11.56
1975 11.72 11.56
1976 10.89 11.59
1977 11.39 11.56
1978 10.67 11.66
1979 12.17 11.62
1980 11.78 11.65
1981 11.79 11.66
1982 11.62 11.74
1983 12.23 11.76
1984 11.96 11.80
1985 11.96 11.86
1986 11.84 11.97
1987 11.73 11.90
1988 11.57 11.93
1989 11.12 11.89
1990 12.90 11.88
1991 12.90 11.79
1992 11.07 11.74
1993 11.96 11.84
1994 11.79 11.93
1995 11.84 11.93
1996 10.96 11.88
1997 11.29 11.84
1998 12.90 11.81
1999 12.51 11.76
2000 11.07 11.79
2001 12.40 11.87
2002 12.46 11.94
2003 10.79 11.99
2004 11.40 11.82
2005 12.12 11.83
2006 12.68 11.96
2007 11.68 12.01
2008 11.84 11.96
2009 11.12 12.01
2010 12.62 12.12
2011 12.46 12.19
2012 12.96 12.16
2013 11.84 12.21
2014 11.34 12.22
2015 12.62 12.39
2016 12.90
2017 12.40
2018 12.18
2019 12.01
2020 12.96
_______________________________

[Oneiric 3]

Thanks for all that M.T. I'll take a further look at the data you produced later on this evening and perhaps even look at the CET data also to see if there was a similar trend here in NW Europe at the end of the 19th century (but no doubt you yourself have probably already investigated)

After I posted last night, I was looking around on the internet to see if there was any long term datasets in the US from more rural regions. Came across this site and although text data seems to be lacking (unless I just cannot find them) a few nice graphs are available:

http://climate.missouri.edu/charts.php

Sample:



Just 'eyeballing' that single graph from the heart of the N American continent, and it is clear that something cyclical is going on regarding longer term temp trends. Just a pity we don't have the actual data from this spot to do further work with.

[M.T. Cranium]

I'm just working on a small excel file with only the annual means for the three locations, and running means. That should be small enough to post here and maybe if I can post it I can also get the copy graph function to work. Anyway, that may take a while because I can't do it on this laptop (my excel files all on the old mainframe) and I have to manually override one line of data which was formula generated.

As to the CET warming at end of 19th century, I don't think there was quite the same dip in the 1870s and 1880s as in North America, I guess with all that arctic air pouring south in those decades, it at least maintained some kind of trough over the central Atlantic even if the years were not overly warm. Even from Toronto to NYC there are a few occasions where this arctic dominated pattern seemed to weaken once into the northeastern states, June 1883 for example produced a much different ranking at the two locations with a storm track evident between the Great Lakes and the Atlantic coast.

So without that dip in temperatures (which is evident in the Toronto data starting earlier than NYC) there was no low level from which to rebound in the 1890s, hence the difference in signal. The warmth from 1948 to 1955 seems to be a constant at all locations now including that Missouri data set. That may be in part a rebound from extreme cold in early 1947 since the warming probably began with the warm summer of 1947. This seems to be more and more of a feature, when there were extremely cold winters in the 19th century, it would usually be a longer and slower recovery, but any time we get very cold winters nowadays, the rebound is quick and substantial (thinking of the change from the polar vortex cold records in early 2015 to summer autumn and winter temperatures that followed, also winter 1994 that was very cold in N America had two very warm years right afterwards.

[M.T. Cranium]

Here's that simplified excel file showing only the annual means (with urban heat island adjustments for Toronto and NYC).

Will see if I can copy the graph of the comparison of 11-yr running means, if not, this file opens up to that.

Not getting the chart, only the type around the chart. Maybe somebody can p.m. me with the method for uploading a chart to a post here, I know how to do it on other forums.

[Oneiric 3]

Originally Posted by M.T. Cranium

This seems to be more and more of a feature, when there were extremely cold winters in the 19th century, it would usually be a longer and slower recovery, but any time we get very cold winters nowadays, the rebound is quick and substantial (thinking of the change from the polar vortex cold records in early 2015 to summer autumn and winter temperatures that followed, also winter 1994 that was very cold in N America had two very warm years right afterwards.

This is an interesting observation M.T, because I have noticed this same trend with our own 'IMT' data in recent years, in that short, sharp cooler periods (summer 2015, late winter/early spring 2018 as just a couple of examples) are followed quite quickly by a period of great warmth (relatively speaking). This year has bucked that trend a little though. The longer cooler (rather than cold) Jan/Feb period has not, so far at least, been followed by a sustained period of warmth, and just looking at the series anomaly for this year (compared with the average of the last ten) we are still running about 0.5c below that normal for this part of the year.

Thanks again for putting together all of that data and sharing it with us. I'll defo take a good look at it over the course of tomorrow as should have more time to actually do it.

In the meantime, a shot of one of your graphs on the data sheet you posted, which I look forward to having a good look through tomorrow,




Looks good. A way to upload a chart like that to a forum like this is to just use the 'Snip' tool in Windows (it can be found by just searching in the search bar on bottom taskbar) and then just highlight area you want to capture, such as the graph above, saving it as an image and then just upload it to a image hosting site like this:

https://imgur.com/upload

then just a matter of getting the uploaded image url (usually found by right clicking it) and sharing it on the likes of here.

[M.T. Cranium]

Thanks, will try that, as you may have noticed on Netweather, the simple copy function captures a graph there, here it just captures the text in the same range.

That annualmeans microfile now has an interesting addition so I will repost it, the addition is a graph showing the average temperature of CET (weighted 1.0) and the adjusted for u.h.i. TOR and NYC (each weighted 0.5), so it's basically an average of comparable data sets with the two similar ones (for location) averaged out first.

What this shows is (a) the 1890s warming is a general feature although stronger in North America, (b) 1921 was a relatively very warm year but had little impact on running means at that time, (c) there was a peak of relative warmth around the intervals 1944-54 or 1945-55, followed by a general decline to 1962-72 which ran about 0.4 C deg colder, then the much publicized modern warming which seemed to set in around the late 1980s and levelled off once the "modern" warming was established by about 2012. If you already downloaded "annualmeans.xlsx" that first version doesn't have this feature which I added in rows 41-42 with a graph.

Another addition is a running 30-year average for the three locations (the graph posted in the range AM10 to AT24).

I would add, that graph for Missouri is similar in most regards, except that the 1930s show up warmer, that summer heat while significant further east was more sustained in the central regions, it tended to come and go further east and when not present it could be quite cool reducing some of the summer month means to the 20-40 rank range, oddly the 1930s warming seemed to be mostly about summer and winter while springs and autumns were generally somewhat on the cool side of average after 1931 which did keep pace, just a somewhat different climatic episode, the late 1940s and early 1950s warming seemed to be more about sustained warmth and did particularly well with spring and autumn rankings.

You do tend to see the same years in all these locations doing similar things sometimes, like 1917 being relatively cold.

[Oneiric 3]

MT, Attached (I hope) are the ECMWF monthly (from 1900 to 2010) reanalysis data from the Toronto region (not sure if the coordinates are exactly right for the actual station. Will take a look at this data myself later on to compare to the actual recorded data in your sheets, but for now, thought you might like to take a look:

Data is generated from this site and edit: anomalies in the 'anom' sheet are based around the rather curious 79-10 averages that Climate Reanalyzer uses.

https://climatereanalyzer.org/reanal...nthly_tseries/

[M.T. Cranium]

Thanks, will also take a look. Nearest lat-long point to Toronto downtown is 44N 79W ... would have to wonder if it's reanalysis if they factor in lake Ontario suppression in March through July, that's a cold body of water in the spring.

Anyone who downloaded the Toronto file should go in and correct one cell, I was working on the area of rankings for UHI corrections just before posting and updated them for Jan 2021, which involved adding one to ranks higher than its rank. In that process, a number slipped (probably one of those instances where the computer freezes up and you think you just corrected cell A but the result ends up in cell B). So what we have to correct is cell HE1969 (Jan 2015), which shows as 3 (at least on my home version it did, maybe the change happened after I uploaded). If you see a 3 there, it needs to be changed to 136 and that will trigger one other cell change for an average of 2015 rankings.

I went back through the chain and all other entries for Jan 2015 are unaffected. The rank in the raw data table is okay and the actual temps listed in all tables are correct. So that was the only place this error impacted and as there had been no graphics done for that table, nothing to note there.

What I'm doing recently is to construct the same colour-coded display for the UHI corrected ranks as already exists for the raw data ranks. Then since some of those change (early ones show some increases, late ones some decreases) I outline the changed colour code cells to give an overview of how the urban heat island correction affects the look of the overall data. It certainly flattens out the initial "sharp increase" look without totally removing the overall increase.

Anyone who has the file but finds the change more work than necessary can wait for the next seasonal update where this error will be corrected and the new graphic as mentioned will be available.

I have a few quality control panels to try to avoid data contamination problems, as you'll know if you work on excel files, the odd time you can type some intended change in and not see it where you thought it should be, which means perhaps it fell into a cell you had been looking at earlier. That would contaminate your data. But with the quality control panels, I can check at regular intervals that nothing like that has happened. The ranking tables are going to be placed in a newly designed q.c. check system where out of place ranks will show up on a graphic display. Given that there's probably over a million data points in the file, the odd error is probably difficult to avoid, even the pros have errors in their files as we found out with the NYC data.

That "3" cell must have shown up in the edit because the original work was done from tables with accurate ranking for Jan 2015 and I would have been typing numbers like 136 at that point, suspect that it was during a computer slowdown during the 2021 rank insertion updates. Have checked to see if any other obvious errors in the table, the new q.c. system should ensure that there are no significant errors (a lot of ties using the C data, and some random scatter of different ranks from the older F numbers although the higher and lower ranks tend to be immune to scrambling as they are separated out more).

Speaking of NYC, if you downloaded that file, a very minor change can be made, the record high on 25 Jan (60, 1967) was not colour coded (red). The day before was also a record, so if you can't see how to achieve the colour code but want to make the change, just copy the cell for the 24th (68 with red highlight) down one cell, then change the number you see to 60. That minor error is located in cell DA25.

There are some interesting month to month or season to season variations in the overall trends. One was that in the "climate warming" episode around the 1890s, June was off to a faster start than July or August in that warming. Another was that the more recent warming shows the opposite trend, July remaining closer to average than faster-warming June and August. The winters in the 1840s tended to be considerably milder in terms of rankings than the other three seasons, in fact after the UHI corrections the winters ended up looking fairly similar to the mid-20th century before the "modern warming" spike. Some of those observations are illustrated by graphs in the edited version that will be available after May ends.

[M.T. Cranium]

Originally Posted by Oneiric 3

MT, Attached (I hope) are the ECMWF monthly (from 1900 to 2010) reanalysis data from the Toronto region (not sure if the coordinates are exactly right for the actual station. Will take a look at this data myself later on to compare to the actual recorded data in your sheets, but for now, thought you might like to take a look:

Data is generated from this site and edit: anomalies in the 'anom' sheet are based around the rather curious 79-10 averages that Climate Reanalyzer uses.

https://climatereanalyzer.org/reanal...nthly_tseries/

I had a look at the raw data, turns out to be somewhat colder in general for January, their mean for the 111 years sampled was -7.6, the data average -4.7 and adjust to -5.5 after urban heat island removed.

The warmest January (1932) was 1.9 in raw data, 1.3 adjusted and -0.6 in their data set. A recent very cold January (1994) was -10.0 (-11.1 adjusted) and the data set gives -12.2.

So they seem just about perfect for a non-urban point in the general area somewhat further inland from Lake Ontario, I would imagine all of those data points would fit actual readings made in a farmer's field 10 miles outside the suburban limits. I haven't looked at other months or their anomalies.

I would say they are safe to use for any purpose, not sure what to say about a 1979-2010 base for anomalies, it can't be all that different from 1981-2010. Why do the data sets end in 2010?

Meanwhile if anyone is correcting their downloaded data sheets, relatively good news on the ranking quality control, I did that task and found only one other suspect rank, cell EG1941 (which refers to raw data for Sept 1939) got a bad edit somewhere and went from t-86 to t-7 (to be clear it should be t-86 or 86 inside a grey highlight box). Probably the same rigamarole as with the other egregious error. So that one can be changed manually and it will result in a couple of other cells changing. Once again, all data in tables and other places in the excel file remain valid for that month. I suspect it's an edit error because the colour code was right in the template of temperature anomaly groups by thirds.

No other monthly ranks are mismatched with either the temperature series or the other ranking table now.

If this does not concern you enough to bother changing it, the next edition will of course have that error corrected, and I now have a q.c. table installed to monitor for any unseen changes in the future.

In recent weather news, March 11 set a record at Toronto (66 F) and March 26 set a record at NYC (82 F) and there are several other new records at both locations for tied or new high mins. The month will end up around 20th warmest at both locations, raw data and 30th adjusted. There will be only traces of snow which is below the March average for both locations but not all that unusual for mild cases.

[Oneiric 3]

Originally Posted by M.T. Cranium

I would say they are safe to use for any purpose, not sure what to say about a 1979-2010 base for anomalies, it can't be all that different from 1981-2010. Why do the data sets end in 2010?

.

This was a typo on my part M.T, the climate based used by C.R is the 1979-00 and not the 1979-10 that I said. I have very awkward hands when typing (and when not).

I will get that ECMWF database for the co-ordinates you gave for downtown Toronto to see if there is any difference between those of the raw data and the data I posted from the greater Toronto region. Not sure why this data set ends in 2010, but later editions of this reanalysis data go further out, which should be accessible in that CR link I posted.

[M.T. Cranium]

Those co-ordinates are not carved in stone, what's the grid tolerance, nearest degree or 0.1 degree? The exact lat long as given in the EC source is 43.67N and 79.4W.

What exactly is re-analysis, estimates from maps? The maps from before 1945 would have no upper air measurements and any recreated maps showing that are based partly on actual temperatures; my data base has probably the only actual temperatures near the grid point before the airport opened in 1938, so really this is just a sort of estimate of what my data base applied to estimated maps comes out as after the fact. But I do think it correlates at a very high level from what I checked so far, mainly the January month by month and the more extreme values of other months, don't see any outliers. It's all a bit on the cold side of the actual or even urban-adjusted numbers.