Eircode - its implemetation (merged)

PDVerse wrote: »

All opinions are not equal. The opinions of experts outweigh the opinions of non-experts if all we have are competing opinions.

Appeal to authority. Otherwise, an expert on clocks could tell us today is Monday, and we'd have to believe them.

Let's take an example of using Small Areas in the Eircode design. Experts explained that there would be considerable changes to Eircodes required after Small Areas were changed for each Census. Non-experts made pronouncements like the following

https://www.boards.ie/vbulletin/showpost.php?p=96345955&postcount=7385
https://www.boards.ie/vbulletin/showpost.php?p=96350729&postcount=7404
https://www.boards.ie/vbulletin/showpost.php?p=96356110&postcount=7416

Less than a year after Eircode launch Census 2016 was undertaken which required changes to Small Area boundaries to reflect the change in household formation and to protect privacy. This results in 43,362 Eircodes that have a different Small Area designation for the 2016 Census than they had for the 2011 Census. Either a large scale project would be required to re-allocate new Eircodes for existing addresses and ensure they were updated in everyone's databases, or the accuracy and usability of this feature would have eroded each Census.

Now that we have data we no longer need to rely on opinion.

Your point is not proven. You will have to explain why the ideas in the third link could not be used. The fact that there is no direct link between small areas and Eircodes means there was no requirement to use the technique described.

In fact, I can imagine that the CSO might logically have gone in the opposite direction. If they found that an SA needed to be split, it seems logical to me that both the original SA and the newly created one, would be given new identifiers, precisely to avoid confusion in statistics. Granted, if a formal link between SAs and Eircodes existed, then they would not have that freedom. That's a different issue. But, if that is what they did, then it makes the number of changes look much larger than might have been. I don't have access to the data. So, I can't confirm this or any other question I might have.

In any case, pretty much every other postcode in the world is hierarchical and deals with this issue. UK postcodes are smaller than CSO small areas, so they have to deal with it specifically, and the sky doesn't seem to be falling over there.

By the way, I went looking for SA shape files for the 2016 census. They don't seem to be available. That's unfortunate if true, and a regressive step by the CSO, which closes off any possible non-commercial usage of Eircode with google maps, and SA's.

plodder wrote: »

Appeal to authority. Otherwise, an expert on clocks could tell us today is Monday, and we'd have to believe them.

No. Anyone who is an expert recognisies that that the argument from authority is invalid. It appears that you are not an expert in any area, thus you missed the "if all we have is..." part of the sentence.

plodder wrote: »

Your point is not proven. You will have to explain why the ideas in the third link could not be used. The fact that there is no direct link between small areas and Eircodes means there was no requirement to use the technique described.

In fact, I can imagine that the CSO might logically have gone in the opposite direction. If they found that an SA needed to be split, it seems logical to me that both the original SA and the newly created one, would be given new identifiers, precisely to avoid confusion in statistics. Granted, if a formal link between SAs and Eircodes existed, then they would not have that freedom. That's a different issue. But, if that is what they did, then it makes the number of changes look much larger than might have been. I don't have access to the data. So, I can't confirm this or any other question I might have.

In any case, pretty much every other postcode in the world is hierarchical and deals with this issue. UK postcodes are smaller than CSO small areas, so they have to deal with it specifically, and the sky doesn't seem to be falling over there.

By the way, I went looking for SA shape files for the 2016 census. They don't seem to be available. That's unfortunate if true, and a regressive step by the CSO, which closes off any possible non-commercial usage of Eircode with google maps, and SA's.

Experts are aware of the existence of Small Area 2016 boundaries for Census analysis. They use them on a daily basis. Not only were you unaware, but you lacked the basic competence to even google their existence. What you did next is to draw a conclusion from your own incompetence that these boundaries are not available.
No. I do not need to explain anything to you. You lack such basic knowledge, let alone expertise, in this area that it would be a complete waste of my time.

"It find it hard to understand/believe" isn't a valid argument, just an admission of ignorance.

PDVerse wrote:

Experts are aware of the existence of Small Area 2016 boundaries for Census analysis. They use them on a daily basis. Not only were you unaware, but you lacked the basic competence to even google their existence.

Cheap shot. I've been talking about Small Areas for years. So, I'm well aware of their existence.

echat wrote: »

Any statistics on what % of paper post has an Eircode on it or were they never intended to be used for post?

Would be interesting. Maybe people here can say what percentage of their Christmas post had an Eircode on it. That would be a good indicator of voluntary use by the public. In my own case, I got only one card with an Eircode, though it was misaddressed - sent to the wrong name at a non existent address that looked similar to mine. They went to the trouble of looking up the Eircode thinking that would make it more likely to get to whoever it was intended for.

HelgaWard wrote: »

You can get them here: https://data.gov.ie/dataset/small-areas-ungeneralised-osi-national-statistical-boundaries

Links to boundaries used in table on this link
http://www.cso.ie/en/census/census2016reports/census2016smallareapopulationstatistics/

That is really useful. JSON is so much easier to work with than shape files.

The ungeneralised data works really well for identifying SA's for points retrieved from google maps.

On the issue of the changes since 2011, I can see that 336 small areas out of 18,641 changed in 2016 (assuming sa_pub2011 shows the id from 2011). Do you know is there an explanation for all of the fields in that database, such as the "changecode"?

So, after looking at the data yesterday, the majority of the changes (310) are straightforward splits of an area into 2 or more new ones. Around 18 are the opposite, ie combining two or more existing areas into one new one. The combining case shouldn't have any effect on postcodes. But, the splits would obviously. However, not all areas would be affected. From each group of split areas, the postcodes in one area would not be affected. So, that reduces the number of affected areas to around 175 (excluding ten with a change code that I haven't figured out). Also, it would make sense to choose the largest of the split areas to remain the same, reducing the number of affected properties again. And lastly, for each changing postcode, only the one or two letters relating to the small area would change. The rest would be the same.

That is basically as far as you can get with the data in the public domain and assumes all the changes were made consistently throughout the entire dataset.

PDVerse wrote: »

Of course none of this needs to be done. Your analysis is absolutely pointless. Do you have access to the ungeneralised 2011 boundaries? No. Do you see all of the minor boundary changes between the 2011 and 2016 small areas? No. Do you understand that this would have meant Eircodes assigned to the wrong Small Areas, and only detected after the boundaries were updated? No. Do you know how many updates to the coordinates of an issued Eircode have moved them from one Small Area to another? No. Just admit you're wrong and move on, a hierarchical postcode design would have provided absolutely no benefit for statistical analysis, it would simply have added an administrative burden and cost to both Eircode and CSO and lead to a series of unnecessary problems. Stop flogging the poor horse.

Just because an eircode is assigned to a new small area code would not require a hierarchical postcode to be changed in all cases. That was my main point. All of the codes in the biggest new sub-area would stay the same. Codes in merged areas could also stay the same.

As to the extent of other minor boundary changes, I can only go on the data that is available to me. I would need the ECAD (or geojson files and a complete list of eircodes for both datasets) to analyse it definitively. But, you do have it and you came up with the figure of 43,362 which is the right order of magnitude for the number of properties in all 363 changed areas, which it appears to me, is an over estimate.

I'm not getting into the argument again about hierarchical versus opaque postcodes. It is pointless and been done to death already. But any renumbering effort would be a trivial task for someone with access to ECAD. Tables of changed eircodes could also be supplied to users who would add the data to their own databases.

I’m not desperate about anything. You quoted this figure of 43,362 and I’m questioning its significance.

PDVerse wrote:

This results in 43,362 Eircodes that have a different Small Area designation for the 2016 Census than they had for the 2011 Census.

Regardless of how that figure was arrived at, the real number of postcodes that would have to change, if the SA were built into the structure of the code, is far less, for the reasons I mentioned.

I think that was worth pointing out.

PDVerse wrote: »

You are questioning its significance because the data undermines your argument.

I’m questioning its signficance because the “data” is highly misleading. If the actual number of codes that would have to change is a lot smaller, then that undermines YOUR argument.

How is it worth knowing exactly how many thousands of people would be inconvenienced for the sake of a feature that isn't required for statistical analysis?

We both know the arguments. They’ve been gone over dozens of times here.

It’s not *required* for statistical analysis when you have the proprietary data. YOU have the data. So, you can do all the analysis you wish when it suits whatever argument you want to make. I don’t have it, so I can’t.

The company that analysed those debt judgments didn’t have it either. So, what they produced was described here (by someone who says they deal with spatial data) as “meaningless fluff”. If that’s what it is, then it’s because of our postcode design. The exact same technique used on UK postcodes would have produced data that was useful and consistent.

Statistical analysis is not the main benefit of a hierarchical postcode. It is the fact that the structure of the code being open, and not hidden, facilitates all kinds of uses without having to use computers or software at all. You can group locations just by looking at the postcode. Yes, the downside of that is the probability that some postcodes might have to change when the underlying areas change. And that's why it is interesting at this point to have an accurate assessment of what would have changed.

TheChizler wrote: »

Surely it's because of them using an inappropriate identifier to group data, not because of an identifier failing to do it wasn't required or designed to do?

What other identifier could they have used?

TheChizler wrote: »

Small area as you suggest, or a grouped small area. Or county. Or mobile phone prefix or surname. Some of those are more appropriate than others, it's up to the researcher compiling the data to decide which is the most appropriate way to group published statistics. A 3 character code used by An Post to sort mail is arguably inappropriate but you can't blame Eircode for that, same as if they used phone prefix, it would be a bit mad to blame Comreg.

I think it's a bit meaningless to report absolute debt figures without providing the context of how many people or properties it's distributed amongst, even if you did figure out an appropriate way to geographically group them. Lazy but flashy reporting if you ask me.

But they didn't have access to anything other than the eircodes and the judgement values. They would have put them all in a big file, sorted them and extracted the totals out per routing-key. Routing key was all they had.

plodder wrote: »

But they didn't have access to anything other than the eircodes and the judgement values. They would have put them all in a big file, sorted them and extracted the totals out per routing-key. Routing key was all they had.

Really? Did the multi-million euro turnover company not have the €180 for a single seat licence of the ECAD database to perform any analysis they wanted? Or maybe a big boy stopped them from doing their work properly?

Congratulations Plodder. That's quite an achievement, managing to jump the shark on a horse you've flogged to death.

plodder wrote: »

I suppose it is possible (likely even) that they used eircode data to match addresses to routing keys, if eircodes had not been provided in the data from the courts system. So, the question then is why did they publish it using RK rather than small area? Probably the same reason as to why the CSO publishes property prices by RK (post #1802)

Publishing a list of 9+ digit small area codes would mean nothing to the general public. And that point is central to the design of the postcode itself. If the routing keys were a consistent size, the data would have been more meaningful. If the postcode design were hierarchical then they could have published it at RK level or Small Area level, or both.

To quote Wolfgang Pauli, this is not even wrong. I'm busy now, I'll explain why later.

plodder wrote: »

Publishing a list of 9+ digit small area codes would mean nothing to the general public. And that point is central to the design of the postcode itself. If the routing keys were a consistent size, the data would have been more meaningful. If the postcode design were hierarchical then they could have published it at RK level or Small Area level, or both.

plodder wrote: »

I await with interest your explanation as to how numeric small area codes (like 057093001, or 057081001/05708100) would be meaningful to the public.

Whereas if they were built into the postcode itself (translated obviously into RK plus a one or two character code) people would be familiar with them because they would be part of everyone's postcode.

In any case, the fundamental problem is the fact that the data is monetised (unlike the very impressive CSO data), which obviously isn’t a problem for commercial users, but is a problem for small-scale use. If they even published a list of valid Eircodes for free with no other data, that would be of huge benefit, as the information available from google and the CSO would allow useful datasets to be created, without affecting the main commercial business case for ECAD/ECAF.

Postcode areas, if designed correctly, do not have a consistent size. You don't seem to understand this, and you are under the false impression that the UK Postcode areas are of a consistent size. The equivalent of a Routing Key is a Postcode District, of which there are approximately 3,000. They range in population size from 136 to over 150,000. You simply don't know what you are talking about.

Your claims above are false, the assumptions are incorrect and the conclusions are invalid.

If you explain the concept of North to someone, and then explain that the North pole is the most Northerly point on the planet, it is quite exasperating to listen to "no, that's not the most Northerly, I can just keep walking further North". The person may think they are making a valid point, but they are just demonstrating their lack of understanding.

Finding out that you are wrong is a good thing. It means that you have gained knowledge and have a better understanding of the world. Your ego should'nt be so fragile that you reject knowledge if it proves an idea you hold is wrong. But we all do this, to different degrees with different frequency; no one is immune. One thing you can do is become familiar with how our minds actually think, the tricks that conciousness plays on us that lead to cognitive biases. I thoroughly recommend "Thinking Fast and Slow" by Daniel Kahneman.

After you've read that book, take some time to consider how you interact with experts. You are currently experiencing the following thoughts:

"I've had an idea, after five seconds of thought, on a subject in which I have no knowledge or expertise, but I'm convinced it is a better idea than proposed by experts who have dedicated their lives to the study and understanding of the subject and have a wealth of experience to draw on. This is where experts go wrong, not asking ordinary people their opinions."

If you step back and analyse this, it's pretty arrogant isn't it? I highly recommend "The Death of Expertise" by Tom Nichols. Expertise is a valuable commodity in society, and should be appreciated rather than dismissed. If you've ever tried your hand at plastering, you'll appreciate the expertise of a skilled plasterer.

If you read these two books and take their lessons on board it will help you approach your interactions with experts with the required humility to engage in an intellectually honest manner that can be productive for all concerned.

plodder wrote: »

Even though routing keys are a terrible basis for statistics, as predicted, people will continue to use them, because they are still public identifiers which cover the whole country, and you don't have to pay any license fees to Eircode to use them. Here is the latest example:

https://www.independent.ie/irish-news/value-of-debt-judgments-down-by-a-third-but-recovery-remains-patchy-36466240.html



Routing key areas vary so much in size and population that the sums of money in the table can't be compared meaningfully.

The research here was done by a UK consultancy, probably using exactly the same technique that they would use in the UK - by matching debt judgments against postcodes, but because the only public element of Irish postcodes is the routing key, what they end up with is not all that useful. At least the data in this survey is public domain to begin with, and no direct harm can come from this.

One of the companies mentioned as doing that analysis, Sagacity Solutions, is an Eircode reseller so they must have used the routing key because it mirrored using UK postcodes.

Newbridge, Dublin 15 and 18 dominate so they must be property debts as they would be boom time building areas.


The Irish Independent article refers to 500,000 pieces of Court data, how could that be true for Ireland?

plodder wrote: »

.

Obviously, when the UK postcode was designed back in the 1950's it had to map to their delivery network and that could have varied quite a bit, but there is still a high level of consistency. The most important thing is the postcode unit itself having 12-15 addresses typically. Similarly, with Irish Small Areas, it really doesn't matter if SA's vary from 80 to 150 or whatever households. Routing keys varying from tens of thousands to only hundreds is a big problem for stats based on them.

Each time some new organisation comes along with a new set of data based on Eircode Routing Keys, this issue is going to arise again and again.

Don't base stats on them.

SAs were designed for statistics. And If you need to group bigger areas you have a hierarchical structure in place from SA to ED etc.

Your obsession with clarity for the lay person seems to ignore that SAs and EDs can be grouped into the counties and LA areas we all know and love and in some cases Townlands and towns. There is NO reason for eircodes to be set up to incorporate stats. That they can be is up to the end user. But why you would get yourself into such knots over this is beyond me.

Eircodes are working and working well. Simples.