Advertisement
If you have a new account but are having problems posting or verifying your account, please email us on [email protected] for help. Thanks :)
Hello All, This is just a friendly reminder to read the Forum Charter where you wish to post before posting in it. :)

Coding Horror

1272830323336

Comments

  • Technology & Internet Moderators Posts: 28,687 Mod ✭✭✭✭ oscarBravo


    typesafe wrote: »
    11. People’s names are all mapped in Unicode code points.

    This one intrigues me. What names can't be mapped to Unicode?


  • Registered Users Posts: 6 typesafe


    oscarBravo wrote: »
    This one intrigues me. What names can't be mapped to Unicode?

    Unicode Consortium has Urgently Needed Character proposal procedure for such cases. For example:
    IRG N2068 proposal documents a Hanja character approved for use in Korean names.
    RG N2078 proposal documents five Kanji characters to be used on Japanese residence cards.


  • Registered Users Posts: 768 ✭✭✭ 14ned


    typesafe wrote: »
    Unicode Consortium has Urgently Needed Character proposal procedure for such cases. For example:
    IRG N2068 proposal documents a Hanja character approved for use in Korean names.
    RG N2078 proposal documents five Kanji characters to be used on Japanese residence cards.

    Those issues will be fixed eventually, by adding yet more codepoints.

    There are far more serious problems with Unicode, in terms of that it can't work correctly no matter what you do for some text representations. Ultimately it was designed by a committee which was starting from a Western viewpoint of 1 codepoint = 1 character, so we got a Western-centric design based around alphabets well known to Western Europeans only. That leads to many possible byte sequences mapping to one text for some scripts, which introduces all sorts of impossible to avoid text corruption for even simple operations like copy and paste.

    This problem is worst for Asian and Indian text, see https://en.wikipedia.org/wiki/Unicode#Issues. It is suspected much worse is to come from African text soon.

    Niall


  • Registered Users Posts: 1,275 ✭✭✭ bpmurray


    14ned wrote: »
    Those issues will be fixed eventually, by adding yet more codepoints.

    There are far more serious problems with Unicode, in terms of that it can't work correctly no matter what you do for some text representations. Ultimately it was designed by a committee which was starting from a Western viewpoint of 1 codepoint = 1 character, so we got a Western-centric design based around alphabets well known to Western Europeans only. That leads to many possible byte sequences mapping to one text for some scripts, which introduces all sorts of impossible to avoid text corruption for even simple operations like copy and paste.

    This problem is worst for Asian and Indian text, see https://en.wikipedia.org/wiki/Unicode#Issues. It is suspected much worse is to come from African text soon.

    Niall

    Actually, it wasn't designed by committee. Mark Davis, Joe Becker and Lee Collins produced the first version. Since then, additional characters have been added, usually at the behest of the individual countries (who presumably are the experts). One such that I think was wrong was the including of some 14000 precomposed Korean characters - encoding the few jamo instead would have provided a lot of flexibility, including the addition of other Korean characters, but the Koreans wanted it that way. Or I should say, the South Koreans - the North Koreans have claimed that it's all wrong! Even minority languages have been proposed by experts, including Mike Eversen who lived in Ireland for some time.

    Probably the single biggest misunderstanding is with the Han Unification: Unicode encodes characters not glyphs, i.e. it encodes the semantics of a character and not how it displays nor its meaning.

    I see Ken Lunde is now on the committee. He wrote the CJKV book so is possibly the world expert in this area. As for Indian scripts, many Indians have been so used to ISCII that they don't get Unicode. The major scripts are encoded individually, and there are transcription algorithms that can be used to convert between them and even roman script. Ligatures are not encoded in any other script, since that is a function of rendering, not one of encoding, although precomposed accented characters are included as are a few old Arabic ligatures. The red herring of round-trip is something that Unicode never promised so don't expect it.

    That Wikipedia article is stretching, creating issues where none exist.

    And, for the record, I represented my corporate masters on the Unicode committee for a number of years.


  • Registered Users Posts: 768 ✭✭✭ 14ned


    Firstly, that's great extra detail. I knew that Ireland punched above its weight in Unicode standards (for some reason), but I'm very much a few links removed from the Unicode coal face. On WG21, any papers involving text must go before the Unicode study group for review and approval, and from there you get second hand information. But almost never do you hear anything from the horse's mouth, as it were.
    bpmurray wrote: »
    Actually, it wasn't designed by committee. Mark Davis, Joe Becker and Lee Collins produced the first version. Since then, additional characters have been added, usually at the behest of the individual countries (who presumably are the experts).

    "Design by committee" very often involves a small group of people who come up with something behind closed doors, and then steer it into a standard. Myself and about five others did the modern i/o proposal for C++, for example, even though 300 or so folk sit on the committee, and will have a hand in its design.

    Much of the "design by committee" problem stems from having to deliberately design a technical thing to be able to pass through a body of experts. So, you often don't choose a technically ideal way, because it's unexplainable to others in a short enough period of time to be realistically achievable.

    I don't know enough about computer encoding of human script to say what would be better than Unicode, but I have talked to those who do, and they say that Unicode, for better or worse, is at least a large improvement on what came before, even if it has serious problems in certain parts. I think on that we can all agree.

    Niall


  • Advertisement
  • Moderators, Recreation & Hobbies Moderators, Science, Health & Environment Moderators, Technology & Internet Moderators Posts: 84,861 Mod ✭✭✭✭ Capt'n Midnight


    https://www.theregister.co.uk/2020/04/02/boeing_787_power_cycle_51_days_stale_data/
    The US Federal Aviation Administration has ordered Boeing 787 operators to switch their aircraft off and on every 51 days to prevent what it called "several potentially catastrophic failure scenarios" – including the crashing of onboard network switches.
    Another symptom was for the flight management system to simply go blank and freeze, triggered by selection of a standard arrival path (STAR) with exactly 14 waypoints – such as the BIMPA 4U approach to Poland's rather busy Warsaw Airport.


  • Technology & Internet Moderators Posts: 28,687 Mod ✭✭✭✭ oscarBravo


    I used to be fluent in RPG, but I managed to miss out on the COBOL fad.


  • Registered Users Posts: 9,534 ✭✭✭ DublinWriter


    Wombatman wrote: »
    Anybody speak COBOL?
    Did it in my first year in TCD back in 89 and even then it seemed antiquated.

    PeopleSoft (now Oracle HR) still uses it for batch report processing AFAIK.


  • Registered Users Posts: 2,019 ✭✭✭ antimatterx


    Did it in my first year in TCD back in 89 and even then it seemed antiquated.

    PeopleSoft (now Oracle HR) still uses it for batch report processing AFAIK.

    Why can't these companies upgrade to modern tech? I always worry what the hindrance is


  • Advertisement
  • Technology & Internet Moderators Posts: 28,687 Mod ✭✭✭✭ oscarBravo


    Why can't these companies upgrade to modern tech? I always worry what the hindrance is

    When you have mission-critical software that Just Works, replacing it with software that might not work is a difficult risk to justify.


  • Registered Users Posts: 3,478 ✭✭✭ swampgas


    Why can't these companies upgrade to modern tech? I always worry what the hindrance is

    It's a good question.

    Massive numbers of man-hours have gone into legacy systems, and quite often the original developers are long, long departed. If those systems are mission-critical then anything to replace them has to "just work" - quite a tall order.

    I worked for years on a telco platform that had it's own proprietary programming language that ran on custom hardware. It turned out to be easier and more reliable to build a fast emulator for the old hardware using modern CPUs, and keep the old software, than it was to convert the existing code to a new language. (They tried ... and ended up with a different product. But that's another story.)


  • Registered Users Posts: 130 ✭✭ rdhma


    A long time ago, I took over maintenance of a certain program.
    At one point in the program, an array had been filled with values, which could determined by a simple formula.
    some hundreds of elements were assigned in the code like this:
    element[16]= 128
    element[17]= 255
    ...
    ...

    Yes, every one was typed in by hand like that. If the numbers had needed to be changed, too bad.
    I wrote a six line loop to do the job instead.
    "Who wrote this? It's unbelievable", I would have said to the boss.
    But I didn't say that, as he was the one who wrote it.


  • Registered Users Posts: 5,419 ✭✭✭ Slutmonkey57b


    Why can't these companies upgrade to modern tech? I always worry what the hindrance is

    In my experience the answer is always "incompetent management". When you think about it, these systems are generally so old that the companies who put them in at the time were probably cutting edge and management were confident and understood the need for infrastructure. Modern managers are generally accountants and outsourcers at heart and they don't care about infrastructure until it breaks, at which point they'd rather spend a few million paying for a consultants report that whitewashes then from blame than spend the same amount to actually fix the root cause.


  • Registered Users Posts: 14,706 ✭✭✭✭ Earthhorse


    At the risk of ringing in a cliché a big part of the problem is inadequate specifications. The old system does what it's meant to and everybody "knows" what that is but no one can articulate it in clear, understandable English. So you end up trying to port code back to intent and back to code again. Incredibly difficult.

    Another issue is resources. State bodies always seem overfunded but a lot of the time that's because they don't get big capital investment in order to improve systems so they never streamline and huge funding is required just to keep them running. It's sort of a Catch-22.

    Either way, I think it's always easy from the outside to think you have all the answers. It's always more difficult once you're the one involved.


  • Registered Users Posts: 25,943 ✭✭✭✭ GreeBo


    Earthhorse wrote: »
    At the risk of ringing in a cliché a big part of the problem is inadequate specifications. The old system does what it's meant to and everybody "knows" what that is but no one can articulate it in clear, understandable English. So you end up trying to port code back to intent and back to code again. Incredibly difficult.
    Its always this in my experience.
    When someone starts to notice the maintenance costs or outages from the old software the idea to rewrite springs up.
    Problem is that the requirements are often literally "The new system should do the same as the old system with transparent migrations"
    No one has a clue what the old system does from a business requirements standpoint.

    It often makes far more sense to start from scratch and figure out what you want the new system to do (avoids rewriting features that no one uses) but that involves difficult conversations with customers so you end up in emulator hell.
    I've seen emulators on top of emulators on top of emulators, its not pretty and it doesnt reduce the TCO at all, makes things more brittle (oh emulator 4 has a 2K limit on message sizes and emulator 2 cant handle double byte characters)

    If you are used to startups or small business then these rewrites seem so simple and logical, if you are used to billion dollar companies that have 50 years of legacy your opinion changes.


  • Registered Users Posts: 5,419 ✭✭✭ Slutmonkey57b


    A lot of that can be avoided if you're prepared to create a greenfield infrastructure and run it in a shadow mode beside the legacy for a reasonable period. Discover gaps as you go and fix. However the assumption seems to be that the only way to do things is to try and scrape everyone's brain, and have a grand switchover day on something you designed 5 years ago in a meeting room.


  • Registered Users Posts: 14,706 ✭✭✭✭ Earthhorse


    You can certainly do that. Though I'm not 100% sure how a shadow mode would work for, say, communicating with 3rd party systems. A lot of buy in is required from the client though, especially where there may be a wide variety of configurations (so each one will need to run in shadow mode). Perhaps you've seen this work in the wild and I'd certainly be interested in hearing how it did but it doesn't sound much better than picking people's brains. "Discover gaps as you go and fix" is exactly how most projects I've been on operate; can't say I'm a fan.


  • Technology & Internet Moderators Posts: 28,687 Mod ✭✭✭✭ oscarBravo


    GreeBo wrote: »
    It often makes far more sense to start from scratch and figure out what you want the new system to do...

    Pretty sure I've told the story before, but I was asked to modify a giant report shortly after starting a new job. I gave up on trying to understand the code, and instead reverse-engineered the end-user's requirements, and rewrote it from scratch. When the old and new versions of the report didn't match I was told my report was wrong - I had to painstakingly prove, line by line, that the old report was riddled with rounding errors and that in fact mine was correct.


  • Registered Users Posts: 25,943 ✭✭✭✭ GreeBo


    Earthhorse wrote: »
    You can certainly do that. Though I'm not 100% sure how a shadow mode would work for, say, communicating with 3rd party systems. A lot of buy in is required from the client though, especially where there may be a wide variety of configurations (so each one will need to run in shadow mode). Perhaps you've seen this work in the wild and I'd certainly be interested in hearing how it did but it doesn't sound much better than picking people's brains. "Discover gaps as you go and fix" is exactly how most projects I've been on operate; can't say I'm a fan.

    Yeah, if the system is a pipeline or workflow that calls out to multiple other backends including 3rd party or maybe even billing you cant just shadow it.

    You end up with yet more emulators and even then, unless you know every possible input and ouput you are just guessing.

    The bottom line is that in a lot of cases its not worth the business risk to try to change this stuff as the ultimate client can often have very penal contract clauses.

    Its only when the fragility of the system outweighs this risk that something starts to happen and by then its often a multi year, multi million dollar effort.


  • Advertisement
  • Registered Users Posts: 25,943 ✭✭✭✭ GreeBo


    oscarBravo wrote: »
    Pretty sure I've told the story before, but I was asked to modify a giant report shortly after starting a new job. I gave up on trying to understand the code, and instead reverse-engineered the end-user's requirements, and rewrote it from scratch. When the old and new versions of the report didn't match I was told my report was wrong - I had to painstakingly prove, line by line, that the old report was riddled with rounding errors and that in fact mine was correct.
    But even this is a great example of the difficulties of "upgrading" things.

    You could have had hundreds of clients who's own software *depended* on those errors. Fixing them on your side could cost them millions on their side.

    (I often deal with clients who's own software is ancient and in fact they dont even have developers anymore, its all outsourced on demand)
    Changing your code now involves significant business risk across two businesses.


  • Technology & Internet Moderators Posts: 28,687 Mod ✭✭✭✭ oscarBravo


    GreeBo wrote: »
    You could have had hundreds of clients who's own software *depended* on those errors. Fixing them on your side could cost them millions on their side.

    Case in point: Excel still thinks 1900 was a leap year, because fixing that bug would break spreadsheets.


  • Registered Users Posts: 14,706 ✭✭✭✭ Earthhorse


    GreeBo wrote: »
    You could have had hundreds of clients who's own software *depended* on those errors. Fixing them on your side could cost them millions on their side.

    100%

    I have found things that are clearly wrong (not major things in fairness) in the software I'm porting. But as other reports rely on this wrong information I've been told to leave it as is.


  • Registered Users Posts: 3,738 ✭✭✭ johnmcdnl


    oscarBravo wrote: »
    Case in point: Excel still thinks 1900 was a leap year, because fixing that bug would break spreadsheets.

    Better yet -- Excel actually deliberately implemented this as IBM Lotus which was the major spreadsheet software at the time had this bug. MS needed their software to be compatible with Lotus to help attract customers and the 'bug' itself is now part of the official specification http://www.ecma-international.org/publications/standards/Ecma-376.htm

    Users of Excel now have hacks or whatever in place to fix this issue if it's important to them, so if you actually 'fixed' the bug, you'd actually potentially cause a massive headache for every one of your customers who have already implemented their systems knowing about this issue.


  • Closed Accounts Posts: 22,674 ✭✭✭✭ beauf


    Wasnt this the problem that XML and data transformations were meant to solve. Only the solutions are more overhead than the problem it was trying to solve.

    I feel very jaded. Got every t shirt for every problem mentioned in this thread. I don't feel people want solutions and especially not innovation. They want to you to tick all the boxes on their change requests. They do not want you to point out any problems that you might have seen a million times before.

    I was trying to think of one thats a common denominator for me these days. I think it's setting deadlines and release dates without any analysis or concept of how long something is going to take. Also
    information silo's. All projects seem to run using a secret form of Chinese whispers.


  • Registered Users Posts: 9,534 ✭✭✭ DublinWriter


    T'is all about politics.

    Middle management and non-technical PMs/BAs want to tick the big project closure box at the end of the day and move on with their brilliant careers to the next glittering project. They will chose the path of least resistance in terms of change and usually be over cautious and not take on board wider holistic issues.


  • Closed Accounts Posts: 22,674 ✭✭✭✭ beauf


    ...Middle management and non-technical PMs/BAs ...

    I think the issue is we have middle management masquerading as PMs/BAs but have no idea how to do either.


  • Registered Users Posts: 25,943 ✭✭✭✭ GreeBo


    beauf wrote: »
    Wasnt this the problem that XML and data transformations were meant to solve. Only the solutions are more overhead than the problem it was trying to solve.

    I feel very jaded. Got every t shirt for every problem mentioned in this thread. I don't feel people want solutions and especially not innovation. They want to you to tick all the boxes on their change requests. They do not want you to point out any problems that you might have seen a million times before.

    I was trying to think of one thats a common denominator for me these days. I think it's setting deadlines and release dates without any analysis or concept of how long something is going to take. Also
    information silo's. All projects seem to run using a secret form of Chinese whispers.

    People dont want innovation if innovation isnt required.
    What percent of innovations actually result in something that makes money for the company, 1% maybe? Theres a reason the IDA gives support for R&D, even to multinationals, its because mostly it goes nowhere.

    If your company isnt in the business of R&D then you trying to solve some business problem via a GraphDB for 12 months and failing miserably isnt going to do much for the future of your company.


  • Closed Accounts Posts: 22,674 ✭✭✭✭ beauf


    GreeBo wrote: »
    People dont want innovation if innovation isnt required.
    What percent of innovations actually result in something that makes money for the company, 1% ...

    If you have metrics on the before and after that's easy calculated. These days I've stopped trying to sell what should be obvious to the clueless. They won't look at the metrics or understand them even if created them. Metrics are handy for promotions and job interviews. That's mainly why I still do them.

    Recently I had to cover for someone else, and got fed to with some insane support they were doing, that now I was getting tasked with. I rewrote a bit of code here on a couple of systems, changed a couple database jobs, changed the workflow slightly. End result its no longer generating work items for 4 or more people and generating any of the associate support emails and calls back and forth.

    I don't really care what the value of this is for organision. I only care that I'm no longer interrupted by this dumb task.

    Most of the projects that innovate that I've been involved in have mostly been successful. It's usually the ones that stifle innovation and lateral thinking, that fail. Oh and also those run by business heads out of their technical depth. Often because they simply don't know how to run a project or take advice.


  • Advertisement
  • Closed Accounts Posts: 22,674 ✭✭✭✭ beauf


    GreeBo wrote: »
    People dont want innovation if innovation isnt required.
    What percent of innovations actually result in something that makes money for the company, 1% maybe? Theres a reason the IDA gives support for R&D, even to multinationals, its because mostly it goes nowhere.

    If your company isnt in the business of R&D then you trying to solve some business problem via a GraphDB for 12 months and failing miserably isnt going to do much for the future of your company.

    One company I was in the bean counters got control and shut down the RD unit which was probably about 100 people in it. Strong mix of software engineering and creatives.

    Within a year or two that company was in trouble and within two or three years asset stripped and consumed by a competitor. Without RD they didn't innovate didn't keep their product fresh and lost their competitive edge. They'd also lost all the problem solvers once RD was gone.

    If your R&D is not being productive it's not being managed properly.


Advertisement