Advertisement
If you have a new account but are having problems posting or verifying your account, please email us on hello@boards.ie for help. Thanks :)
Hello all! Please ensure that you are posting a new thread or question in the appropriate forum. The Feedback forum is overwhelmed with questions that are having to be moved elsewhere. If you need help to verify your account contact hello@boards.ie
Hi there,
There is an issue with role permissions that is being worked on at the moment.
If you are having trouble with access or permissions on regional forums please post here to get access: https://www.boards.ie/discussion/2058365403/you-do-not-have-permission-for-that#latest

Making the move from academia to the private sector

  • 04-05-2016 4:36pm
    #1
    Registered Users, Registered Users 2 Posts: 13,104 ✭✭✭✭


    Hello all,

    I am in need of some professional industry opinions.

    I am an engineer who has been working in academic research for about 10 years now (4 years for my PhD, about 6 years in a PostDoc). Most of that time has been spent designing image analysis algorithms and developing them, where possible, into Java-based applications for scientists to use, specifically, experimental biologists. I am now approaching the end of my current contract and, due to the lack of opportunities in academia, I have to consider moving into the private sector.

    Although I would not consider myself a software developer as such, the fact that the bulk of my work involves coding (mainly in Java) and it is therefore featured prominently on my CV, the first thing that will spring to most people’s minds when they see my CV is “Java developer”. But, I would mainly consider myself to be an algorithm designer/developer – is that pedantic? I would not be at all confident in referring to myself as a developer, mainly because I have essentially been a one-man show - I have no experience of working in a development "team". I do my best to implement best practices, but my code/software undoubtedly has numerous shortcomings relative to that produced by “professional” developers, mainly due to time constraints and the emphasis on getting something functional as quickly as possible. That said, I have produced a number of software tools (all open source), one of which has been particularly well received by scientists in various labs around the world.

    I realise it depends heavily on the position applied for, but I’m wondering, in general, at just how much of a disadvantage I am having worked in academia for so long, relative to someone with say, ten years industry experience? There are a lot of industry “best practices” that I am likely to be totally unfamiliar with. Does anyone else have any experience of working in academia and, if so, what was your experience of the transition?

    Thanks in advance for any feedback anyone can provide – much appreciated.


Comments

  • Registered Users, Registered Users 2 Posts: 2,655 ✭✭✭draiochtanois


    This post has been deleted.


  • Registered Users, Registered Users 2 Posts: 23,212 ✭✭✭✭Tom Dunne


    Look at emerging fields such as data analytics - from my admittedly limited knowledge, it involves a lot of algorithms, plus some coding (brush up on SQL).

    Aside from that, make sure you do a basic search on the various jobsites for 'PhD' - that should give you an idea of what is out there and what you could be aiming at.


  • Registered Users, Registered Users 2 Posts: 13,104 ✭✭✭✭djpbarry


    Yeah, I'm seeing the nebulous concept of "big data" being mentioned a lot so I'm trying to learn about the concepts involved - Hadoop seems to be getting more and more mentions in job specs.


  • Registered Users, Registered Users 2 Posts: 768 ✭✭✭14ned


    djpbarry wrote: »
    I realise it depends heavily on the position applied for, but I’m wondering, in general, at just how much of a disadvantage I am having worked in academia for so long, relative to someone with say, ten years industry experience? There are a lot of industry “best practices” that I am likely to be totally unfamiliar with. Does anyone else have any experience of working in academia and, if so, what was your experience of the transition?

    I'll be blunt: if I see a CV in front of me for an engineer with only academic experience, alarm bells ring very loudly. Most academic programmers write spectacularly bad code, worse than even a fresh graduate, because they have only ever written once-off solutions rather than easily maintainable, rigorously specified software with outstanding unit, functional, integration and acceptance testing. Far worse again, they won't accept how awful they are and are therefore unretrainable.

    That's the sweeping generalisation, but perhaps 10% of them are superb programmers, as good or better than anyone with twenty years at the top of the industry. You basically need to make employers very sure you're in that category.

    One path might be to follow Louis Dionne, author of the amazingly impressive http://www.boost.org/doc/libs/master/libs/hana/doc/html/index.html. He comes from a pure maths background, and only got into coding through needing to write proof programs. That eventually led into C++ metaprogramming (programming the C++ compiler to generate C++ programs), and he came to present at one of the three big global C++ conferences where he was well received. That turned into presenting at two global conferences per year for two years.

    Even just a year ago, his code was clearly written by someone gifted but lacking in professional software design experience. But he underwent peer review to enter the Boost C++ Libraries, and caught up with modern design and practice through facing the critique of world experts. Boost.Hana is now an exemplar of a new wave of exciting C++ 14 only libraries, and will no doubt be enormously influential to the next generation of C++ libraries.

    Multinationals had been chasing him for a while given such an entry into the international conference circuit and the fact he got a library into Boost at such a young age. He turned them down, but eventually Facebook made him an offer he couldn't refuse, so he joins them this summer at their HQ. They paid for his relocation, everything, and shortly he will be quite a wealthy young man.

    tldr; Transitioning successfully into the private sector isn't hard, but requires a bit of forward planning. Submit and present to the major global conferences for at least a year or two. Put yourself through a formal code peer review and come out successful. You should find yourself with the pick of roles because you've proven you are not a typical academic programmer.

    Niall


  • Registered Users, Registered Users 2 Posts: 2,655 ✭✭✭draiochtanois


    This post has been deleted.


  • Advertisement
  • Registered Users, Registered Users 2 Posts: 768 ✭✭✭14ned


    This post has been deleted.

    I'm not sure where this hate is coming from. What I said is my experience of the matter. Other people will have different experiences. I haven't personally transitioned from academic programming into private, but I can put the OP in touch with people who have if he PMs me, and I think their opinions will not be dissimilar to mine, after all they've worked with academic programmers for many years and the topic has come up in the conference bar repeatedly over many years. I'm echoing what I've heard there, and from code written by academic programmers I've studied either as part of replicating an algorithm or as part of interviewing a candidate.

    Niall


  • Registered Users, Registered Users 2 Posts: 2,655 ✭✭✭draiochtanois


    This post has been deleted.


  • Registered Users, Registered Users 2 Posts: 1,717 ✭✭✭Raging_Ninja


    Disagreement isn't hate.

    You make sweeping generalisations about programmers from an academic background without any data to back it up.

    It's nonsense plain and simple.

    Does past experience count as data?


  • Closed Accounts Posts: 22,648 ✭✭✭✭beauf


    14ned wrote: »
    ...Most academic programmers write spectacularly bad code, worse than even a fresh graduate, because they have only ever written once-off solutions rather than easily maintainable, rigorously specified software with outstanding unit, functional, integration and acceptance testing. Far worse again, they won't accept how awful they are and are therefore unretrainable....

    I think a lot of programmers are like that. A lot of the development world does not work to best practises and is hard to break bad habits.


  • Registered Users, Registered Users 2 Posts: 2,655 ✭✭✭draiochtanois


    This post has been deleted.


  • Advertisement
  • Registered Users, Registered Users 2 Posts: 768 ✭✭✭14ned


    This post has been deleted.

    You're totally missing the point.

    It doesn't matter what the truth is. What matters is what the people sitting on hiring committees think is true.

    And those people will probably hold the same prejudices as me given that most of the engineers I've ever met at conferences think the same.

    Let me give you a recent example. I recently presented at the ACCU conference in Bristol a few weeks ago. One of the keynote speakers there was an academic researcher called Marian Petre, she researches how software engineering culture affects engineering quality. In the lobby after her talk we got chatting over lunch, and after a lengthy discussion of how generally awful it is amongst a group of about eight people, half of whom were very senior long standing career engineers, she directly asked each of us the question: "what would you recommend to raise the quality of academic programming?"

    My personal reply was that until grant awarding bodies insist that all research programmers must regularly take skill inservice courses to teach them basic things like "source control", "unit testing" - and PASS those courses - you would see little movement because the incentives are not aligned for quality software engineering by the grant making process, the academic career promotion process, nor the publication count process.

    It may not be your experience that academic programmers couldn't be bothered with source control nor unit testing, but from what Marian told me - and this is her long term research field - it is the majority practice by far. No one is claiming this is universal, just there is an enormous gap between best practice and the median, and a far bigger gap than in any other software practice domain. That raises alarm bells in those who review CVs from academic programmers.

    So I'll return to my original advice to the OP - you may see biases against you due to your career history (same as we all do BTW, just we experience different biases) and my advice is to take steps to remediate any such biases as soon as possible lest it get in the way of future employment. I suggested one path I know to have worked, but I entirely accept that path takes many years of forward planning and I have no suggestion for anything others (including yourself) haven't already suggested.

    Finally, the OP may find it useful to know that in my current contract I assist a team who takes code written by the blue sky R&D team and rewrites it to test commercial viability. None of the blue sky R&D team are superb programmers, but they're great at maths and theory, and it's the next layer who need to figure out how to write a bit of software which can be sold implementing their ideas. Downstream from us is a ton of commercialisation teams who support products sold to customers, they take over once a piece of software starts earning significant cash flows.

    There is a rough 3:1 ratio between each level, so three of us for each blue sky R&D team member, and three of the main programmers for each of us writing the initial implementation. Getting a job in the blue sky R&D team is exceptionally hard, getting a job in our team is very hard, getting a job in the commercial support team is just hard. If you need a job in a finite time period, you need to aim your CV where the jobs are, and invest the outside of work effort to get yourself eventually back into blue sky R&D if that's where your passion is.

    I'll end with mentioning that everyone here in the blue sky R&D team is well over fifty years old, and I've seen the same pattern in BlackBerry. There would seem to be a big age gap between the twenty something academic programmer and getting hired to do the same in a multinational. I don't know if this is typical - lack of data points - but it might be worth bearing in mind.

    Niall


  • Closed Accounts Posts: 22,648 ✭✭✭✭beauf


    I think there are roles below the level 14ned is referring, which the OP might be happy in. I think they very much appreciate theres a gap to cross here.

    http://www.learnvisualstudio.net/2013/05/17/net-developer-career-advice-getting-started-later-in-life/
    http://sijinjoseph.com/programmer-competency-matrix/


  • Registered Users, Registered Users 2 Posts: 2,655 ✭✭✭draiochtanois


    This post has been deleted.


  • Registered Users, Registered Users 2 Posts: 1,717 ✭✭✭Raging_Ninja


    wow, just wow

    You don't have to like it. Tbh I've heard the same sort of thing myself, and I've seen it with my friends who decided to stay in academia.


  • Closed Accounts Posts: 22,648 ✭✭✭✭beauf


    The better the programmer the more elitist they become. In general. Surprised you haven't experienced that before.

    However there are other rungs on the ladder, even if those towards the top, don't remember ever being on it.


  • Registered Users, Registered Users 2 Posts: 768 ✭✭✭14ned


    beauf wrote: »
    The better the programmer the more elitist they become. In general. Surprised you haven't experienced that before.

    However there are other rungs on the ladder, even if those towards the top, don't remember ever being on it.

    Ehhh ... elitist is the wrong word. Most of the guys at the top are very approachable, if you approach them the right way which is a sort of elitism I guess. If you conform to what they collectively expect of good engineers, you get warmly received, else you get ignored.

    But better than the phrase "elitist" is "guided by rules of thumb". I don't know if it's normal outside programming, but the better the engineer the more mistakes they routinely make (they are the "right" kind of mistake though) and the more their code is guided by rules of thumb and instinct than planning. Marian Petre, that academic I mentioned, was speaking at the ACCU conference on exactly this topic - what culturally separates good software engineers from bad, and had tons of empirical evidence supporting her research conclusions most of which - but not all - matched the rules of thumb (prejudices) of the very senior engineers given the reaction of those sitting beside me during the talk.

    Unfortunately the video of that keynote was not made available online, else I'd highly recommend watching it. Those really interested can probably find plenty of reading material in her publications list at http://mcs.open.ac.uk/mp8/.

    Niall


  • Closed Accounts Posts: 22,648 ✭✭✭✭beauf


    Elitist. Is the right word. Its pervasive throughout the industry. Which is why people find it hard to get into, and recruiters, are only interested in the top 20% of candidates. Regardless if that's appropriate for the role.

    Its not unique to development though.


  • Registered Users, Registered Users 2 Posts: 13,104 ✭✭✭✭djpbarry


    14ned wrote: »
    Most academic programmers write spectacularly bad code, worse than even a fresh graduate, because they have only ever written once-off solutions rather than easily maintainable, rigorously specified software with outstanding unit, functional, integration and acceptance testing.
    Yep, that’s pretty much my experience.

    Although I think “spectacularly” bad is probably a bit harsh!
    14ned wrote: »
    Far worse again, they won't accept how awful they are and are therefore unretrainable.
    Well, I’m not sure I accept that. I mean, I’m well aware of the fact that my code could be vastly improved upon – I just struggle to find the time to do the necessary refactoring. Any of my colleagues who write code are also only too aware of the need for improved software engineering standards in academia, mainly because we all draw on (badly written, badly documented) code produced by other academics pretty regularly. But, finding the time to implement the necessary initiatives is challenging.

    Why can’t we find the time for these things? Mainly because our employers don’t understand the value of well-written, well-tested, easily extendible and maintainable code, because they don’t write it themselves. They want results for publication ASAP – they don’t really care what produces those results.


  • Registered Users, Registered Users 2 Posts: 13,104 ✭✭✭✭djpbarry


    14ned wrote: »
    Ehhh ... elitist is the wrong word. Most of the guys at the top are very approachable, if you approach them the right way which is a sort of elitism I guess. If you conform to what they collectively expect of good engineers, you get warmly received, else you get ignored.
    Well, to be fair, that’s not really approachable, is it?


  • Registered Users, Registered Users 2 Posts: 1,717 ✭✭✭Raging_Ninja


    djpbarry wrote: »
    Why can’t we find the time for these things? Mainly because our employers don’t understand the value of well-written, well-tested, easily extendible and maintainable code, because they don’t write it themselves. They want results for publication ASAP – they don’t really care what produces those results.

    Most bosses in the private sector are the same way.


  • Advertisement
  • Registered Users, Registered Users 2 Posts: 768 ✭✭✭14ned


    djpbarry wrote: »
    Well, to be fair, that’s not really approachable, is it?

    It's true it's all relative.

    I used the qualifier "most" with good reason. There are some outstanding engineers whom are extremely unapproachable. They contribute a lot of value to processes like setting international engineering standards, but interacting with them is never a pleasant experience. You endure it because of their stature in the industry and because involving them does lead to much better final standards. You also tend to learn much you wouldn't otherwise from working with them.

    I understand it's very similar in academic research. Most of the top research fliers are approachable in the same way as the top engineers, and are personally quite affable. But there is a small minority of brilliant academics who are quite difficult to work with, though they contribute a lot of value.

    Niall


  • Closed Accounts Posts: 22,648 ✭✭✭✭beauf


    I suspect this a bit like applying premier league standard to the pub team.
    Where 99.9% of people will never have these problems.


  • Registered Users, Registered Users 2 Posts: 7,521 ✭✭✭jmcc


    djpbarry wrote: »
    Yeah, I'm seeing the nebulous concept of "big data" being mentioned a lot so I'm trying to learn about the concepts involved - Hadoop seems to be getting more and more mentions in job specs.
    Hadoop is just one of a set of tools. Issues like Computability and algorithms are important when it comes to Big Data. In order to get any Big Data project to the stage where it needs software, it is sometimes necessary to apply a lot of theory.

    Also try Microsoft. The next step after its Cortana effort may be imaging algorithms for some types of avatar interactions.

    Regards...jmcc


  • Registered Users, Registered Users 2 Posts: 6,283 ✭✭✭Talisman


    djpbarry wrote: »
    I would not be at all confident in referring to myself as a developer, mainly because I have essentially been a one-man show - I have no experience of working in a development "team". I do my best to implement best practices, but my code/software undoubtedly has numerous shortcomings relative to that produced by “professional” developers, mainly due to time constraints and the emphasis on getting something functional as quickly as possible. That said, I have produced a number of software tools (all open source), one of which has been particularly well received by scientists in various labs around the world.
    Give yourself some credit for recognising this. You have described the work case scenario for possibly 90% of developers. The management/client generally do not care about the quality/design of the code so long as it works - but it inevitably leads to issues down the line.

    Obviously you can write code and have a portfolio of tools which you have released. Presumably you have used a version control system such as Git.

    Have you written unit tests? If you haven't then start writing unit tests for some of the code you have released. Write tests that simulate faults in the system not just trivial tests that simulate expected behaviour.

    Do you document your code? Writing documentation can help improve the design of your code because it makes you think about it in a more formalised way.

    If you test and document your code then you should comfortably be able to speak about it in an interview situation. After that it becomes a case of whether or not you are a good fit in terms of your personality in the environment of the team.

    Don't let the fact that you haven't worked in a team of developers undermine you. Behavioural questions about teamwork are used to get a sense of what you would be like to work with. There are plenty of resources online that will teach you how to prepare for such questions.

    The ability to help others to further develop their skills and experience is also a valuable attribute in a team environment. Have you ever had to teach a colleague how to use your software? What were the circumstances? Why did the individual need assistance? How did you go about the task? What was the outcome? If you can answer such questions in a positive manner then you'll do okay in an interview.


  • Registered Users, Registered Users 2 Posts: 1,275 ✭✭✭bpmurray


    Have you considered research in a private-sector company? IBM has a research place in Dublin, where post-docs and more do their thing.


  • Registered Users, Registered Users 2 Posts: 13,104 ✭✭✭✭djpbarry


    bpmurray wrote: »
    Have you considered research in a private-sector company?
    Yeah, that would be ideal, but such positions are hard to come by.


  • Registered Users, Registered Users 2 Posts: 1,922 ✭✭✭fergalr


    14ned wrote: »
    You're totally missing the point.

    It doesn't matter what the truth is. What matters is what the people sitting on hiring committees think is true.

    And those people will probably hold the same prejudices as me given that most of the engineers I've ever met at conferences think the same.

    Let me give you a recent example. I recently presented at the ACCU conference in Bristol a few weeks ago. One of the keynote speakers there was an academic researcher called Marian Petre, she researches how software engineering culture affects engineering quality. In the lobby after her talk we got chatting over lunch, and after a lengthy discussion of how generally awful it is amongst a group of about eight people, half of whom were very senior long standing career engineers, she directly asked each of us the question: "what would you recommend to raise the quality of academic programming?"

    My personal reply was that until grant awarding bodies insist that all research programmers must regularly take skill inservice courses to teach them basic things like "source control", "unit testing" - and PASS those courses - you would see little movement because the incentives are not aligned for quality software engineering by the grant making process, the academic career promotion process, nor the publication count process.

    It may not be your experience that academic programmers couldn't be bothered with source control nor unit testing, but from what Marian told me - and this is her long term research field - it is the majority practice by far. No one is claiming this is universal, just there is an enormous gap between best practice and the median, and a far bigger gap than in any other software practice domain. That raises alarm bells in those who review CVs from academic programmers.

    So I'll return to my original advice to the OP - you may see biases against you due to your career history (same as we all do BTW, just we experience different biases) and my advice is to take steps to remediate any such biases as soon as possible lest it get in the way of future employment. I suggested one path I know to have worked, but I entirely accept that path takes many years of forward planning and I have no suggestion for anything others (including yourself) haven't already suggested.

    Finally, the OP may find it useful to know that in my current contract I assist a team who takes code written by the blue sky R&D team and rewrites it to test commercial viability. None of the blue sky R&D team are superb programmers, but they're great at maths and theory, and it's the next layer who need to figure out how to write a bit of software which can be sold implementing their ideas. Downstream from us is a ton of commercialisation teams who support products sold to customers, they take over once a piece of software starts earning significant cash flows.

    There is a rough 3:1 ratio between each level, so three of us for each blue sky R&D team member, and three of the main programmers for each of us writing the initial implementation. Getting a job in the blue sky R&D team is exceptionally hard, getting a job in our team is very hard, getting a job in the commercial support team is just hard. If you need a job in a finite time period, you need to aim your CV where the jobs are, and invest the outside of work effort to get yourself eventually back into blue sky R&D if that's where your passion is.

    I'll end with mentioning that everyone here in the blue sky R&D team is well over fifty years old, and I've seen the same pattern in BlackBerry. There would seem to be a big age gap between the twenty something academic programmer and getting hired to do the same in a multinational. I don't know if this is typical - lack of data points - but it might be worth bearing in mind.

    Niall

    My experience is you generally write pretty good stuff on here!
    But I think you're a little bit wide of the mark here.

    14ned wrote: »
    It doesn't matter what the truth is. What matters is what the people sitting on hiring committees think is true.
    First off, that is kind of an admission of failure. The better they are at their job, the closer what they think is true will be to the truth. But anyway...
    14ned wrote: »
    Facebook made him an offer he couldn't refuse
    [...]
    tldr; Transitioning successfully into the private sector isn't hard, but requires a bit of forward planning. Submit and present to the major global conferences for at least a year or two.
    You are talking about things at a very top level there.

    Maybe hanging around so you can submit to global conferences for a couple of years, is the right path if all you would be happy with is to be headhunted by Facebook.
    But:
    1) that's overkill for most people more modest career aspirations
    2) is a strategy with a pretty high failure rate even then.

    "Just develop a the next widely used replacement for Hadoop before you leave, so people will headhunt you."
    ...ok.

    14ned wrote: »
    It may not be your experience that academic programmers couldn't be bothered with source control nor unit testing, but from what Marian told me - and this is her long term research field - it is the majority practice by far.
    I'm not sure what you're trying to say here; but, source control and unit testing take no time at all to pick up, if you have been writing algorithms for 5 years. An 'academic programmer' could learn that stuff in no time at all on the job.


    FWIW I wrote pretty much no *unit testing* on my academic work.
    Even though I had previously worked as a professional software engineer at a high level.
    I certainly did not use test driven development.
    It just wasn't the right tool for what I was trying to do.
    (I did have end-to-end functional tests that I could use to validate algorithmic performance, and catch errors.)

    The systems I was building tended to be small in terms of lines of code, but very dense algorithmicly.

    I used source control, but pretty much never referred to history, and there was no code review, as I was working on single person projects. I would not have lost much by just storing my files on dropbox.


    So, honestly, it is a little 'cargo-cult' to think somebody doesn't know what they're doing because they make certain trade-offs that best suit the very prototype style of coding that might be optimal for their academic work.


    That said:
    There are absolutely some people who work on academic code for 5 years, who would not be suitable for a senior software engineering position. I'm thinking of, for example, physicists, some of whose code I have used, who write very 'Fortran style' code, and, never learned basic software engineering, never learn about encapsulation, modularity, coupling, architecture of any sort etc.


    But, it's not that these people learned bad practices in academia, or that their training was deficient.
    It's just that they never learned any software engineering, because that's not what they were doing.

    It is a problem if somebody like this is hired directly into a software engineering role.

    But this is not a property of academia, any more than if you hire in a wet lab research biologist as a senior software engineer, and are surprised they can't develop software.


    Any hiring org worth their salt should be able to hone in on this distinction very fast in the hiring process.
    If you are a competent software engineer who just happened to spend time working on interesting academic projects, in a hacky style, I wouldn't worry too much.

    If you've never learned basic software engineering, pick up at least a little before applying for jobs.
    Start out by reading some books (google 'software engineering books') to map out the skills.


  • Registered Users, Registered Users 2 Posts: 1,922 ✭✭✭fergalr


    14ned wrote: »
    One path might be to follow Louis Dionne, author of the amazingly impressive http://www.boost.org/doc/libs/master/libs/hana/doc/html/index.html. He comes from a pure maths background, and only got into coding through needing to write proof programs. That eventually led into C++ metaprogramming (programming the C++ compiler to generate C++ programs), and he came to present at one of the three big global C++ conferences where he was well received. That turned into presenting at two global conferences per year for two years.

    Even just a year ago, his code was clearly written by someone gifted but lacking in professional software design experience. But he underwent peer review to enter the Boost C++ Libraries, and caught up with modern design and practice through facing the critique of world experts. Boost.Hana is now an exemplar of a new wave of exciting C++ 14 only libraries, and will no doubt be enormously influential to the next generation of C++ libraries.

    Multinationals had been chasing him for a while given such an entry into the international conference circuit and the fact he got a library into Boost at such a young age. He turned them down, but eventually Facebook made him an offer he couldn't refuse, so he joins them this summer at their HQ. They paid for his relocation, everything, and shortly he will be quite a wealthy young man.

    I also kind of can't believe Boost C++ metaprogramming is being held up as the example of non-academic, industry friendly, savvy software engineering...

    Yuck...
    :-P


  • Registered Users, Registered Users 2 Posts: 768 ✭✭✭14ned


    fergalr wrote: »
    My experience is you generally write pretty good stuff on here!
    But I think you're a little bit wide of the mark here.

    First off, that is kind of an admission of failure. The better they are at their job, the closer what they think is true will be to the truth. But anyway...

    I'd actually disagree :)

    Much if not most of any specialist knowledge field is a sort of bubble created both collectively by the practitioners and individually by the expert. In many ways, the more expert the engineer, the more what they think is true is make believe with no basis in evidence. Otherwise they wouldn't be an expert pushing the boundaries of what is possible.

    As with any cultural ecosystem, there are a set of widely held biases and prejudices on various topics. For example, most C++ experts dislike Ruby often quite violently. They might have once been able to say why in detail, but after a few years the detail fades and it just becomes a widely held belief which is uncontroversial in C++ circles. Similarly, most C++ experts think Haskell and Python are great, but again often for quite woolly reasons.

    The same thing applies for hiring decisions. Certain universities, genders, nationalities and career backgrounds get preferred over others. It's rare that these biases are publicly mentioned, even in a spoken aside during a meeting, but everyone knows it happens. I tried to reply to the OP that in my experience such a bias against him might apply here because of his career background.
    fergalr wrote: »
    I'm not sure what you're trying to say here; but, source control and unit testing take no time at all to pick up, if you have been writing algorithms for 5 years. An 'academic programmer' could learn that stuff in no time at all on the job.

    Thing is, they on average apparently don't. And that isn't me claiming this, Marian said so, and that's her research field so she is genuinely an expert on this. She mentioned she's on some advisory board for some UK scientific funding agency as apparently the lack of this stuff causes significant auditing problems when there is a failure to replicate results because it's not possible to track down who "broke the software", and apparently some want to enforce the use of a centralised publicly operated source control repo for all publicly funded science where software must be written, and mandate that software implement regression test suites so it becomes impossible to change the software in a result invalidating way. Apparently this notion is extremely controversial, and apparently academics went all up in arms about the idea, and it all got very heated.

    But I'm very much speaking from hearsay and possibly faulty memory now. I have no idea if any of the stuff above is true, but it sounds plausible.
    fergalr wrote: »
    So, honestly, it is a little 'cargo-cult' to think somebody doesn't know what they're doing because they make certain trade-offs that best suit the very prototype style of coding that might be optimal for their academic work.

    We're getting quite far off the OP's topic now, and again I stress I know very little personally about academic programming. But perhaps the above repeated hearsay might clarify how the tradeoffs chosen by academic programmers may benefit them but not their scientific field at large.
    But, it's not that these people learned bad practices in academia, or that their training was deficient.
    It's just that they never learned any software engineering, because that's not what they were doing.

    There is not an invalid argument that a large and increasing chunk of the knowledge industry is really some variant of software engineering.

    For example, most of finance nowadays is just moving numbers between storage silos. One could argue trading is just a stylised memcpy() and you wouldn't be entirely wrong.

    Similar arguments can be made for logistics, a fair chunk of design and art, even music creation nowadays mostly happens inside a computer. Back when I was working on EuroFighter, everybody thought they were building a plane. In fact 70% of the effort went into a software platform. The hardware plane itself became relatively incidental in the big picture of delivering the project.

    I'd argue it's not much difference in academia. Researchers might think they are researching some physical phenomenon or other, but in terms of effort invested I'd betcha an increasing share is going into the computer systems underpinning that research. That makes engineering some software unavoidable, so one could argue it might be wise that everyone thinking they are doing task X accepts it's really a variant of programming a computer.
    fergalr wrote: »
    I also kind of can't believe Boost C++ metaprogramming is being held up as the example of non-academic, industry friendly, savvy software engineering...

    Yuck...
    :-P

    The theory is that using a library like Hana makes most of the yuck go away :). I'm not personally convinced of that yet, but I do think Eric's Ranges v2 a.k.a. the expected v2 of the C++ STL ought to be eventually fairly yuck free around 2020 or so.

    Regarding being a poor choice of example, sure I was limited by the cases I knew the facts of and where I was sure the example didn't mind being named. Had I had more free reign I could have done a lot better.

    Niall


  • Advertisement
  • Closed Accounts Posts: 22,648 ✭✭✭✭beauf


    The op could do some courses to make them aware of good practise in commercial development like agile courses etc. Most people will recognise someone who is curious, interested and a problem solver and not feckless. The hard part is getting through the hr and agencies who just look at paper specs. Direct application to companies might work better. Especially those working in similar areas to the op experience.

    Doing a few interviews would be a learning experience that might prompt areas to study.


  • Registered Users, Registered Users 2 Posts: 13,104 ✭✭✭✭djpbarry


    14ned wrote: »
    I tried to reply to the OP that in my experience such a bias against him might apply here because of his career background.
    Whoops. ;)
    14ned wrote: »
    She mentioned she's on some advisory board for some UK scientific funding agency as apparently the lack of this stuff causes significant auditing problems when there is a failure to replicate results because it's not possible to track down who "broke the software", and apparently some want to enforce the use of a centralised publicly operated source control repo for all publicly funded science where software must be written, and mandate that software implement regression test suites so it becomes impossible to change the software in a result invalidating way. Apparently this notion is extremely controversial, and apparently academics went all up in arms about the idea, and it all got very heated.

    But I'm very much speaking from hearsay and possibly faulty memory now. I have no idea if any of the stuff above is true, but it sounds plausible.
    Well, I work in academia in the UK and I’ve heard absolutely nothing about the above, so I’d be inclined to take it with a pinch of salt. Most academic programmers I know would be only too happy to see more oversight when it comes to developing “academic” software – it’s all too easy to get a paper published, in high-profile journals, with virtually no review or testing of any software that was developed to produce results in said paper (in the life sciences at least).
    14ned wrote: »
    We're getting quite far off the OP's topic now…
    Not at all – it’s an interesting discussion.


  • Registered Users, Registered Users 2 Posts: 13,104 ✭✭✭✭djpbarry


    beauf wrote: »
    Doing a few interviews would be a learning experience that might prompt areas to study.
    Yep, that’s definitely already happened! Had a technical interview that was somewhat soul-destroying, but it was useful in the sense that it highlighted areas I need to work on.


  • Closed Accounts Posts: 22,648 ✭✭✭✭beauf


    Its like a free career guidance.


  • Registered Users, Registered Users 2 Posts: 1,922 ✭✭✭fergalr


    14ned wrote: »
    I'd actually disagree :)

    Much if not most of any specialist knowledge field is a sort of bubble created both collectively by the practitioners and individually by the expert. In many ways, the more expert the engineer, the more what they think is true is make believe with no basis in evidence. Otherwise they wouldn't be an expert pushing the boundaries of what is possible.

    As with any cultural ecosystem, there are a set of widely held biases and prejudices on various topics. For example, most C++ experts dislike Ruby often quite violently. They might have once been able to say why in detail, but after a few years the detail fades and it just becomes a widely held belief which is uncontroversial in C++ circles. Similarly, most C++ experts think Haskell and Python are great, but again often for quite woolly reasons.

    The same thing applies for hiring decisions. Certain universities, genders, nationalities and career backgrounds get preferred over others. It's rare that these biases are publicly mentioned, even in a spoken aside during a meeting, but everyone knows it happens. I tried to reply to the OP that in my experience such a bias against him might apply here because of his career background.

    I'm not disputing that bias exists, and that hiring committees have biases.

    I'm just saying that it's the job of the hiring committee to be unbiased, and the better they are, the better they should be at that job, and the less biased they should be.

    To clarify:
    Each candidate has some quality q
    You can't measure that directly, so you look for signals that reveal information about it.
    (The grade of their schoolwork, their performance in the coding interview etc)
    You try and form some estimate e(q) from the signals you can see.

    The better the hiring committee is at their job, the less the gap (the error) between e(q) and q - that's all I'm saying. If the hiring committee is really bad, they think that certain signals carry much more information about q than they actually do, which increases the error.

    14ned wrote: »
    In many ways, the more expert the engineer, the more what they think is true is make believe with no basis in evidence.

    Maybe you are not making a statement about how the world should be - maybe you are just saying "I have noticed empirically that, sadly, hiring committees tend to be biased, and that more expert committees more biased."

    If so, then they aren't likely to be *that* good, in my opinion. They lack good metacognitive skills - poor self-awareness. If these are the people on your hiring committee, someone is doing something wrong.


    But fair enough if you are not advocating or defending this state of affairs, just describing it.
    Maybe I'm too optimistic.

    14ned wrote: »
    Thing is, they on average apparently don't. And that isn't me claiming this, Marian said so, and that's her research field so she is genuinely an expert on this. She mentioned she's on some advisory board for some UK scientific funding agency as apparently the lack of this stuff causes significant auditing problems when there is a failure to replicate results because it's not possible to track down who "broke the software", and apparently some want to enforce the use of a centralised publicly operated source control repo for all publicly funded science where software must be written, and mandate that software implement regression test suites so it becomes impossible to change the software in a result invalidating way. Apparently this notion is extremely controversial, and apparently academics went all up in arms about the idea, and it all got very heated.

    But I'm very much speaking from hearsay and possibly faulty memory now. I have no idea if any of the stuff above is true, but it sounds plausible.



    We're getting quite far off the OP's topic now, and again I stress I know very little personally about academic programming. But perhaps the above repeated hearsay might clarify how the tradeoffs chosen by academic programmers may benefit them but not their scientific field at large.


    There is not an invalid argument that a large and increasing chunk of the knowledge industry is really some variant of software engineering.

    For example, most of finance nowadays is just moving numbers between storage silos. One could argue trading is just a stylised memcpy() and you wouldn't be entirely wrong.

    That's taking things too far - there is a domain complexity you aren't going to see by thinking about memcpy. That's a bit like saying all programming is just ones and zeros. It might be true, at a very low level, but is not useful - you're not going to build very much with that view.


    14ned wrote: »
    Similar arguments can be made for logistics, a fair chunk of design and art, even music creation nowadays mostly happens inside a computer. Back when I was working on EuroFighter, everybody thought they were building a plane. In fact 70% of the effort went into a software platform. The hardware plane itself became relatively incidental in the big picture of delivering the project.

    Now, that is perspective I very much agree with!
    I read this article recently:

    Although the JPO has given the new 3i software the green light, test pilots at Edwards Air Force Base, California, are still seeing frequent stability issues on the ground. Officials told Defense News last week that development test pilots at Edwards have trouble booting up their jets about once every three flights.
    http://www.defensenews.com/story/defense/air-space/2016/05/09/f-35-program-office-signs-off-air-force-3i-software/84138390/


    I thought it was funny, the world we are now living in.


    I still disagree about academic programming.
    My disagreement is to do with the separability of scientific inquiries:
    you are clearly thinking about big systems projects - things like fighter plane software, or maybe the LHC - giant academic or industrial software systems projects.

    But a lot of academic work has the opposite character - tons of people can do research PhDs, especially in fields such as computer science or mathematics, where they work very independently, and the software they write is very small and self-contained - algorithmic analysis, proof of concepts, data analysis. They use platforms, but are productive working without large teams or software engineering methodology.


    14ned wrote: »
    I'd argue it's not much difference in academia. Researchers might think they are researching some physical phenomenon or other, but in terms of effort invested I'd betcha an increasing share is going into the computer systems underpinning that research. That makes engineering some software unavoidable, so one could argue it might be wise that everyone thinking they are doing task X accepts it's really a variant of programming a computer.

    Computers are becoming a more and more ubiquitous and necessary tool.
    However, software and environments are increasing in power and abstraction all the time.

    As such, I did some research in my PhD working solo (which meant I did not need to use much software engineering) because I could use modern tools, which if I was to do it 40 years ago using assembly language, would have been a large team project.

    The scale of research we want to do is increasing - but the tooling is increasing in sophistication too, enabling individuals to do more independently.

    It's not clear to me that the future is that every research project is a giant collaboration - individual researchers, and small informally manageable teams, are now probably more productive than ever.
    14ned wrote: »
    The theory is that using a library like Hana makes most of the yuck go away :). I'm not personally convinced of that yet, but I do think Eric's Ranges v2 a.k.a. the expected v2 of the C++ STL ought to be eventually fairly yuck free around 2020 or so.

    Regarding being a poor choice of example, sure I was limited by the cases I knew the facts of and where I was sure the example didn't mind being named. Had I had more free reign I could have done a lot better.

    Niall

    Time will tell - colour me sceptical - I had a pretty bad experience with the boost graph library a few years ago, that made me think that C++ with its heavy use of templating, had gone down a dead-end.
    (Having previously been a happy STL user.)

    Python and friends are the near-term future!
    (Medium-term I'm not sure - I'm pretty sure we will be back to static typing - but I don't think it will be C++)


  • Registered Users, Registered Users 2 Posts: 768 ✭✭✭14ned


    fergalr wrote: »
    Maybe you are not making a statement about how the world should be - maybe you are just saying "I have noticed empirically that, sadly, hiring committees tend to be biased, and that more expert committees more biased."

    I never said more expert committees are more biased. If anything, typically truly expert people only bother turning up at hiring committees in order to get someone particular hired whom they are fairly sure wouldn't be otherwise. Hiring committees also vary enormously between orgs, what Microsoft does is utterly different to Google for example. Each has its pros and cons.

    My sweeping hand waving generalisation about biases is part based on personal experience, but much more based on conversations with senior Googlers who set up and repeatedly reformed the Google hiring process until they reach a perceived "good enough" process.

    Interestingly, they really really tried hard to make Google's process objective and meritocratic as possible, and the ways in which they failed and all the unintended consequences they found according to their own internal metrics would make a fascinating book.

    But it's all uncorroborated hearsay if I speak it. Let's just say they failed to fix the bias in favour of Ivy League universities and the bias against women and some, but not all, minorities. This was despite very strenuous internal efforts, but in the end there is so far you can push a population of humans all of whom are behaving mostly subconsciously and quite irrationally (and get upset and offended if you push too hard the empirical evidence that they, in aggregate, are not being rational statistically speaking)
    If so, then they aren't likely to be *that* good, in my opinion. They lack good metacognitive skills - poor self-awareness. If these are the people on your hiring committee, someone is doing something wrong.

    Nobody anywhere I've ever worked likes hiring duties. Interviewing people is enormously disruptive to your schedule, and it's not pleasant rejecting often very nice human beings. It's natural to want to avoid those duties if you can, and sufficient seniority usually gives you to clout to make excuses to get out of it.
    Time will tell - colour me sceptical - I had a pretty bad experience with the boost graph library a few years ago, that made me think that C++ with its heavy use of templating, had gone down a dead-end.
    (Having previously been a happy STL user.)

    You may be interested to know that recent Boost peer reviews are very hard on any "frivolous" usage of metaprogramming in new libraries. You need to justify with ample rationale why it is absolutely necessary.

    In other words, the pendulum has swung back. Back when it was new and cool everyone pushed it too far and made anti-social libraries. Recent C++ 14 only libraries go to great lengths to not expose metaprogramming internals to end users. This will get a lot better again once C++ 17 Concepts land next year.
    Python and friends are the near-term future!
    (Medium-term I'm not sure - I'm pretty sure we will be back to static typing - but I don't think it will be C++)

    STL v2 as currently expected is based on coroutines, generators, list comprehensions and algorithmic views of things. Everything can also be lazily executed, so for example you can apply a sorted view onto some partial range of stuff and the sorting only happens incrementally as needed. Coroutines also happily migrate onto kernel threads if asked, so it becomes a cinch to scale out processing over SMP CPU cores.

    It looks a lot like Python and intentionally so, but has the advantage of all being compiled statically so it is rather quick.

    Niall


  • Advertisement
Advertisement