does "big data" employment exist in ireland?

Unemployable

or is it just airbrushed database administration?

ChRoMe

Unemployable wrote: »

or is it just airbrushed database administration?

Its airbrushed DBA everywhere tbh.

jmcc

Apparently it does. However it tends to be quite specialised and, in some cases, highly mathematical. But that's probably the theoretical side of things.

Regards...jmcc

SalteeDog

Groupon were recently looking for back end developers/data analysts who had experience in Hadoop and NoSQL databases.

mightyz

I love these marketing propaganda..
Big Data, Cloud Computing.. etc

How big is big data? 10TB, 1000TB or PRISM?

keith16

SalteeDog wrote: »

Groupon were recently looking for back end developers/data analysts who had experience in Hadoop and NoSQL databases.

Meh- Groupon are a bit of a fad tbh....

mightyz wrote: »

I love these marketing propaganda..
Big Data, Cloud Computing.. etc

How big is big data? 10TB, 1000TB or PRISM?

It's not about the size alone, it's the complexity too. Big data analytics is a massive area and is not marketing speak in the same way "cloud" is...

Manach

Big data also seems to be a step beyond running analysis on some of the data-sets and does seem to involve other technologies beyond basic SQL.

Feathers

ChRoMe wrote: »

Its airbrushed DBA everywhere tbh.

Surely big-data as a term is a field, rather than a role though, no? If you're a DBA working with big data, you'll be doing DBA stuff, if you're a programmer you'll be writing code.

IRLConor

Big data is very much a relative term. It's about working with amounts of data so large and/or complex that the tools available aren't powerful enough to process it easily or in a timely fashion.

Big data for a small firm could be a few GB scattered across a couple of otherwise innocuous MySQL servers. The same volume of data would be trivial to a company that already has data centres full of fast machines under management with a toolchain already in place.

Also, it's hard to take the phrase "big data" seriously after you've seen this:

fergalr

IRLConor wrote: »

Big data is very much a relative term. It's about working with amounts of data so large and/or complex that the tools available aren't powerful enough to process it easily or in a timely fashion.

Big data for a small firm could be a few GB scattered across a couple of otherwise innocuous MySQL servers. The same volume of data would be trivial to a company that already has data centres full of fast machines under management with a toolchain already in place.

I think thats one pretty good way of thinking about what 'big data' means - that 'big data' refers to the difficulty an organisation has in processing and dealing with their data. This ties in with the general idea that you are talking about 'big data' when you are talking about data that is too big to process normally, in some sense.

Alternatively, we could write the whole thing off as a silly marketing buzzword, that doesnt mean anything. In some sense, today, maybe thats the most accurate (!)

We could also say that 'big data' reflects a business change, because its the idea that a firm's data, perhaps gathered passively, now contains actionable intelligence. Massive firms have been doing this for years (think walmart), but now the practice is spreading.
So perhaps 'big data' is really more of a term for the change in business practices, rather than a technical change. As such, some techies might look down on it - 'cloud' sometimes incurs similar scorn - but that'd be adopting a very narrow techie view.


But, theres a final meaning, which I think is really interesting technically, and marks a change in how we approach certain technical problems - if we are generous to the term.


You know the way people sometimes say that a shop, selling X online, can exist profitably, in a way that a bricks-and-motor shopping selling X could? Not just because its margins are lower - but because:
- it can have a bigger inventory of stock than any retail location
- it can serve a wider catchment area

Those two things can result in a shop that is successful by servicing the 'long tail' of demand. The shop can make money by catering to a large number of very small customer segments, each of whom have very specific tastes, and, while none of the individual customer segments would be enough to make a shop profitable, the combination of all the segments can actually be bigger than the mainstream market.

Thats the idea of the 'long tail' - and maybe its being oversold, but it serves as an interesting analogy, showing how you can a change in quantity can result in a qualitative change.
Like that phrase 'quantity is a quality all of its own'.


There are certain CS problems - Natural Language Translation is now a famous example - where CS people spent huge numbers of human years of time to try and come up with good linguistic and rule-based algorithms to do speech translation.

Now, there were also statistical techniques, based largely on what words tended to occur suspiciously often in the same version of a document in two languages - (e.g. if every time you see 'Yes' in your english documents, 'Oui' appears nearby in your french documents, perhaps 'Yes' is 'Oui' in french?) But these methods didnt give good accuracy in practice.

But then, in whats now a famous story - whats now Google translate came along, and applied this simple, brute force, statistical technique, but they applied it on a *scale* far greater than had previously been done.


And, its a bit like the 'long tail' example with the shop. Now the machine translation would suddenly get acceptable accuracy - using the pretty dumb algorithm. But only when you had such vast training data that you had seen huge numbers of examples. Until you had these vast numbers of examples, the technique is useless.

And, that, for me, is what 'big data' means. Its when we have problems which can almost be solved by the data itself, using very simple techniques, but which only become accessible when we get a certain density of examples over the training space for our algorithm, which, in practice, we can typically only get due to having vast amounts of data.

And I think thats whats driving a lot of the 'big data' change at the moment. I think its been driven by machine learning, and by the relative ease with which we can apply machine learning techniques (or data mining, if you prefer to call it that) to extract good signal from our data. With sufficient data, the noise averages out, and we get good coverage. But, for many problems, we need very large amounts of data before this becomes true enough to make our solutions work. Theres a bit of a revolution in parts of machine learning that looks like its brewing at the moment, with 'deep learning' tech, which again, only works with very large scales of data. Interesting times.

If you want to watch Peter Norvig talk on this exact topic for an hour, this is a pretty good talk on 'the unreasonable effectiveness of data': http://www.youtube.com/watch?v=yvDCzhbjYWs

On the original topic, I'm in a small startup working in this space; there is some very technical work in this general area in dublin, but not a huge amount, I dont think.

shootermacg

Big data analysis heavily involves MDX queries and OLAP.
It can be quite tricky finding good calculated members etc, definitely a specialised field and can easily leave you with a migraine ^ ^
They used it heavily in Intel when I was there and its primary use was creating pivot table views (generally in excel) for the BAs so they could then process the information.
It surprises me that none of the companies I've worked at since have bothered with OLAP, the amount of money they spend on static reports really gets on my nerves.

fergalr

shootermacg wrote: »

Big data analysis heavily involves MDX queries and OLAP.
It can be quite tricky finding good calculated members etc, definitely a specialised field and can easily leave you with a migraine ^ ^
They used it heavily in Intel when I was there and its primary use was creating pivot table views (generally in excel) for the BAs so they could then process the information.
It surprises me that none of the companies I've worked at since have bothered with OLAP, the amount of money they spend on static reports really gets on my nerves.

Would you then consider someone using, say, Hadoop on a large distributed dataset with billions of entities, to not be doing big data? As they aren't using OLAP at all, and might even be doing quite simple queries, or offline reports?

I'm not meaning to contradict you or say you are wrong; but I would have thought that there were large numbers of people now doing 'big data', but not using the traditional tools of data warehousing.

shootermacg

fergalr wrote: »

Would you then consider someone using, say, Hadoop on a large distributed dataset with billions of entities, to not be doing big data? As they aren't using OLAP at all, and might even be doing quite simple queries, or offline reports?

I'm not meaning to contradict you or say you are wrong; but I would have thought that there were large numbers of people now doing 'big data', but not using the traditional tools of data warehousing.

I was talking about "Big data analysis" and in my experience bigger companies still keep their data in smaller regional cells, whereas they do now days make efforts to standardise their data.

I was working with millions of rows of data at intel, so many in fact, that a lot of my work involved importing the data into warehouses and then standardising the data before offering up views to the management for analysis.
By far the hardest part of my work was standardising the data, quite a lot of binary trees were formed at the time, let me tell you that ^ ^.
The came recursive SQL which made my life a lot easier.

I've since moved on to a more application dev role, but I've seen a lot of jobs offering big money for people with experience in the field. Makes me want to go back and take it up again.

Unemployable

Thanks for the replies but I suppose what I was trying to get at is that Im thinking about doing a course in data analytics because I enjoyed some data mining and machine learning I did in a previous course and Im interested in statistical analysis but from a quick bit of research it seems like the only jobs youre likely to get from the course are basically tarted up database administration which I already have a basic qualification in and I dont want to waste my time, money and energy to end up just as qualified as I am now. So I guess the heading should be is there any data analytics jobs in Ireland?

stevenmu

That sounds more like Business Intelligence, than Big Data, and there's solid demand at the moment for BI people and services.


And if you can do your Business Intelligence by processing Big Data in the Cloud, people will be throwing bags of money at you.