Big Data - Who cares??

Alba1

There is alot of hoopla in the media recently about the arrival of "Big Data" and how business' that can access and use all the data in their own data warehouses / PC hard drives can take their business to new levels of performance.

Apart from the Googles and Amazons out there, is any one actually working on "big Data" projects on their own business data ?

What are the challenges you find ?
Does it live up to the hype ??

Pat

Ray Palmer

Worked on several project where companies started data diving to figure out where their business was successful. They were able to find products that weren't worth keeping due to sales and space. They also were able to target customers based on their purchases better.

bullpost

What "Big Data" technologies did you use or was this traditional data mining?

Ray Palmer wrote: »

Worked on several project where companies started data diving to figure out where their business was successful. They were able to find products that weren't worth keeping due to sales and space. They also were able to target customers based on their purchases better.

GreeBo

We are looking at analyzing the terabytes of business data that we have collected over the years.

The key is to collect everything and analyse later...you never know what patterns you can find once you start looking, but only if you have the raw data. Collecting data in a processed format is a mistake.

ironclaw

The likes of SAP do this in a huge way. Its very easy to mine data once you have an idea of what you are looking to achieve. Personally it always been very obvious to me, its just media and advertising companies putting the usual spin and swizz on it i.e. 'Big Data' etc.

GreeBo

ironclaw wrote: »

The likes of SAP do this in a huge way. Its very easy to mine data once you have an idea of what you are looking to achieve. Personally it always been very obvious to me, its just media and advertising companies putting the usual spin and swizz on it i.e. 'Big Data' etc.

I dont think its spin, sure we have been mining and analysing data for years, but there has never been anything approaching the amount of (largely social media driven) data that there is today.

Some crazy stat like 90% of the data in the world was created in the last 5 years.

ironclaw

GreeBo wrote: »

Some crazy stat like 90% of the data in the world was created in the last 5 years.

I can well believe it. There was a great program on BBC Horizons about Big Data about 6 weeks ago. Very clever stuff being done Stateside at the moment even as far as predicting crime in L.A.

Peterdalkey

business' that can access and use all the data in their own data warehouses / PC hard drives can take their business to new levels of performance.


When I look back 20 years or so at how we, entered a customer order, printed out a multipart order picking/ delivery note/work order, batch processed sales invoices, then enveloped and posted out a paper copy to the customer, filed a copy in the customer file and an accounts copy in numerical order. Now an order is entered by the customer service/sales person and perhaps 30% of the paper is generated, all filing is electronic, it is mostly mailed electronically as are the monthly account statements. Filing is now a minor irritation compared to the huge chore( expensive) it was back then.
The time and money spent each month on tons of green and white listing paper reports, that were often never used, was immense.
Now it is all electronic, there has been a huge reduction in the business of the number people solely tasked with “admin” . Everybody has access to real-time up to day information on the screen and all half decent ERP systems allow you to access the data in pretty any format you may wish to see it. Even small organisations now have comparatively large amounts of data that may or may not be of particular use to them going forward. Most well run businesses use that data to manage stock levels, predict and monitor demand curves etc etc.
What I think is clear, is that the systems that record that data have lead to significant cost savings, with vastly increased efficiency and much lower costs per unit of business conducted. It may not sound a sexy as “Big Data” and “Data Mining” but it goes on quietly now as a normal part of most businesses. The size of the data is now big compared to the size of the organisation when contrasted to that of less than a generation ago.
It is not size that matters, it is how you use it!

AndyJB

Peterdalkey wrote: »

It is not size that matters, it is how you use it!

How true, well said Peter. Alas common sense is not very common!!

AJB

jmcc

Big Data is important to those of us who use it. (One of the sites in my sig has the hosting history over 361 million domain names tracked and tracks the domain hosting statistics for four million or so webhosters. The deployed database is over 200GB but there are other larger databases in use here for research work.) The "technology" journalists in the Irish media generally have no technology background or any detailed understanding of the issues. You should therefore not rely on those people.

Regards...jmcc

Alba1

Interesting discussion. GreeBo make a very valid point about collecting data. I always preferr to get my data as "green" as possible. There is always a danger when re-baking pre-processed data that you are mis-lead in your analysis due to inbuilt assumptions someone else made.
As for media spin, I think they are just taking their lead from some of the big software vendors such as IBM, SAS and Oracle as they try to sell new updates. The use of Social media "appears" as a major driver for business research- how to use Facebook or twitter data to target specific adverts to your screen...although for me this is why I dont bother with a facebook or twitter account. (that and the fact I dont really want to know that much detail about people )

That also brings up alot of ethical issues about the responsible mining of Big data. Along with the issue of where the data is held and what individual and customer rights are unknowningly violated. EU and US data protection rights are very different and most people dont think twice about where their personal data is stored. I'm sure this will become a bigger issue in the future once the gee-whizz factor of the technology dies down. For now it seems to be confined to a few digital rights activist groups.

P