Understanding Big Data

Understanding Big Data

Big data processing

Look around at the technology we have today.....Its all about Data. As consumers, we have an increasing appetite for rich media, both in terms of the movies we watch and the pictures and videos we create and upload. We also, often without thinking, leave a trail of data across the Web as we perform the actions of
our daily lives. Not only is the amount of data being generated increasing, but the rate of increase is also accelerating. From emails to Facebook posts, from purchase histories to web links, there are large data sets growing everywhere. The challenge is in extracting from this data the most valuable aspects; sometimes this means particular data elements, and at other times, the
focus is instead on identifying trends and relationships between pieces of data.

Large companies have realized the value in data for some time and have been using it to improve the services they provide to their customers. Consider how Google displays advertisements relevant to our web surfing, or how Amazon recommend new products or titles that often match well to our tastes and interests.

These corporations wouldn't invest in large-scale data processing if it didn't provide a meaningful return on the investment or a competitive advantage

Key aspects of Big Data

1. Some questions only give value when asked of sufficiently large data sets. Use the viewing history of ten million other people and the chances of detecting patterns that can be used to give relevant recommendations improve dramatically

2. Big data tools often enable the processing of data on a larger scale and at a lower cost than previous solutions

3. Big data tools allow data volumes to be increased while keeping processing time under control, usually by matching the increased data volume with additional hardware

4. Sufficiently large data sets and flexible tools allow previously unimagined questions to be answered

Hadoop

An open source framework for large-scale data processing. Hadoop was not created in a vacuum;It came because of the explosion in the amount of data being created and consumed not just by multinational but by smaller organizations as well. At the
same time, other trends have changed how software and systems are deployed, using cloud resources alongside or even in preference to more traditional infrastructures.