Share This

In August 2015 we had 4.4 Zettabytes of data (4.4 * 1 followed by 21 zeroes) Of this about 2% has been analyzed. Rest 98% is sitting there waiting to be taken action upon. In the year 2020 the data size will increase to 44 Zettabytes. What can we possibly do with such a data deluge ? Data coming at us not just in volumes but high velocity and variety. This data coming from various sources like sensors, actuators etc, need to be analyzed and we need to take some inferences out of it. Then there are programs which analyze this data. An RDBMS (Relational Database Management System) does not scale well when we look at this kind of data volumes. Hence we have to use NoSQL (Not Only SQL) or New SQL (Analytic DBMS) as possible solutions. The reason why RDBMS are not cut out for this job is that they are ‘Schema on Write’, which simply means that you need a definition of the data before it is written. Whereas solutions like Hadoop are ‘Schema on Read’ which means that you assign a meaning to the data when you read it. There are others like SPARK (In memory database) which can be faster than Hadoop.

Nature is like data and our mind is like the program which analyzes it. No matter how big the challenge that nature throws at us, we rise up to it. Consciousness is the field in which both the seer (program or the mind) and seen (data or nature) exist. Data is further transformed into information and this when it becomes actionable is called knowledge. When we can compare and spot patterns in knowledge that helps us to do the current job better, it’s called understanding or intelligence. the discrimination that we apply as to whether this is good or bad is the Wisdom that we collect. Above the Wisdom is the Ego which can play spoilsport sometimes. But above all this is our Soul. The Witness. Everything that has happened or will happen are stored in Akashic Records or Global Memory. This is the bank that the mind accesses. It’s very much non-local. So if you try to look it up in the brain, you may not get it there. Every cell in our body does a trillion things in a second. And we are made up of trillion cells like these. Imagine the complexity.

So how do we address all this ocean of data ? Nano-computing ? Bio-computing ? Quantum-computing ? The answer lies there somewhere. However this data magnitude is simply not the thing that we will be able to deal with, no matter what. So we need focus. Let’s break down this data or information into chunks that we can swallow and go after it, one piece at a time. Sure, we will miss out on a majority, but still we would have managed to scrape about 80% of the problem by focusing on it. Divide  and conquer. Now comes the program part. What kind of algorithms are apt for ruminating this data size. Machine Learning. Artificial Intelligence. And the works. But our hardware needs to scale up disproportionately to address these challenges. Parallel processing is the answer. Think like the brain. But we are far away in terms of what today’s computers can do to solve such complex data patterns. Data coming from sensors and other devices will break the bank. We need to separate out this data and have a good MDM (Master Data Management) system to recognize the lineage of this data coming to us from all places.

Not all of this data that we see in nature are good. Some are simply noise, hence we need something’s akin to noise cancelling headphones. The veracity of data is important. Our computers (brains) can only handle this much of data. Let’s not over stress our resources else a breakdown is inevitable. However, when we traverse this path, we will come across new avenues which have not been found or conquered by mankind. A new world, so interesting that we lose ourselves completely, in order to have that experience that we are all awaiting for. Moksha or Liberation. And mind you, for God this is just his playground. He is experiencing all this through us. So let’s give Him something that he will never forget. Before we become united with him (Death) let’s explore this world and most importantly understand ourselves. The various dimensions within us and creation. Let’s not leave a stone untouched for God and I’m sure that if we proceed with a clear mind we will soon be staring at the ultimate, that we are all trying to comprehend – The Emperor’s Mind.

 

Handling Big Data

Hey there ! Hope you enjoyed reading this blog ...

Translate »