Robust data architecture is critical to properly capture, preserve, access, and transform this data in its journey – not just in cloud data centres, but at edge servers and endpoints
As the world tackled the threat of the Covid-19 pandemic, a key role was played by connected technologies and Internet of Things (IoT). Add to this the sharp rise in the number of people working as well as studying from home, and it has created a huge upsurge in the demand for data. Industry analysts estimate there will be 29 billion connected devices by 2021, and by 2025 a person will be interacting with a connected device 4,800 times a day. That is one in every 18 seconds!
This has led to data being generated at a pace which is difficult to comprehend. In fact, in 2020, people created 1.7 MB of data every second. According to ABI Research in its Edge Analytics in IoT report, only 0.0013% of generated data will be captured, and out of it only 4% will be transmitted. What about the other 96%? Just imagine the potential insights we are unable to derive from the huge amount of data that is not captured and stored.
Therefore, the top priority for cloud and data providers will be to scale up data capacity, accelerate cloud compute, and share disaggregated flash storage resources.
The rapidly evolving Internet of Things (IoT) landscape
Today a large number of industries are generating massive amounts of data (including from IoT devices), be it healthcare or pharmaceuticals, aviation, automobiles or others. In any of these industries, the number of use cases and the amount of data being generated are difficult to limit.
The healthcare industry could benefit largely from aggregated IoT data as it can help identify opportunities to improve the health habits of end-users by using data-informed decisions and could also create more personalized and proactive treatments, especially as telehealth and remote healthcare continue to progress. It is predicted that by 2022, over 1 billion wearable devices will be in use around the world – generating data to track sleep patterns, measure daily movements, and identify nutrition and blood oxygen levels.
Similarly, in the automobile industry, large-scale IoT devices like autonomous and connected cars are loaded with sensors, cameras, radar, and other devices generating data – estimated to reach 2TB+ per day. This data can help to make real-time driving decisions using various technologies and to provide personalised infotainment and in-vehicle services that improve the passenger experience.
Cloud, Edge, Endpoints – The IoT Data Journey
We know that connected machines, fueled by the Industry 4.0 transition, and wearables are projected to contribute an increasing amount of IoT data - the question is how to store it successfully?
Data storage can be done in three ways. We can start with the cloud, where high-capacity drives – now reaching 20TB – store massive amounts of data for big data use cases like the healthcare industry.
Then, there is edge, where data is often cached in distributed, edge servers for near real-time usage such as in autonomous vehicles, cloud gaming, and manufacturing robotics.
Finally, we reach the endpoints with real-time usage, where data is generated by connected machines, smart devices, and wearables.
The key is to reduce network latencies and increase throughput between these layers (cloud-to-edge, edge-to-end points) for data-intensive use cases. 5G could offer a potential solution by creating “data superhighways” for latency and bandwidth-sensitive innovations.
Moving from general-purpose to purpose-built data storage
Robust data architecture is critical to properly capture, preserve, access, and transform this data in its journey – not just in cloud data centres, but at edge servers and endpoints. To get the most value out of IoT data, it is advisable to move toward purpose-built architecture which uses devices, platforms, systems, and solutions that maximise the value of data for real-time IoT use cases. This would ensure that the needs of IoT applications and workloads for consumers and enterprises are fully met.
Hence, it is prudent to ask the right questions and move toward your own specific storage strategy. As we move forward, we have to accept that there will be the creation of immense amounts of data from all quarters. We need to be ready with a plan to be able to store and utilise it in the best possible manner to our advantage.