Introduction

Big Data, Artificial Intelligence, Machine Learning, etc. are the buzz words in today’s digital world. There are thousands of articles online advocating ways in which these technologies can help benefit organizations who choose to adapt it. These benefits vary from making grids more efficient, predicting customer churn to improving operational efficiencies.

 

I am GIS guy who primarily works with Utilities & Telcos. My field of work has naturally made me incline towards the understanding Big Data and the roles it can play in #GIS #Utilities and #Telcos. Despite searching across the web intensely, I found that very few of these articles had answers to “How Big Data can help make GIS solutions efficient?” Upon pondering, I understood that the most of these articles on the web talk about how companies employ Big Data in improving their “external sections” of the Utilities & Telcos, whereas the solutions that I deliver help sections “internal” to these organizations.

External & Internal Sections

Let me try to explain what I mean when I say “external” & “internal” sections of an organization. When I talk about making the external sections more efficient, I am referring to the services that the organizations offer to their customers.

This includes

• Ensuring low down time whenever there is an Outage,
• Offering Tailored services to Customers by analyzing their behavior,
• Any other services that might be offered to make the Customers happy.

So basically “external sections” means the services offered by the organizations with an objective to enhance the customer’s user experience. While “external sections” focus on the deliverable, the “internal sections” are things that assist the Utilities & Telcos in these services.

Internal sections can be

• Efficient Business Process cycles
• Identifying and removing the performance bottlenecks of GIS systems or  any other systems that participate in Business Processes.

That covers the “what” of it. Next up “Why”.

Why is making internal sections efficient a Big Data problem?

Any problem can be classified as a Big Data problem if it involves the 3 Vs (i.e.) Velocity, Variety & Volume. As I mentioned in the above section, the two primary contenders for making internal sections efficient are improving system performance & building efficient Business Process.

Improving system performance is traditionally done either by using profiling (source code analysis) or using system logs. While profiling adds substantial overhead to the already bad performance, use of system logs is a classical Big Data problem. Here’s why:

a. Average size of the log file generated from a regular  user session is about 5-10 MB (Volume)

b. Almost no two user log are similar (Variety) and

c. Logs are generated every day containing different/new information on what the administrator is looking into (Velocity).

Similar framework can also be applied for cases where organizations, either proactively (or) re-actively, decide to improve their business processes. They have to involve numerous stakeholders to understand where the bottlenecks are (volume & variety) and they have to analyse user system logs to validate their claims (velocity).

That leaves us only with the “how?” and therein, as the bard would tell us, lies the rub (reference for the naive). Read on.

Big Data Tools for addressing internal section problems

There are tools available in the market which are designed specifically to address the internal section problems of an organization. For example, there is a Diagnostics for organizations which use GE Smallworld as their GIS platform, ArcGIS Monitor for organizations which use ESRI.

Some of the salient features that these tools share include:

Intrusive Monitoring

These tools bind closely with the GIS solution and are hence able to track almost everything that happens inside the GIS application in real-time. This means they allow real-time tracking every single user click, every database transaction, frequently/rarely used tools, time taken to perform a particular task or to execute a particular query, etc.

Negligible Impact on the User Experience

Intrusive monitoring does raise concerns of these tools adding additional performance overhead to the GIS systems that they are monitoring. Most of these performance motoring tools, at-least the ones mentioned above, are designed to ensure that they have little or no impact on the user experience.

Here is a two minute video of me explaining how Diagnostics makes it possible.

Easy to understand results

These tools leverage the power of Big Data to crunch the rapidly flowing stream of humongous real-time data and make the results available to the administrators in the form of easy to dashboards. These dashboards allow administrators to identify bottlenecks in their GIS systems (or) business processes and take necessary steps to resolve them.

Pro-Active Maintenance

Ability of these tools to monitor almost everything that is happening inside the GIS solution in real-time allows administrators to be pro-active in their maintenance effort. For example, administrators can configure these tools to send them email whenever the size of the database reaches a particular value so that they can extend its storage capacity.

Integrated Monitoring

Since all of the processed monitoring data are made available in a single platform, it allows administrators to not only identify bottlenecks but also to understand tools/processes which are getting affected by them. So for example, if the administrator is able to identify queries which take a lot of time to execute, he can then go ahead and also find out tools which use these queries or they can go a step further and identify Business Processes which are getting directly affected by these queries. These information can then help justify any performance enhancing investments made by the organization.

Summary

Utilities and Telcos can gain massively by actively investing in solutions that help turn the beam of analytical prowess, offered by Big Data, towards their internal sections. Some of these organizations, having identified this opportunity, have already began making such investments. This is a brilliant step forward by these giants as this promises to illuminate many-a-previously-eclipsed performance snags, thereby resulting in happier users, smoother and more efficient business processes.