HOW TO: achieve a performance boost with the PI Web API
At Cuurios we are keen to develop software solutions that use your process data to deliver valuable insights. The OSIsoft PI Asset Frameworks (PI AF) provides a hierarchical, asset centric model of data with detailed history, which is indispensable to monitor and further analyse vital production data in the industry. It means millions of records of real-time and historical data. This data is invaluable, provides not only useful insight but also the possibility of improvement for better control and efficiency when processed and displayed right.
Have you chosen wisely?
Having a database filled with vital information is not more than a pile of values, up until the moment when you put that data to work. Every number, every date, every remark can be useless or on the contrary: it should help your company to work smarter, not harder.
Making the right connections, creating useful relations, calculations and transforming your data into concrete actions is what our motto, data to action means. Poor choice of data to process means poor, sometimes even useless insights. Modern technology provides you the tools, to make your company’s job easier – an overflow of data on the other hand will just make everyone’s life harder.
Let’s get technical
During our several encounter with PI AF (using the Web API) we learned some useful lessons and tricks. For those with an IT experience, the fact that the extended use of a framework also means dealing with its limitations and shortcomings is not all too surprising. In case of the PI AF, the main obstacles we had to overcome were related to the limit of search queries, result sets, and performance issues due to the excessive amount of information.
One of our use cases were monitoring and handling inhibition values on offshore gas production platforms – more precisely detecting when an inhibition for any checkpoint is turned on, and track it’s changes until it is turned off. The added value lies in a well-managed and monitored safety system.
Finding the PI points
These checkpoints do not just belong to one asset or even one asset type, in fact, they can be found all over the asset tree. This is where the PI AF built in search query comes to the rescue.
The above query gives us almost 6.000 results. Although PI AF has the possibility to set the size of the result set, the built in maximum limit is 1.000 item per page, which means a minimum of 5 REST requests in parallel to process the almost six thousand items.
What makes it complicated to work with PI AF at the very beginning is the fact, that the above query does not provide us values, or any actual data about the PI point other than it’s WebID, the unique identifier used in PI AF. As a next step, querying the WebID gives you direct access to the actual values of a certain checkpoint.
Performance with over 6.000 REST request?
Although now we have all the inhibitions points directly accessible with their unique WebID, the reality is, checking all of their values takes roughly six thousand get requests. If you can’t imagine what that exactly means, below an example:
6.000 of these requests need to be sent and processed. It takes in our particular setup over 5 minutes. Can you imagine pressing a button and waiting for over 5 minutes for the result? Let’s all just be honest – the reality is, most people lose their patience even after a couple of seconds, because even that loading time is unacceptable nowadays, let alone 5+ minutes.
Let’s build stream sets!
Implementing a working Refresh button at this point was the last straw. While the updates where running only hourly in the background, this 5 minutes wasn’t great, but still barely noticeable from a user point of view. Up to the point, when we placed a button on the page for manual refresh purposes. Then it became clear: we need another solution.
After some research we found out, that the PI AF Web API supports bulk data retrieval even for unrelated pi points, the only thing you need is the list of WebID’s. These queries are called stream sets and are providing fast bulk retrieval.
Only a couple of things to keep in mind:
- the URL has a length limit, in our particular setup we could retrieve the values for approximately 400 WebID’s in one stream set (the length of the webids varies, make sure you test this extensively in your setup);
- it is necessary to adjust your method to handle the response accordingly, as the result is a list of json objects called Items,
- if any of the WebID’s in the stream set runs on error, the whole request runs on error, meaning even one unavailable WebID and the whole request goes down the drain.
- Implementing stream sets meant we could reduce the number of REST request from 6.000 to 15. Additionally, a stream set request response time is not noticeably longer than a simple value request, so our performance got a big boost: from over 5 minutes down to an average of 10 to 15 seconds.
Building something really meaningful on top of a database with literally millions of available records, if done smart, can improve the way you organize your work. Focusing on what you need to know, what you need to see from those information at first blink means you can put the carefully collected data to work. It is not going to make the decisions for you, but it can help pointing out when and where you need to make decisions, take actions.
You can share this blog post
More blog posts
It is never easy, to let go of things, that accompanied you from the beginning. But change isn’t supposed to be easy. With heavy heart towards the old, but also excitement for the new, we are saying goodbye to our old office and old website.
With our increased number of colleagues, our office simply does not have enough room anymore, to give space for all of our members, and while we are mainly working from home according to the current recommendations, we are hoping that soon enough, we will be able to have a full house – in our new office in the new NEXT Delftbuilding!
The first time I heard "she/he is a developer" was when I started working at a software company in 2012. We were told we can ask them about everything we want and I thought, "wow, this is so cool". From that day for about 6 years I worked with engineers on a daily basis an average 10 hours each day. Due to the fact that all things were new to me, I googled every word they said in standup.
I learned from them, grew with them, and became more and more interested in coding. Imagining millions of people are using something based on my idea and I imagined that I was the one who built it. My dream was to create my own website and share with the world everything I was passionate about, like a movie director.
New chapter of my life
“Change you job!”
“This is the thing you always wanted! “
“This is your destiny!”
These words were etched into my head over the years. The thought was both ambitious and frightening.
At first I started learning by myself, reading product codes, participating in technical solution design, and enjoying the moments as much as I could. Finally, I heard the voice clearly and decided to follow my heart. Thus, I quit my job and started the new chapter of my life.
I knew I would face big challenges, but I prepared myself for them. Even though I got frustrated at times, cried for hours, and did many other things, I had faith in the path I had chosen. Thank you to my best and lovely friend who supports, teaches, and pushes me, especially when I feel frustrated.
In less than a year, I began interviewing for developer roles. Each interview taught me something new.
In between, I began my project "Book Lovers" - one of tens of ideas on my long list. At the beginning, I had to design, develop the backend as well as the front end. BOOM. How much work you have to do!
As part of this process, I applied for a frontend developer role at Cuurios.
I will never forget my first impression of Cuurios. I was full of stress on the day of my interview, but they were so gracious to me. The company's co-founder asked me questions and listened carefully to my responses with a smile on his face.
On the other side, the CTO was reviewing my project and asking technical questions, but before I could answer he said, "It's okay if you don't know it, it's totally fine. You are a junior developer and you will learn it soon." He said this, and at last I could breathe.
Now I'm working with them at Cuurios, and I'm so happy I made this choice.
Open-minded managers who accept your ideas, count on you, and encourage you to make decisions and mistakes, along with phenomenal colleagues who support you, teach you, and guide you through the learning process.
Now, when I finish a feature or task and then see the impact, I am excited, high-fiving myself and more eager to get started on the next one.
Curios about the world
My biggest hobby when I'm not working is reading books and watching documentaries. I am fascinated by physics, the universe, history, and by my favourite writer and thinker, Stephen Hawking. By reading about the past, forming opinions about the future, and putting those into the present, I am taught how to think, behave, and make my mark on this world. These things lead me to the conclusion that I should try harder and be better, and I think being a developer will help me achieve these goals.
Last but not least, I enjoy working with code and solving problems for users. I am a strong believer that I am on the right track and will help everyone who has a deep love for coding!
Happy to work with you Anna, Tamara, Michael, Leen, Gaetan, Adyasha, Marco and Andi!
Does not matter if you are 5, 15 or 50. Everyone knows this creeping moment in the middle of the night when you suddenly „feel” the monster under the bed. You know it's not real but you will still pull your feet back under the safety of your blanket.
The reason is the unknown. You know this room like the back of your hand, every corner, every furniture, every color, but in the dark, it looks awfully different, it holds the possibility of something going terribly wrong.
Companies that has been around for a while, that has known methods, a consensual way of working, are tend to have their ways set in stone. No matter how tedious it is, they know it like the back of their hand, and the unknown future that comes with digital transformation is their monster under the bed.
Curiosity killed... my job?
„You just export the data from here, and then take this blank excel sheet, and copy the data in here. Make sure you make no mistakes, check it 2-3 times, and then compare it there, and you e-mail John if this one is above the target range, and message Carla if that one seems to be showing a decreasing tendency over 3 months.
For now you can find all the previous months in this folder, but if you do it long enough, you will just know and won’t need to check back. If all is done, just email the final pdf to the list shared in here, but make sure to double check it, it changes regularly.”
Of the above „job description”, every word can be automated.
Human workforce is one of the most valuable asset a company has – and still some people are just wasted away on doing monotonous data management, generating spreadsheets and nudging people to look at them. It would be great to instead have empowered employees that take actions on automatically generated spreadsheets and dashboards.
Nevertheless, you most likely face resistance when the monster of change is creeping in: people will just want to stay in the safety of their comfort zone, where things are unchanged and their job is safe.
Talking about domain hierarchy, moreover managing domain structure can be a challenge. When it comes to data management, analysis and display, more often than not developers takes the classic road: tables, lists and charts are in charge of displaying your data.
Using one of the above techniques to display the structure of a domain while not impossible, can be very confusing and challenging to use and oversee as a user. Structure and hierarchy are best displayed visualized, for example as diagrams.
Developed and tested for a business user
This may sound contradicting, but a lot of software are made by engineers for engineers. It has great tools, cool solutions, and everything available what an engineer would require and consider awesome. But having the right tool for the right job means not to think as an engineer, when designing or testing a software.
Since day one as a software engineer, I think of myself as an end-user with the tools of making the changes happen. That was the leading idea behind the asset management tool, every button, every functionality was designed and placed the way we would see it comfortable during the everyday use.
Domain build from scratch
One company has departments, others has platforms, solar plants or windmill parks. The common ground is they are all organized in a unique structure, where some technical terms may be included, but eventually, they all are very specific to the given organization.
For example solar plants has multiple rows all built up from individual solar panels, but they can also be part of a higher hierarchy grouped by location. Each asset in this hierarchy has its own unique name, that tells nothing to an outsider, but gives a lot of information for the members of the company.
Fitting these structures and relations into a premade hierarchy with limited connections is not only hard but can make the asset hierarchy representation detached from reality.
Your assets, your connections as you know it
Our tool provides you with a flexible, unlimited configuration, so each and every asset will have its own, well-known and company-wide recognized name in the system. No need to learn a new structure, you can just build your own from the ground up.
Our objective here at Cuurios is to provide the tools that will digitalize and streamline industrial operations, foster collaboration and ultimately unleash an industry data revolution. One of the very valuable and logical use cases for our software platform is the ability to manage, assign and track production related actions. These actions are (in many organizations) currently not managed in a standardized way and I’m sure most of you recognize all these different action lists in various Excel files on shared drives you’re supposed to keep an eye on…
What is it that we do differently?
Exactly that is what motivated us to do it differently, starting by listening carefully to the key criteria we collected during user interviews and conversations with clients. The resulting key criteria we sketched on the board at that time were the following:
- User friendly, intuitive to use
- Performant, also on remote connections with limited bandwidth
- Flexible, customizable and scalable
- Asset centred, industry ready
We want our users to like working with our tools by motivation and not because they are forced to. This simply means we put user experience as our top priority. The simplest things can make a huge difference when it comes to usability. During the first trials with users, we’ve put all our effort in it to make sure every button, entry field or other user interaction in the tool is logical and intuitive for them (and not just for us as product owners :) ).
What does it imply if we put user experience first?
The tool is optimized for use in limited bandwidth environments, resulting in a very snappy and performant UI, even in remote locations. This is the result of a strong focus on the optimization of the application data structure and architecture as well as the optimization of all our data calls (only load the data you need seems a simple philosophy, but unfortunately not the reality in many applications).
Flexible because we understand not every company is the same. So if a client is used to name the priority of an action ‘severity’, they can configure it to be labelled as severity. And if you want three simple straightforward severity levels to select from or five or ten, it’s your choice. If you want an action to be constructed of five simple properties or twenty, all fine! We basically allow you to fully customize the template of your actions and their lifecycle (workflow) to fit your case.
Why is it such a good fit for asset-rich industries?
Asset centred because we focus on solutions for asset rich environments. Actions in these environments are simply always related to assets. A plant, a unit, a pump a vessel, you name it. Your asset structure is represented in the solution by importing the asset hierarchy and definition from your asset management tool. This allows to link the actions to the asset it concerns. It also allows you to set the user permissions linked to the assets.
And the best of all? You can fully automate the creation and the state (i.e. from open to closed) based on your streaming production and asset data. Translating your data to meaningful insights using simple rules, thresholds and complex algorithms has never been so easy. You’ll never miss any of these insights as we provide powerful features, to track, assign and manage these actionable insights.
Want to learn more? We’re happy to demonstrate our product: please leave us a message at: info@Cuurios.com and we’ll contact you to set up a demo!
Delft, 29 June 2021 - The Rabobank Startup & Scale-Up team has provided Cuurios with substantial growth funding to further scale the company. Cuurios is a fast growing software technology provider transforming the industry and redefining leadership in a digitized world. The combination of an investment round closed earlier this year and the growth capital raised today, Cuurios is in the position to gear up for an exciting journey!
Our objective is to provide the tools that will digitalize and streamline industrial operations, foster collaboration and ultimately unleash an industry data revolution. For one of our clients the automated and data driven production advises in order to produce to the optimum and avoid asset failures contributed to a 3% production increase.