Analysis of OsiSoft’s PI SMT (System Management Tools)

This article will discuss how the SMT perform access to OsiSoft’s PI Point in the Current Values Plugin. This plug-in is used to view the real time current value of selected PI Points.

This plug-in is governed by a module in OSIsoft.SMT.Plugins.PICurrentValue.dll file. So, when the user clicks the search button after entering the proper tag name for tag selection, the module will call pisdk!CPIPoints::AddPIPointsInternal function to actually retrieves the PI Points.

This is the implementation of IPIPointsPrivate of AddPIPointsInternal in pisdk.dll which has IDL definition prototype as follows:

The psa2DAttrArray structure is array of retrieved PI Points structure. Let’s examine it:

As you can see, the structure contains header information and one or more records, depending on tag query result.

But how the SMT client connects to the server to retrieved the PI Points data ? Connection is implemented using PI Service called PI Network Manager. This module is executed as windows service. The executable name for this service is pinetmgr.exe.

For the purpose with communication with the client, the pinetmgr.exe will create a named pipe, for example in my machine is called PI3\Local:Pipe.

The client will then perform connection to this pipe using CreateFile in Kernel32.dll module. Then sending and receiving is done by using WriteFile and ReadFile respectively.

The data exchange is using Windows SSPI (Security Support Provider Interface) API encryption/decryption method.

Here you can see that the SMT will perform the communication with PI Network Manager using the given pipe name above. If the pinetmgr.exe service is not active, it will give error message as follows:

Error: Pinetmgr service is not running and caller has insufficient privileges to start it. Detected in PISDK subroutine New(). Application Exiting.

But again, where the actual PI point data came from ? This can be done by examining the pinetmgr.exe service.

When the pinetmgr received data from the client’s named pipe, it will try to communicate with PI Data Archive Server using port 5450. What program that is listening on port 5450 in PI Data Archive Server ? Also pinetmgr.exe.

The data is in encrypted format. Then it will receive the data, also in encrypted format to be forwarded to client’s pipe.

So, you can see that pinetmgr.exe can have a dual role, one as a client and one as a server that listen on port 5450. Now there arise the question, how to setup the pinetmgr.exe that can act as a server ?

To examine the above question, the pinetmgr.exe is copied to a testing server, then perform create service as follows:

Then perform the start service, but on detailed examination, the program still behaves as a client. Try to perform registry copy of HKEY_LOCAL_MACHINE\SOFTWARE\PISystem\PI from source server to testing server, re-start the pinetmgr.exe, but I have an error in Event Viewer as follows: Error locating rendezvous path.

This error is rather obscure and without the given debug symbol, it will be difficult to check for the reason of the above error.

The pinetmgr.exe service seems can be run at command line:

But performing the debugging using WinDBG do not arrive at the above error, but instead, it has different error message in event viewer which says Error 1063: Unable to start Service Control Dispatcher.

This is caused by the service tries to call the function StartServiceCtrlDispatcher and this function is failed. Let’s check why this is so ?

This is caused by some routines inside pinetmgr.exe that tries to call StartServiceCtrlDispatcher with the assumption it is called from the Service Control Manager.

So there’s no other way than to perform debugging inside the SCM, but several steps should be done before it is possible.

Perform the addition of registry values ServicesPipeTimeout and WaitToKillServiceTimeout inside HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control as follows:

Then inside the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options perform addition of pinetmgr.exe key with content as follows:

So when the service is started, it will call the WinDBG that waits in the TCP port. Then using another WinDBG to try to connect to this port to perform debugging process.

And in this fashion, I notice that the error come after it is processing pisubsys.cfg. By using procmon.exe it is found that the service tries to find this file in the wrong location. This can be fixed by updating the InstallationPath key in HKEY_LOCAL_MACHINE\SOFTWARE\PISystem\PI.

After this, there are no service error exception, but I notice that the service is still not listening in the intended port of 5450.

After performing close examination of the source server that runs pinetmgr.exe, I notice that there are different size of pinetmgr.exe inside the PI and PIPC folder. Let’s perform delete of current create service and try to create the service from the one inside PI folder:

But still, the pinetmgr.exe service is not listening on port 5450. Why ?

Later on, I came across the link from OsiSoft’s web side https://techsupport.osisoft.com/Troubleshooting/KB/KB00685, with detail that On startup, pinetmgr will not listen to port 5450 until all PI core subsystems are fully started and their RPCs published with pinetmgr. In other words, the other subsystems must start and tell pinetmgr that they have started. In addition, PI Archive Subsystem (piarchss) will wait for PI Snapshot Subsystem (pisnapss) to fully start before publishing its RPC. However, it does not wait indefinitely. It waits for 30 minutes and if pisnapss is still not running, piarchss will shut itself down. If pinetmgr thinks piarchss was not started successfully, then pinetmgr will not accept any connections.

You can find list of PI core subsystems by searching through the net using “PI Data Archive core subsystems” keyword.

So, I perform service registration on remaining PI core subsystems, and then tries to start each service, but I found the service error as follows:

Failed to load tables for snapshot from C:\Program Files\PI\dat\piarcmem.dat. [2] The system cannot find the file specified.

Message ID: 2126

Fatal error in PI subsystem pisnapss: Failed to load archive memory tables, status: [2] The system cannot find the file specified.

Using procmon.exe, it is found that pisnapss service is trying to load non existent piarcmem.dat file in DAT folder. This file should be copied from the source server, but since it is used by the running service in production environment and can’t be stopped, I use hobocopy utility to perform copying process.

The hobocopy.exe installation file I used is hobocopy-unstable-64bit-20110505-01. Then I perform copy command as follows:

Then there are another error in pibasess subsystem which says Fatal error in PI subsystem pibasess: Failed to load PI configuration database, status: [2] The system cannot find the file specified. And also Fatal error in PI subsystem pibasess: Failed to load Acl Table, status: [2] The system cannot find the file specified.

By monitoring procmon.exe for missing files and perform shadow copy of the required files, the pibasess service is now can be executed without any error or exception.

Then I came across a rather cryptic error which says Fatal error in PI subsystem piarchss: Archive Point Count: 3110, Point Database count: 3112, status: [-11162] Archive and Base Point Count Mismatch – Contact Technical Support.

This is because the piarchss subsystem tries to perform comparison between piarcmem.dat and pipoints.dat and if the record count is not the same, it will throws the above error.

The solution is just make sure that all the piarcmem*.dat is copied from the source server to destination.

Then all core subsystems can be activated successfully and the pinetmgr service is now can listen on 5450 port:

You can leave a response, or trackback from your own site.

Leave a Reply

Powered by WordPress and Bootstrap4