OEE Suite Self Service and Troubleshooting guide

Vorne Industries has been producing its next generation (V2) Production Monitoring Device for several years now.

Deb-Tech Systems has been offering the OEE Suite, software that offers real-time, tactical and strategic reporting, as well as alerting and the ability to report on an arbitrary number of devices simultaneously, but it only is compatible with the first generation (V1) of Vorne devices. There are no plans to support V2 as V2 lacks public APIs we would require.

Vorne only has a limited number of V1 devices left and will stop taking orders for them in the near future.

When the initial OEE Suite license (10-pack) is purchased, it includes 2 hours of training. The intention is that a client designates internal staff to follow that training, so they can become familiar and will be able to train new staff (we ‘train the trainer’).

With V1 being sunset, Deb-Tech Systems will need to scale back and will be less able to offer support and training, with reduced hours.

For this reason, we have been working on documentation on the most common support issues, with a detailed description on how to handle them, to allow you to handle support on these issues internally.

In many cases this should suffice to handle the majority of the questions you encounter. We are positioning these support documents as our first level support, while Deb-Tech Systems provides second level support using the usual channels (email, phone).

Alternatively, you may opt to upgrade your Vorne devices to V2, but that would imply losing the benefits of OEE Suite.

A Vorne device (V1) collects production data and stores it in its own internal database. This holds data for a single line, the one the sensors are monitoring. It has a wired ethernet port, which allows it to be connected to a network. Assuming the Vorne device is on the network, it also offers a built-in web interface, accessible by going to the IP address of the device.

Using the network, data can be extracted from a device and stored in a central MS SQL database. This is done by the XL Bolt-on Data Collector from Sulzer Consulting. Every device gets assigned a unique ID, the device key, to allow all data to co-exist in a shared database. Sharing the database allows for visualization of multiple lines simultaneously. To allow this to work, TCP port 80 must be open.

The database can be the free MSSQL Express (limited to 10 GB of data, as well as some performance throttles), or one of the full-blown commercial versions, depending on your needs.

Once the data is in the shared database, OEEAlert and OEEStudio report over that data.

OEEAlert is browser based and after installation offers two access points: OEEAlert, for the visualization and OEESetup, focused on setup and configuration. OEEAlert also includes a Windows Service (OEEAlert Service V3 - VetiiGROUP), which handles the monitoring for alert conditions.

OEEStudio has two components, a Windows desktop application to do the analysis. And a scheduler, a Windows service that creates scheduled reports.

Note: Your installation may also include a component called XL Bolt-On Data Link, to work with PLCs. Support for this is handled by Sulzer Consulting (jim@sulzerconsulting.com).

Database structure

The most important tables are two streams of data, intervals (jobs, shifts) and production events (run, down, setup, standby). The machine hosting the database and/or the data collector (see below) must be permanently on, to always collect data.

Data for a specific device is uniquely identified by the Device Key, setup when configuring it in the Data Collector.

This unique ID is not necessarily equal to the Asset ID of the device, especially when multiple plant locations share a database, as Asset IDs can repeat across locations, but the device keys must be unique.

Records of the stream data (intervals, events) are each assigned unique numbers by Vorne (event number, job number, shift number). These numbers become more relevant when replacing devices, as we need to make sure there are no number collisions (overlaps).

Vorne APIs

In addition to extracting data, the V1 devices support two APIs, one serial (for scanners): Type 1, and one TCP/IP based, Type 2.

The Type 2 protocol is used by OEEAlert for a variety of things: setting or updating the event reasons, adjusting counts, etc. By default, this uses TCP port 4001.

Roles

OEEAlert (and OEESetup) follow the same roles as the Vorne devices: Operator, Supervisor and Administrator. Make sure there is at least one administrator and you know the credentials, as that role has diagnostic capabilities. Out of the box there is an account (a user) called ‘administrator’ with password ‘aragorn’ (same as the default Vorne password for administrator). If you logon to Windows with the local (Windows) user account ‘administrator’, it automatically assumes the OEEAlert administrator role.

OEEAlert: many features can be configured through the browser interface, but for more low-level settings, there is an XML based configuration file, ProductConfiguration.xml, on the app server, in location:

• C:\Users\Public\Documents\DebTechSystems

This is where the database connection is configured. Note: The file must contain valid XML for the application to work, typos can break OEEAlert for everyone. Best to make a backup before editing it.

Verify that the credentials for the database server (and the machine running the Data Collector, if different) are documented and available.

If you changed the default Administrator passwords for the Vorne devices, verify you have the passwords available.

Be advised that there is a wealth of help information available in your OEEAlert > Info > Help and OEESetup > Info > Help. For OEEStudio, OEE Studio YouTube Channel offers many help videos, and OEE Studio Toolbox offers many help tips and tools.

Some issues require access to the database using the (SSMS) SQL Server Management Studio, to run queries and or updates. Make sure you have the appropriate access or access to someone who does.

No data is being collected.

All data comes from the Vorne XLv1 boards. Start with a board:

1. Verify the Vorne board is visible on the network. If you know the IP address of the Vorne device, enter it in a browser. This should take you to the Vorne native web interface. The IP address is configured in the Data Collector. If you don't see it in your browser, then there is an issue with the board and~or network.

Alternatively, if you don’t know the IP address, go to either OEEAlert > Info > About, or OEESetup > Info > About. In the line below the Trademarks heading is a link called [devices]. Clicking this link will show device details, including IP addresses. On OEESetup, the IP address is a link, which, when clicked will test port 80 and port 4001 access. This will let you know if you have internal network or firewall issues in your company.

2. If the Vorne board is accessible and no data is collected, the Data Collector service can be down.

3. If you don’t have access to the server where the Data Collector is installed, but have the Administrator role in OEEAlert, go to OEESetup > Info > Diagnostics and verify if the Data Collector is running. If not, click the Start button. You can do this for any of the services that are stopped.

4. If you have access to the server where the Data Collector is installed, login and run services.msc and check the XL BOLT-ON Data Collector service. Start, if necessary.

5. If none of this applies and only one or some devices are not collected, it may be that the Data Collector has put a device in quarantine due to network issues or you swapped a device or replaced the Mother board. To verify, login to the server where the Data Collector is installed and look in C:\Users\Public\Documents\SCLLC\XL BOLT-ON Data Collector\Log, DataCollector.log. You will likely need to contact Sulzer Consulting (jim@sulzerconsulting.com)

6. Another possibility is that the server has run out of disk space. When this happens, Data Collector configuration files may have been converted to empty (0 size) files.
Verify directory C:\Users\Public\Documents\SCLLC\XL BOLT-ON Data Collector\Config on the server, files: configuratorSettings.xml, dataCollector.xml, dataCollectorSettings.xml. If any of these have 0 size, restore them from a backup.
To free up space, you can remove log files from the following locations:

• C:\Users\Public\Documents\SCLLC\XL BOLT-ON Data Collector\Log
• C:\Users\Public\Documents\DebTechSystems\OEEAlert\Logs
• C:\Users\Public\Documents\DebTechSystems\OEEAlertService\Logs
• C:\Users\Public\Documents\DebTechSystems\OEESetup\Logs
• C:\inetpub\logs\LogFiles\W3SVC1

If the server is a virtual machine, you may be able to request IT to allocate more diskspace.

Or, in the Data Collector Configurator: Tools > Examine Data Collector log files -> Examine Current Data Collector – Service Log file.

7. Your last option is to contact Sulzer Consulting (jim@sulzerconsulting.com).

Cannot update reason codes (possibly sporadically) via OEEAlert

This can happen when the app server is blocked on port 4001. OEEAlert uses this port to check with the Vorne board whether the event is the most recent event of a particular state (down, setup, etc.).

If blocked, it may prevent the reason code change to take effect. If it happens sporadically, your network may use a load balancer, like F5 where one of the routes has port 4001 blocked. See the previous section on how to use OEESetup to verify whether port 4001 is blocked (Info > About -> [devices], click on IP address)

If a port is blocked, it will need a change in (one or more) firewalls. You typically will need support from IT or your networking group.

Device Network issues

Sometimes the Vorne network access stops responding on one of the ports. If you still have access to port 80 (HTTP), logon to the Vorne native interface as Administrator (default password: aragorn) and reboot it through software:

• Select Administer > System Health > Recovery Functions -> Reboot device.

If this method, is no longer available, power cycle the Vorne device. Contact Vorne Support

Emails (Alerts) are not being sent

The emailing feature uses the SMTP settings configured in [OEESetup] > Settings > Email Server
Verify with your IT department the required settings.

Alert emails (and SMS messages, if the email address maps to SMS) are sent by the OEEAlert service. After updating settings, restart the OEEAlert service: OEEAlert Service V3 - VetiiGROUP.
This can be done at the server, using the Windows service manager (services.msc), or in OEESetup, logged in as administrator, using Info > Diagnostics

If no emails go out, take a look at the OEEAlert Services logfile , on the server: C:\Users\Public\Documents\DebTechSystems\OEEAlertService\Logs\OEEAlertService.log.

If IT has configured that emailing requires two-factor authentication, this should be turned off, as OEEAlert does not support that.

If protocol errors are reported, verify that you have current TLS versions enabled on the server. TLS 1.2 is often required. Ask IT to verify it is present.
On the server, in the ProductConfiguration.xml file (C:\Users\Public\Documents\DebTechSystems), you can add a line in the AppSettings group to tell OEEAlert to use that version:

• <add key="TLSVersion" value="1.2"/>

There are several potential causes for differences in data between a Vorne device and our applications.

• Data is modified on the Vorne board or in OEEAlert and not propagated to the other.
• The date range criteria used on the Vorne do not match the criteria used in the application.
• An alternate OEE Mode is selected in OEEAlert.

Data Modifications

• You can update reasons in the Vorne board and those changes will not be copied to the database for OEEAlert and OEEStudio. You will need to make the same changes in OEEAlert.
• OEEAlert allows you to change any reason code, not just the last event for a state. When that is done, the data in the Vorne board does not change, therefore the Pareto chart and other reports will be different between the Vorne and OEEAlert (and OEEStudio).
• OEEAlert also allows some users the change the state of an event, i.e., from Down to Stand-by. When that happens, the availability gets recalculated and with that, the OEE.
  If the event being modified spans two or more intervals, the calculations prorate the effect, based on how much the event overlaps an interval.

Mismatched date range criteria

Only compare data when you are certain you compare the same data.

• Make sure you compare shifts with shifts, jobs with jobs, not jobs with shifts
• OEEAlert allows for a more flexible start and end date than the Vorne, at the 30-minute marks. When comparing data, make sure you select a range compatible with the Vorne selection (comparing apples with apples).
• When comparing reports in OEEAlert and OEEStudio, and the start and end date and time do not align with shift (or job) start and end times, there are three ways to select what data to include:
  • Include all intervals overlapping with the selected range. This is common in OEEAlert.
  • Include all intervals included within the selected range. This is typical for OEEStudio.
  • Pro-rate interval data to account for the partial intervals in the selected range. OEEAlert uses this in the KPI page.

Because different applications (and the Vorne board) may have different ways of handling the criteria, this may result in reporting differences.

Alternate Modes for OEE calculation

There are two ways to arrive at OEE:

• Compare actual good pieces produced with good pieces that are produced when everything runs perfectly (Good pieces *Ideal Cycle Time / Planned production time).
• Calculate availability, performance and quality separately and multiply to get OEE.

Vorne only counts pieces produced as good when the line is in a run state when the part is counted.
It also allows you to specify how many pieces a sensor must see before switching back to the run state. Depending on this setting and on how often the line goes down and restarts, you may count fewer good pieces than you actually produce. Counting only pieces when running gets the correct performance, but ultimately may understate OEE.

How OEE is calculated in OEEAlert is controlled in the Preferences page, in the OEE Mode setting. Options are:

• Vorne Compatible. This follows the way Vorne calculates OEE, relying on the run count. This is the best option if you want the values to match between OEEAlert and Vorne.
• Total count based is the method used in earlier versions of OEEAlert. It uses all counts when calculating performance, even those falling outside of run events.
• Run count based. This follows the Vorne method of calculating performance (based on run count), but also calculates how the good pieces occurring outside of run events contribute to overall OEE. Because Quality is strictly good / total, the contribution that counts outside of run events contribute to OEE is applied to the availability (as a multiplier).

OEEAlert uses the three roles of Operator, Supervisor and Administrator, same as Vorne

Every user can be in one (or more) of these roles. OEESetup > Security > Feature Roles defines which features are available to each role.

OEEAlert is browser based and relies on Windows Authentication to know which user is currently logged in. Some browsers will prompt you for credentials when loading OEEAlert, but IE and Edge will know your windows login name.

Everyone can access OEEAlert, even without being in one of the roles, but offered features will be restricted.

Granting roles depends on how the system is setup.

Security (user access) can be configured as one of the following:

• Local
• Active Directory
• Hybrid

This is configured in the application configuration file on the application server, in C:\Users\Public\Documents\DebTechSystems\ProductConfiguration.xml (UserSource).
Hybrid is the most common config setting.

Local

When setup as local, both users and their roles are configured independent of outside systems, with information stored in the database.

If in Role Administrator, use OEESetup > Security > Manage Users to add a user.

For the user name, choose the Windows login name for the user you are adding.

Password/Pin is an initial (one time) pin. When logging in with it, you will be prompted to change it.

Normally a user will not run into the prompt to change it, as we know your Windows login (user name) and as long as the user name matches the Windows login name, the software automatically assigns the configured roles.

The password is not related to any Windows password. It would be used when setting up accounts that do not have a matching Windows account (i.e.: Supervisor). You can use that account by clicking on your login name on the top right of the screen and do a ‘login override’ using the local user account.

Active Directory

When using this setting, users are managed outside of OEEAlert, in the corporate Active Directory.

IT will need to setup Active Directory groups corresponding to the three roles. They will manage who is in which role. The OEEAlert responsibility is limited to mapping those LDAP group names to OEEAlert roles: OEESetup > Security > Map Groups.

Hybrid

Hybrid leverages the Windows login name and maps each name to roles, without actually creating a local user. No password is captured, you don’t have the ability to disable the account (you can remove roles from a user or delete a user). OEESetup > Security > User Roles.

More information is available in OEESetup > Info > Help -> Security.

In summary, to add a user:

• If Active Directory, call IT or the business person responsible
• If Hybrid or Local, someone with the Administrator role either creates the new user as a local user or defines roles for the Windows login for that user.

OEEAlert assigns licenses on a first come, first serve basis. What that means is that when the app is started, it makes all licenses available. When a visualization is requested, i.e.: Production State, all lines which are part of the requested view are assigned a license. If there are more lines in the data than purchased licenses, the extra lines are not included in the visualization and listed at the bottom as ‘Unlicensed’.

Licenses are tracked in memory only. If the application is restarted, it starts over with a blank slate.

You can also reset the license tracking (in administrator role) using

• OEEAlert/Info/About?reset=true

If, by design, there are more lines than licenses, you can define a group called LicenseGroup, or LicenseGroupOEEAlert. Add to it all lines that should be marked as licensed. These will be assigned before any visualization runs and thus be guaranteed to receive a license.
See OEEAlert > Info > Help -> Setup > Asset Groups.

Data from old lines may still be in the database, causing them to pick-up a license.

Note: OEEStudio reverts to Demo mode if it sees more devices in the database than there are licenses. If this happens, first try to reload the OEEStudio license file:

• OEEStudio application: Connection and License, Upload license file.

If this does not correct the situation, verify there are no retired lines still in the database (unique devices in the dbo.device_property table (SQL: select distinct deviceKey from dbo.device_property).

When a new Vorne device is installed, it will require an OEESuite license to work with OEEAlert or OEEStudio. Unless you have unused licenses, you can request a quote from Deb-Tech Systems (Debbie.olk@debtechsystems.com). Once you receive the quote and Deb-Tech Systems receives a Purchase Order, new license files will be issued. There are separate license files for the XL Bolt-On Data Collector, OEEAlert and OEEStudio.

• The Data Collector license file, license.xml, should be copied to:
  C:\Users\Public\Documents\SCLLC\XL BOLT-ON Data Collector\Config
• The OEEAlert license file, OeeSuiteLicense.xml, should be copied to:
  C:\Users\Public\Documents\DebTechSystems
• The OEEStudio license, oeestudio.license, is uploaded through the OEEStudio application: Connection and License, Upload license file.

Once the licenses are installed, the XL Bolt-On Data Collector must be updated with the specifics of the new device.

• On server, Open XL Bolt-On Data Collector Configurator
• In left panel, right mouse click and select ‘new’
• In the Settings tab, enter the preferred name, an optional description and the device Key (required, typically same as name).
• Enter the IP address of the board, check ‘Enabled’ and click ‘Test Connection’. If successful, the Firmware field will be populated. If firmware is a lower number than 1.4.7.2, work with Vorne Support (Vorne Industries) to get the current firmware.
• Open the ‘Collection’ tab. We assume you, as existing client, have an existing collection schedule. Click ‘Collection Schedules’. This opens up a new dialog box.
• In left panel, double click the name of the schedule to use.
• Under ‘Devices to Synchronize’ select the name of the device just created, click ‘>>’, select OK and ‘Close’
• Next select the ‘Properties’ tab. Enter the values appropriate to this device. These values control how the device shows up in OEEStudio and OEEAlert (where properties are referred to as ‘categories’)
• Click ‘OK’
• On main menu, select File > Save
• On main menu, using File > Tools > XL-Bolt-On Data Collector Service, stop and start the service (if not already prompted to do so).

Once saved and the service has restarted, the device data should be collected. Depending on how long the line has already been monitored, it may need some minutes to catch up to the current time.

OEEAlert caches license and category (property) information. You may have to do OEEAlert > Info > About to refresh the cache and make the new device visible in OEEAlert. This does not affect OEEStudio

If you shutdown a line or stop monitoring a line, take the following steps.

• Open the XL Bolt-On Data Collector configurator
• In the left panel, select the device you want to remove
• Right mouse click and select ‘Delete’ and confirm when prompted
• Select File > Save
• Say yes when prompted to restart the Data Collector service.

Deleting the device will free up a Data Collector device license, it does not remove data already collected from the database. Because of this, the device still takes up a OEEAlert and OEEStudio license.

If you don’t care about historic data, you can remove it from the database, see next item.

This requires ‘behind the scene’ action and probably access to a DBA.

To remove all data for a specific line, a Stored Procedure can be run (in SQL Server Management Studio) as follows:

• oee.EradicateAsset ‘yourDeviceKey’

This permanently deletes data, so be sure to verify the deviceKey is correct.

If the Stored Procedure is not present, you likely have an older version of OEEAlert, v3.5.23 (December 13, 2018) or older.
If so, contact us to upgrade you.

A line has two names: The Asset ID, by which the device knows itself.

And the DeviceKey, by which the line is known in the database.

The views in OEEAlert display data by deviceKey, as assetIDs are not guaranteed to be unique.

The deviceKeys typically follow a standard scheme, but sometimes the standards change and the deviceKey needs to change.

The deviceKey is an important database key, changing it requires touching a lot of tables.
Again, this requires a DBA or someone with proper rights to the database.

There is a Stored Procedure to help with this: oee.RenameAsset @oldName, @newName:

• oee.RenameAsset ‘oldDeviceKey’, ‘newDeviceKey’

Sometimes, people want to switch names. This is tricky.

If you have two lines, called Line1 and Line2, you cannot first rename Line1 to Line2 as that would conflict with data for Line2, likely causing a bunch of database conflicts.

To accomplish this, you need to ‘park’ Line1 data, i.e.: under a temp name. Example:

• oee.RenameAsset ‘Line1’, ‘Line1Temp’
• oee.RenameAsset ‘Line2’, ‘Line1’
• oee.RenameAsset ‘Line1Temp’, ‘Line2’

Best to avoid this altogether, if possible. Also, first make a backup of the database, to safeguard.

Sometimes a Vorne device breaks down and needs to be replaced. Both the interval stream and the event stream data have a set of numbers (keys) already stored in the database. The new device is not aware of these numbers. It likely will start with fresh numbers, causing duplicates and therefore reporting errors.

Note: The Vorne database also uses so-called Sequence Numbers, in addition to event, job and shift numbers, for historical reasons.

When replacing a device (or motherboard), use the XL Bolt-On device replacement feature:

• [Data Collector Configurator] Tools > Manage Sequence Numbers

This brings up a big dialog listing the current numbers in both the device and the database and recommends new starting numbers for the device.

For further support with this feature, contact Sulzer Consulting (jim@sulzerconsulting.com)

This is the situation where you take one device (out of storage, or from a different line) and install it on a new line.

It is important here to clear any old data out of the device before starting to use it again:

• Logon to Vorne device as administrator, password ‘aragorn’ (default)
• Select Administer > System Health > Recovery Functions -> Erase Records > Erase All Streams.
• Press Erase Records

A V1 Vorne device has the option to define programs which can change the display, set reasons, start jobs as well as a host of other things.

This is fully within realm of Vorne support. Please contact Vorne Industries for this (Vorne.com).

Typical scenarios include:

• A plant starts at a local desktop PC, but wants to move it to a dedicated server PC
• A move from using MSSQL Express to a full-blown server.
• An installation needs to move from one combined app server & DB server to separate servers
• IT want to move the server from a local, physical server to a cloud-based server.

In all cases, during any interruption, the Vorne devices will still be monitoring the lines and collecting production data. Once the move is complete and the Data Collector is restarted, the collected data will ‘catch up’ pulling in data collected when the move was in progress.

Typical steps:

• Stop the XL Bolt-On Data Collector
• Make a backup of the MSSQL database
• If moving to a fresh PC or server, install a MSSQL instance and Management Studio, if not yet present
• Restore the database.
• On the app server, use the opportunity to bring the XL Bolt-On Data Collector, OEEAlert and OEEStudio up to the current version. If OEEAlert still uses v2.x or earlier, contact Deb-Tech Systems (Debbie.olk@debtechsystems.com) for help.
  Use the respective MSI (OEEAlert/OEEMosaicInstall) and EXEs (Data Collector, OEEStudio)
• Update database connection, if changed (all three components)
  
  • XL Bolt-On Data Collector configurator: Options > Storage Locations, Select item to update, update server name and/or database name
  • OEEAlert, in C:\Users\Public\Documents\DebTechSystems\ProductConfiguration.xml, update <connectionStrings>, name="DSN", connectionString field to reflect new server name and/or database name
  • OEEStudio, in Connection and License, Database Settings, Edit Connection Settings: update server name and/or database name
• Start XL Bolt-On Data Collector service
• Verify that OEEAlert service and OEEStudio Scheduler service are running.

Moving servers to the cloud has special requirements, typically handled by IT.

Details

. . .