QESTNET Internal:Developer QESTNET Creating A New Workflow

This page outlines the steps required to create a new workflow in QESTNET (QN) and QESTField (QF).

Introduction

Workflows are similar to QESTLab (QL) Test Screens, except they can be made up of more than one Document. Workflows encapsulate data entry for a Work Order, an optional Sample, and any number of Documents. A typical Workflow still consists of a Work Order and one parent Sample with one Document on it. When designing a Workflow you must consider how this structure should be set up. A lot of the Workflows have the sample details and test details on one big page. Sometimes it makes more sense to break it up into multiple pages. Sometimes these decisions are customer-driven.

In QN every database table is mapped to an "Entity", which is a Microsoft term used by their Entity Framework technology. As a result; Work Orders, Samples, Documents, etc are really just types of Entities. So Workflows actually just encapsulate data entry for any number of Entities.

Each Workflow is created and edited via buttons in QF. These buttons typically appear on the Work Order Workflow.

[screenshot]

Workflows are broken up into lots of small pieces, rather than being all in one control such as in QL.

If you review the QESTNET Projects Quick Reference Guide you will see there are 2 types of Entities, for the QESTLab.Data project and the QESTLab.Entities project. When coding Workflows you only concern yourself with the latter type, which is mapped back to the data level automatically.

Work Order Workflows can be created on QF's search screen or in QL. After creating a new Work Order or opening an existing one, other Workflows can be created or edited based on the following criteria:

• The Workflow has been licensed via a script for the customer, at the laboratory for the Work Order you are viewing. (Details on how to set this up will be covered when creating the WorkflowFactory later.)
• Each Workflow has a loading plan that lists the required Entities for it to be valid. These Entities must be present under the Work Order somewhere.

Note that this last point means creating a new Workflow with a number of Entities may make multiple other Workflows appear for editing, if they use the same Entities. This allows us to provide multiple ways of editing the same set of data, tailored to specific customer workflows.

The remainder of this guide will cover how to create the required functionality for a Workflow at each level of QN/QF. Almost all of the QN work is done in the QESTNET.Sessions.Lab solution, when searching for where to make changes this is the place to look. You can refer to the following article for an overview of the projects within this solution: QESTNET Projects Quick Reference Guide.

Initial Configuration

Make the usual QL items:

• Review the specification or standards to understand what needs implementing
• Create all required qestlab.qes objects (tests, lists, etc.)
  • Here you need to decide if the test should have the "StoreSIBaseInDatabase" property (or similar name). It is only valid for tests that will not be used in QL. More on units later.
• Make initial database tables, even if they just have the standard fields

Use the qestnet.upgrade tool:

• Make sure you have the appropriate qestnet.upgrade branch checked out for making changes on
• Generate the table and qestObject scripts
• Selectively commit the lines that are relevant to your changes and no others, for both tables and qestObjects
• Most tables will be automatically activated for use in QN/QF. Edit the qestnet.upgrade scripts to make any required manual changes either way. Usually there isn't much.
• Upgrade your development database of choice with these changes

Prepare the QESTNET Schema Generator Tool:

• This can be found under "QESTNET/QESTNET.Sessions.Lab/QESTLab.SchemaGenerator/QESTLab.SchemaGenerator.sln"
• Your development database must have some scripts installed for this tool to work, they are found in the "scripts" folder which is next to the solution file above. Run these scripts on your database once, if they haven't already been ran.
• Add any new test tables to the tool:
  • Open the solution
  • Open the "schema > ConfigHelper.cs" file
  • Add your test to the GetDocumentNames() function
  • You only need list tables in here if you intend to use them with Entities, which we do not 99% of the time, so I do not advise adding list tables.
• Save and compile the Schema Generator Tool

QESTNET Changes

QESTLab.Data Schema Update

Update the QESTLab.Data schema to match the QL database structure:

• Run GenerateDataSchema.ps1
• Build QESTLab.Data project
  • If you receive errors similar to 'Source file "Entities\UserDocument100.cs" could not be found' at this point, right click EntityGenerator.tt, select Run Custom Tool, Do the same for EntityGenerator.Context, then rebuild QESTLab.Data.
• Right click QESTLab_Data_Views.tt and select Run Custom Tool
• Build QESTLab.Data project

Note that we only include tables we need to in QN. This provides some improved performance and reduces code clutter a bit.

QESTLab.Entities

Create your Entities:

• Create files in the appropriate folder, following how the others are done.
• The end of the name should be the type of Entity, Test, Inspection, List, Sample, etc.
• Entities should not have any real logic in them, except simple properties that are derived from other fields. For example if you need a value, and an inverse of that same value, there is no point saving both. A small property with logic to calculate it can be used instead.
• The easiest way to add fields is to open the corresponding Entity in the QESTLab.Data project and copy + paste from there.
• You only need to add fields that are not part of the standard set, as these are covered by the inheritance structure.

Entities represent idealised database tables which means:

• Field names can be different to what's actually in the database
• You can have additional fields that don't exist in the database, but are only used in code.

Set the valid QestIDs:

• At the top of each Entity is a list of the valid QestIDs for that Entity. Add / edit yours.
  • This is stored in what's known as a C# Attribute, this is fancy programming that allows us to write code about our code. C# supports what's known as reflection, a technique to inspect your code with other code. So we can write some behaviour that says "if the user is using this entity, check that it has this QestID". Beyond the scope of what we're doing a bit, but you can Google around for more information.

Set the relations or endpoints between Entities:

• Relation and Endpoint are mostly the same thing, but terminology will be used interchangably.
• Each entity is linked to others. The historical example is Work Orders have child Samples, that have child Tests.
• Unlike QL's tree structure, QN allows a graph structure - every Entity can link to an arbitrary number of others.
  • We mostly stick to trees for simplicity
  • For QESTLab compatibility you must stick to trees, and to the "Work Order - Sample - Document" structure.
• Follow the style of the existing Entities to create your required relationships.
• The [QestParent] attribute specifies which relationship points to a parent. So for example the Sample will have [QestParent] on the Work Order relation. Set parents for QL compatibility and for setting the database fields QestParentID, QestUniqueParentID.
• An Entity can either be linked to 1 or many other Entities of a similar kind, for example Work Orders have multiple Samples. You need to specify the relationship type here.
• Relations are manually loaded to avoid loading too much into memory. This will be discussed more later.
• You get some extra relations for free because of the inheritance on Entities.

Replace all field types with Units of Measure (UOM) unit types:

• UOM allows us to use things like "Mass" rather than "double" for a field type
• This allows us to optionally save the data in SI units, as well as display it in various forms like kilograms, grams, etc.
• Replace where relevant "double?"s with Mass?, Length?, Pressure?, Force?, etc.
  • Note the "?" means it is allowed to be null, most database fields are nullable so this should be included.

QESTLab.Repository: Update for the new Entities

At the time of writing, a lot of code is not yet automated to allow Entities to run seamlessly. Follow these steps to make the Entities you have created functional.

This project will expose you to the Entity Context, this is similar to the QLO in QL. All tests in the database can be accessed and loaded here, along with a number of other things.

In the respository project...

Update the Entity Mappings:

• This is used to map your QESTLab.Entities Entity to the QESTLab.Data Entity.
• In the Entity Mappings folder is 3 files
  • Also update the QestEntityContext.cs file, which may be up a folder.
• Copy the existing style to add your entities in here
• Any Entity that uses UOM needs special handling to include a UOM mapper, you will notice properties that have this on the end: ".Include(units[typeof(Torvane).Name])".
• Look at similar Entities to figure out how to do this
• You will need special handling here if you have:
  • Created Entity field names that are different from the database field names
  • Created additional properties or methods on the Entity
  • An example of this is the CaliforniaBearingRatio Entity's mapping, it has a MaximumDryDensity property but this isn't in the database, so this line is in ToEntityConfigurationSource.cs as: .MapProperty(o => o.Ignore(), d => d.MaximumDryDensity)

Update the Entity Queries:

• In the Entity Queries folder there is 1 file
• This file helps load relationships between Entities
• Follow how the other code is done here, make similar code for your Entities. Keep in mind the relationship between two Entities can either be 1 or Many. The functions here need to change depending on which it is.

Update the Unit Mappings:

• In the "Unit Mappings" folder.
• By this point you should know if you're going to be saving all values in SI Base, or if it needs to work in QL as well.
• Under the "Entities" sub-folder, a class needs to be created for each Entity you have with UOM units on it:
  • These provide default configuration for the units.
  • Copy the way others are done.
  • The main thing to remember is for compatibility with QL screens, we provide a set of defaults for the units for each field in the database. For example, traditionally MDD tests have saved masses in grams so we set grams as the default for each of the mass fields. In contrast to this an SI base test would save these values in kilograms. These defaults are set by logic so you can have different defaults for IP units, different QestIDs, etc.
  • Implement applicable methods: GetSIBase(), GetMetric(), GetIP(), etc. Each defines what unit is used for storing each field in the database. The dictionary that is chosen for a test depends on what units that test stores in its database fields.
  • Modify GetUnits appropriately
• Add call in UnitMapper.SetupUnits
• Add code to UnitsToDataConfiguration and UnitsToEntityConfiguration
• The following commit is a good example: https://github.com/spectraqest/qestnet/commit/f78e8112969ada3c21bbb7c700155b28e0a4719f

We hope to automate a lot of this work eventually.

QESTLab.Entities.Rules

This step can optionally be done last if it's easier to get a "dumb" workflow up and running just to test everything. It is fairly isolated from the rest of the process.

Rules are similar to QL BusinessRules.

Each Entity has a Master Rule:

• This rule invokes all the individual calculations as necessary
• This rule should have the MasterRule attribute applied to the class's definition, eg:
  • [MasterRule(typeof(DailyFieldInspection))]
  • public sealed class RuleDailyFieldInspection : CodeActivity
• There are some standard things in every rule (or most rules):
  • Run RuleSetPerformedBy - Required if you need to work with TestStages or if you need to set the completion status, this applies to most tests. Each test has its own RuleSetPerformedBy that you have to make.
  • Check loaded endpoint relations - Rules should only run if all the Entities and their relationships have been loaded, otherwise you'll get null pointer exceptions.
  • Propagate delete actions - Because our Entity relationships make a graph not a tree structure, it is difficult to know what should be deleted when a single test is deleted. The rules for a single Entity should propagate a delete to all related Entities where it makes sense.

Rules need to run fast which means:

• They should be written wisely, they could be run multiple times each calculate
• Use Change Tracking to check what the user has changed, to see if rules need to be run at all

Rules have 2 input variables:

• The context, or QestEntityContext. This is similar to the QLO and gives you access to things like loading lists, running procedures on the database, loading additional Entities, Documents, database data, etc.
• The Entity that the rule applies to. You can still read/write to related Entities however, but be careful to do so only when it makes sense. It is usually better to only affect the Entity the rule is for.

Create your entity rules:

• Break up the the calculations in a logical way
• Use one class for each calculation
• Namespace the rules appropriately
• Look at others for examples

QESTLab.Workflow: Workflow Factory

Create a new Workflow Factory:

• In the Workflow Factories folder
• Find a good spot, copy an existing example. Some are Fugro-specific these are not usually useful for other customers.
• You can create a separate factory for each Method you want to create, if they all use the same basic Workflow and Entities. See the SteelReinforcingWorkflowFactory for an example. Each Method has its own Guid so the methods can be individually licensed.
• Generate a new GUID attribute for the WorkflowFactory - this is the ID used in licensing the Workflow
  • This can be done in Visual Studio by going to "Tools > Create Guid"

Create an Entity Plan:

• These are done in the constructor of the WorkflowFactory
• These names may change so inspect the constructor of the SteelReinforcingWorkflowFactory for an example.
• These are used to specify which Entities need to exist on a Work Order for the Workflow to be valid
• They can optionally load the relations/endpoints on the Entities automatically while doing this test. This means you don't have to do it manually later.
• Use either a LoadableEntityPlan or EntityPlan for this.
• See the AsConcreteWorkflowFactory for an example that does not automatically load, which manually loads relations in the CreateWorkflow() function.
• Note that some relations can't be loaded with a plan, any time the relation is a collection, e.g. specimens. This is because the Workflow will be considered invalid if there are no specimens, and as zero is usually valid, the Workflow will never appear.

Update the customer's license script to include the new Workflow Guid:

• Find the file in the "/Deployment" folder at the top level named QESTField.Enable_Workflows.XXXXX.sql
• Add a line with the Guid in it here

Set up Workflow Phases:

• Phases are used to break Workflows up into multiple pages. There is special coded dedicated to specifying which containers are used on which phases. The same containers can be used, but different sections activated for different steps, e.g. wet and dry weights on 2 separate pages. There is a lot of behaviour here and it can be learnt as required. Any time you want behaviour to change between pages, this is the place to look.
• The code is normally found at the bottom of the CreateWorkflow() function of the WorkflowFactory.
• It will look something like "NewProcess.Phase(workflow, "SPD").Targets(siteAndProductContainer)"...

Phases use Processes, which are a general way of creating logic and actions for Workflows. They can either be public or private, meaning that they are either triggered by events (such as a container updating), or they are triggered by a visible button on the UI. Public tends to mean the user can see it, private processes are hooked up to container events and run automatically. (This information is a bit rough but mostly right.)

QESTLab.Repository: Work Template / QESTLab.Workflow: Processes

In the QESTLab.Repository project, create a Work Template for your Workflow:

• This includes all the Entities you need to create when the user creates a new instance of this Workflow
• Copy existing examples to get the idea. You start with a Work Order and add Entities to it.
• In the "Work Templates" folder

In the QESTLab.Workflow project, set up processes to invoke the workflow:

• As said elsewhere, processes will be displayed as buttons on the UI to start the Workflow.
• The files of interest are:
  • "Core/ContainerManager.cs" - Which includes a list of process bundles, and these bundles have a bundle of processes. Each of these will become a button.
  • "/Process Source Bundles/*"
• You may need to make a new bundle if a relevant one does not exist
• Use existing examples to create processes for your Workflow, normally one add and one edit process.
• Some of the fancier code there will put the edit buttons under the Work Order even if the test normally sits underneath a Sample.
  • These buttons can be grouped together using certain tags (as of the time of writing, the tags that can be used for this purpose can be found within UpdateWorkflowLinksProcess), e.g. see in the FieldDensityProcessBundle how the tag "field-density" is used as part of the Description that ends up being used in the code to add the button. Field inspections could be similarly grouped using the tag "field-inspection".

QESTLab.Workflow: Container Factories / QESTLab.Models

Models represent groups of fields that belong together on a Workflow, similar to how a frame is used in QL screens to group fields together. For example it might make sense to have a model for sample details, one for equipment details, and a final model for test details. Unlike Entities they do not have to be one-per-entity, they can display fields from multiple Entities, or just from a single Entity. Whatever makes sense to be grouped together. For example, sometimes tests have additional fields that are really sample information on them, so that could be grouped with data from the sample itself.

Create Models:

• In the Models project
• The classes must be named ending in "Model"
• The [DataMember] attribute makes the field visible to QF. Add these to your fields unless you have a good reason not to.
• UOM unit fields must use the Reading classes. For example if you are mapping a Mass? from an Entity, you will need to use a MassReading on the Model. Notice that this is not "MassReading?" (nullable type), the MassReading is a class so it can already be null, but the Mass is a C# Struct, so it needs to be made nullable with the "?".

Containers are used to organise the Workflow into a tree structure, each container can have many sub-containers. These are sent to QF to be displayed in HTML. There are two types of Containers:

• Regular Containers - Which are just there to group other containers together, and can have special properties like Processes attached to them (which end up as buttons).
• Model Containers - Which are used to hold the Models that have Workflow data from the Entities. This data is displayed as forms and fields in QF, so the user can perform the test.

Container Factories are used to create these containers:

• A lot of Container Factories just run other Container Factories for creating a Container with sub-containers. These are commonly referred to as Workspaces because they essentially just group containers together without doing much on their own.
• Model Containers are produced by Container Factories that have two special functions in them:
  • ActionTransferTo - A class used to transfer data from any number of Entities to a Model
  • ActionTransferFrom - A class used to transfer data from a Model back to the Entities
• Factories can also add other things to containers such as:
  • Styles - which are just strings that are used by QF to change the way the container will be displayed. For example if for a specific QestID you wanted to hide 1 field, you might add a style like "hide-field-xxx".
  • Tags - which are the same as Styles but used for any purpose other than styling. For example if you want to select all specimens for a test, you might tag the specimen containers with the string "specimen".
  • Processes - As mentioned a few times, public Processes can be displayed as buttons in QF. Private Processes can be attached to containers to run certain code like Rules whenever a container is changed.

Create Container Factories:

• One per model, plus any additional grouping containers. It's good practice to call these grouping containers "workspaces", see the other similar files.
• Mapping UOM fields is a bit trickier as readings need to be created with units, precision, and mapped back appropriately. See how other factories do it.

QESTField

In order for the Models you've created in QN to correctly appear in QF you must update the Service Reference. This can be done by:

• Opening the Service References folder in Visual Studio
• Right-clicking on the existing Service Reference and clicking "Update"
• This must be done any time you add / remove fields on a Model or add / remove Models themselves.

In QF, the Workflows are displayed by Views:

• All views are in the Views folder
• There is one unique View for each Model Container. There doesn't have to be but it get's difficult to work with if not.
• For Containers that don't have Models, a standard View is used.
• If you want to customise the way Containers are displayed you can use tags and the WorkspaceController.cs file. If you create tags on a Workspace Container to call it something like "concrete-workspace", you can then add code into the Controller to use a special View for it, which you might call "ConcreteWorkspace.cshtml".
• All this means there is a simple standard way things will be done, but you have the flexibility to change things as needed.

If you create a new folder in the Views directory, you need to add it to the "Global.asax" file in the root of the QESTField project. This will be hopefully automated eventually.

Create Views:

• Name them the same name as the Model but without the word Model.
• Copy how the others are done. Change the name of the Model at the top.
• Create Razor-HTML for each field you need.