Connection to a dataspot. data catalog

The dataspot. connection links the DQC Platform with your data catalog in both directions. You connect tables to data objects and rules to data quality indicators. After every ruleset check, the DQC Platform records a measurement on each linked indicator, so data stewards see current quality figures in the place where they govern the data. The connected objects in dataspot. link back to the table, the data source and the rule.


How the objects map

dataspot.

DQC Platform

What the connection does

Data object

Table

You connect a table with the URL of its data object. The data object receives a link to the table

System

Data source

The system the data object is deployed in receives a link to the data source of the table

Data quality indicator

Rule

You link a rule to an existing indicator or let the DQC Platform create one. The indicator receives a link to the rule

Measurement

Result of a rule in a ruleset check

Every finished check records one measurement per linked rule

The links are stored in dataspot. in the custom property dqc_url of each object.


Prerequisites

The DQC Platform signs in as an application through Microsoft Entra ID and acts in dataspot. as a technical user. Ask your dataspot. and Entra administrators for:

  • An app registration in Microsoft Entra ID for the DQC Platform, with a client secret or a certificate

  • The token endpoint of your Entra tenant, in the form https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token

  • The scope of the dataspot. application in Entra ID, in the form <application-id>/.default

  • A technical user in dataspot. with an access key

  • The custom property dqc_url in dataspot. for data objects, systems and data quality indicators. The DQC Platform stores its links there

  • A collection in dataspot. in which the DQC Platform creates indicators

  • The import link in dataspot., if data stewards create rules from indicators. See "Create a rule from an indicator" below

If access to your dataspot. instance is restricted by IP address, allow the static IP of the DQC Platform: 3.123.94.228


Add the connection

  1. Go to Connectors → Catalogs and click Add data catalog

  2. Choose Dataspot as the type

  3. Fill in the fields and save

Field

Value

Name

A name for the connection

Default indicator collection

The URL of the collection in which indicators are created, copied from dataspot. in the form https://<server>/web/<database>/collections/<id>. The server, the database and the collection ID are read from it and shown in the three read-only fields below

dataspot. Auth URL

The token endpoint of your Entra tenant. It has to contain the tenant ID

Client ID

The application (client) ID of the app registration

dataspot. scope

The scope of the dataspot. application, ending in /.default

Authentication Method

Client Secret, or Client Certificate with the PEM-encoded private key and the SHA-1 thumbprint of the certificate

dataspot. access key

The access key of the technical user. It can stay empty if the application was granted an app role for dataspot. in Entra ID

A tenant has one dataspot. connection. Adding and editing catalog connections is reserved for owners.


Connect a table to a data object

  1. Open the table and choose Add data catalog in the menu of the table

  2. Paste the URL of the data object from dataspot. into Object URL, in the form https://<server>/web/<database>/classifiers/<id>

  3. Click Save

The table is now connected. The menu entry changes to View data catalog and shows the connected object with a link to dataspot. To disconnect the table, open View data catalog and click Delete.

  • The data object receives the link to the table, and the system it is deployed in receives the link to the data source

  • Connecting a table needs the Edit permission on the table


Link a rule to a data quality indicator

Open the rule and choose Data catalog in the menu of the rule editor. The entry is available once the tenant has a dataspot. connection. There are two ways to link the rule:

  • Link an existing indicator: Paste the URL of the indicator into Object URL, in the form https://<server>/web/<database>/indicators/<id>, and click Save

  • Let the DQC Platform create the indicator: Click Create in dataspot. The indicator is created in the default indicator collection with the name and the description of the rule. If the table is connected to a data object, the indicator is assigned to it: to the attribute named like the column when the rule checks exactly one column, otherwise to the data object itself

What linking does:

  • The indicator receives the link to the rule, and its measurement type is set to amount, because a rule reports a number of rows out of a total

  • A rule is linked to one indicator. To link another one, click Unlink first. Unlinking also removes the link in dataspot.

  • Deleting the rule removes its link from the indicator. The indicator and its measurements stay in dataspot.

  • Linking a rule needs the Coordinate permission on the ruleset

After linking, the DQC Platform checks whether results can be recorded. If not, the message "Results will not reach the catalog" names the reason.


Measurements

After every finished ruleset check, the DQC Platform records one measurement for each linked rule:

Measurement in dataspot.

Value

Amount

The number of rows without an issue: rows checked minus issues

Total

The number of rows checked

Computed on

The system the measurement belongs to, see below

Date

The day of the check

Comment

The name of the rule with a link to it, the number of issues and the failure rate

  • dataspot. counts the rows that are fine, the DQC Platform counts the issues. A check of 15 rows with 3 issues is recorded as 12 of 15, with the comment "Issues 3 (failure rate 20%)"

  • An indicator holds one measurement per system and day. A later check on the same day replaces the measurement of that day

  • A check that cannot report its results still finishes normally

The system of a measurement

dataspot. requires every measurement to name the system it was computed on. The DQC Platform looks for it in this order:

  1. The system in which the attribute or data object assigned to the indicator is deployed

  2. A deployment of the indicator itself

  3. The system of the data object the table is connected to

If none of them names a system, no measurement is recorded. The Notification Center then shows "Results not reported to the catalog" with the rules concerned. To fix it, assign the indicator in dataspot. to an attribute or data object that is deployed in a system.


Create a rule from an indicator

A data steward can start in dataspot. and turn an indicator into a rule through a link on the indicator. Your dataspot. administrator stores this link in dataspot., in this form, with the ID of the indicator at the end:

https://<your DQC Platform address>/apiV2/integrations/dataspot/?indicator_id=<indicator-id>

The DQC Platform reads the indicator and the objects it applies to, finds the connected table, builds a rule from the description of the indicator with AI and links the rule to the indicator. The page shows each step and opens the new rule at the end. If a step fails, it says what to change and where.

For the import to work:

  • The indicator is assigned to a data object, or to attributes of one, that is connected to exactly one table in the DQC Platform

  • The table has a ruleset

  • You can view the table and hold the Coordinate permission on its ruleset

  • An AI model is available in the tenant

  • The description of the indicator states the condition to check and the fields concerned

An indicator that already has a rule is not imported a second time. The name and the description of the new rule are written in the tenant language.


💡 Notes