Migration Reporting and Action

REAL TIME: All files migrated successfully. The SharePoint metadata and URL issues. The failures.

Consolidated Reporting: All jobs to all destinations from all sources

The repository database gives you a comprehensive reporting solution that enables you to track and fix any issue incrementally and proactively without loosing the global overview. You can focus only on the core root issues and view meaningful processing feedback as the migration jobs progresses. So, there are no CSV or Excel files floating around.

Actionable: Feed the report output into incremental cherry-picking jobs.

Either the same job definition as used originally or modified to fix issues with MangedMetadata tags recognition, invalid users , content-types, unresolved lookups, invalid choices or bad URLs etc.

Cherry-picking for ReMigrate or RetryMigrate

based on any criteria (simple no-code UI) or wrong Content-Type, Manged Metadata tag or just revised rule set for what is right. You name it!

Incremental/sync in every possible way.

Based on any incapsulated and parameterized CAML filter (simple no-code UI) mixed with , date modified, file existence, file nonexistence etc.

Database views Use as source in a DataETL Step or For-Each Step
All files that have been processed. Also the simulated migrations executed in “Test-mode”.
All metadata validation warnings such as unknown users, Managed Metadata, Content-Type etc.
All SharePoint user processing including results for mapping and the unfiltered results from the
API call to SharePoint.Client.EnsureUser()
Key functionality Know what happened to each and every file and take action
  • Consolidated Reporting
 Data about all jobs to all destinations, from all sources. All data is automatically consolidated and always up-to-date in SQL database. No “dead” useless CSV or Excel files floating around.
  • Take action using the reporting data
 Feed the report output directly into incremental cherry-picking jobs. While constantly keeping the consolidated always up-to-date overview
  • Incremental reporting
 Report only the latest instance of any given file across all migration sessions (or any filter). So for example a file that failed yesterday but completed successfully today will be reported as completed
  • Embedded reporting
 Add reporting to any migration setup as an integrated Step
  • Context information
 Add searchable context specific information such as Customer Id, Account Number to each reporting output row based on variables derived from a For-Each loop
  • Output report records to:
 Another job, Excel, CSV, SharePoint List, SQLServer, ODBC , PowerBI

When handling many files it becomes a major task determining to what level the file migration or replication was successful and especially to track and correct the issues that are inevitable to happen.

In an ideal world or a demo setup the reporting and tracking task is easy and often marginalized. But the real world is very different and no one will accept that “maybe some files didn’t make it and, well metadata is wrong to some extent”

There are many pitfalls when migrating files to/between SharePoint Document Libraries
Microsoft 365: The file migration process is a multi phases procedure.
  1. Prepare the file metadata according to Document Library configuration.
  2. Transfer the binary file and prepared metadata to Azure BLOB Storage
  3. Request for an Azure job to combine the binary file with the metadata and create a Document Library Item.
    It is required to track each phases and in phases 3 NEVER do a “Fire and forget” to gain performance. “Fire and forget” is only feasible if an async monitoring agent can pickup the feedback given by Azure. Many errors happens in phases 3 and it is crucially important to pickup information about files that did not make it or had invalid metadata
Non existing users
Invalid mapping or missing
  • Content-Type
  • Managed Metadata tag
  • Choice column value
  • Lookup Value
Badly formatted DateTime or bad time zone calculation
Non numeric value for numeric columns
Non-existing or inconsistently provisioned Site/Document Library
Bad URLs
Last modified: 7 July 2022