Download Migration Flow for SharePoint . New features implemented for next version

November 28, 2022 – New video on handling CSV files

Version 3.15 November 28, 2022
YouTube New button to open the Migration Flow for SharePoint channel
YouTube Video: Read CSV files – Output to For-Each or SharePoint list/Database table
Session dashboard Auto resize window height according to screen resolution
FileManagement The option [x] Delete folders if empty now also works even if the folders referenced are initially empty e.g. there are no files to process. This feature enables you to clean up empty folders in a hierarchy
PowerShell Scripts can now run in both Restricted and Unrestricted environments.
To set the policy on the computer start PowerShell in administrator mode and run the following command:
Set-ExecutionPolicy -ExecutionPolicy Restricted
This is the built-in CMDLETs available to interface your PowerShell scripts
Transformation The Replace action can now process multiple lines having the same “search value” thereby creating a list og values.
Data ETL The Load into Database now has a Delete table toolbar button. Much easier that having to issue a DROP TABLE command
version 3.10 November 2022
Transformation 3 new powerful transformation actions:
  • Expression. Allows you to have cascading expressions.
  • Validation. Allows for validation of individual values and adding of validation message to the Session Log.
  • Tag extraction. Extract any tag from a JSON or XML string.
Mapping List You can now use the DataETL with Mapping lists including the SharePoint users mappings list.
(Select/Insert/Update/Delete)
For-Each A Mapping list can now be the source for a For-Each loop
Session Log For-Each/Group-By is now adding a full report to the Session log for each Group call. This enabled detailed tracking and analysis of complex setups.
Variables New data type Date
All Steps The Call-Tree button now show a dropdown window instead of a dialog box. This makes it quicker to use.
Session log M365/File transfer: New button Cancel pending files that becomes visible when the job has completed but Azure has yet to complete the operation. This will signal to Azure to stop the async transfer rom Blob Storage to SharePoint. The status for files not yet confirmed by Azure is updated to “Canceled” and removed from the consolidated reporting.
All Steps New toolbar button in the upper right corner: Open Steps
Repository New button in the Home section: Browse all Steps . A convenient way to find and open a Step without having to navigate the folders
Notification The Options tab now has help text that explains the way notifications work
Transformations Replace-action has been enhanced with an option to output the matched values within the source string separated by ;#
This can be useful for creating single-value or multi-value metadata based on keywords located within for example folder name or file name.
(Previously you could only replace the matched values within the source string and return the converted string.)
M365 (Preview feature – disabled by default) Asynchronous Metadata Read API (AMR) has been implemented to eliminate throttling caused by reading large Lists & Libraries for example when doing incremental file migration.
AMR is an advanced API that Microsoft is enforcing to replace legacy CSOM for bulk read of list item data and library item metadata. According to Microsoft: If this API is not used then the entire Tenant can be at risk of being throttled, thereby severely impacting all users normal online work.
M365 (Preview feature – disabled by default) New connection property: Use Asynchronous Metadata Read API
The default is Autiomatic however you can disable the API. This very powerful and rather complex Asynchronous Metadata Read API (AMR) is being enforced by Microsoft to avoid throttling. On rare occasions when dealing with lists having few items (<2000) it can make sense to disable it.
FileManagement Dynamic URL/UNC when called from GroupBy (ForEach) can now mix multi-value variables, single-value variables and static text.
FileManagement File duplicate auto-renaming when called from GroupBy (ForEach)
For example myfile.docx myfile(1).docx myfile(2).docx ...
Documentation Job notification. How to create a Power Automate notification flow
Options on all Steps Notification using Power Automate now has a link to the article How to create a Power Automate notification flow
Version 3.00 October 2022
File Open New button to “Browse all Steps”. This makes it easy to quickly locate Steps across folders by start typing the name.
Dashboard New column Custom info that shows the latest assigned value as specified on the Options tab of any Step in your flow setup. This makes it easy to track and find specific sessions based on the data in context.
Dashboard The query now searches the columns Custom info and Message
Dashboard Made larger to display more rows.
Variables The Variable’s values are now saved form UI session to UI session. So when you have a variable having a specific value the this value is reloaded the next time Migration Flow launched.
On the Variables window you can disable this behavior.
Job notification Windows notification (sliding window above the system-tray) is triggered when a job is completed. This is not the case for scheduled jobs.
Session log The status “Pending” is now automatically changed when the file has been processed. So you no longer need to “Retrieve latest”
General Windows-F8 is a HOT-Key to bring Migration Flow in focus
General Program icon in the Windows System Tray. Click it to bring Migration Flow in focus.
Azure New real-time statistics columns on the Session log and Session dashboard that show how many migration packages that has been created and submitted to Azure and also how many packages that Azure has picked up for processing.
Version 2.92 October , 2022
Microsoft 365 SharePoint Connection: Azure AD App authentication type added.
The new authentication type enables you to more securely manage the permission given to the SharePoint account used by the connection.
Also, here is a step-by-step guide for how to create a self signed certificate, add the Azure AD app and configure it all.
The new authentication types are named:
  • Azure AD App (MyCert.pfx)
  • Azure AD App (Installed cert)
Documentation New article: How to create & configure Azure AD app for authentication.
This article is a step-by-step guide for how to create a self signed certificate, add the Azure AD app and configure it all. It is generic and has no content related specifically to Migration Flow
Scheduled jobs You can now Hold alle jobs from the top panel.
Job tab The buttons layout in the UI have been updated for optimal use
Job tab You can now cancel a job running on the Job Server from within the Job tab
Job tab The job schedule is now showing in the top panel and a hyperlink takes you to the schedule.
Dashboard New column Location that show where the job is running (Window/Job Server).
Repository SQL Server repository: Upgraded to use Microsoft ODBC Driver 18 for SQL Server released by Microsoft August 8, 2022
The driver is automatically installed by the Migration Flow installer
Version 2.80 September 2022
DataETL FileShare discovery: x2-20 scanning performance enhancement depending on number of files. The more files the more significant the permeance enhancement is.
DataETL FileShare discovery: New option to include permissions.
(AccessControlList entries also known as the ACL)
DataETL FileShare discovery: New column [AD owner]
DataETL FileShare discovery: New “Data extraction filter” option in toolbar.
The filter allows you to do advanced parameterized filtering identically to the “File Selection Filter” in FileManagement.
The filter supports Master-Detail filtering when referencing the permissions columns (AccessControlList/ACL). This allows you to find files having specific ACL entries.
Installer Microsoft OLE DB Driver for SQL Server v19 is now automatically installed
Database Support for Microsoft OLE DB Driver for SQL Server v19. Using that driver you will be able to leverage all the latest SQL language features in SQL Server 2022. You can also use it to connect to older versions of SQLServer.
FileShare The FileShare filter window now has help hyperlink to the documentation for the advanced Match string function that enables “light regular expressions” in the filter criterias
Version 2.69 August 2022
General New Save All button in the top right toolbar to save all open Steps.
General The Call Tree dialog box is enhanced to also show FROM references.
(The Steps from where the current Step is being called from)
For-Each New ability to create a transformation on a GroupBy assignment. This simplifies the process of making calculated grouping values by eliminating the need for a separate Variable assignment with associated and transformation.
DataETL Load into Excel: x11 Performance enhancement
SessionLog x14 Performance enhancement when opening reports in Excel
SessionLog x29 Performance enhancement when storing reports in the session log.
SessionLog New r-click menu item “View report” on embedded reports like Source.txt
SessionLog
Data tab
Data set viewer now shows row info: Current row, Filtered rows, Row count
Data tab When changing the limit the new result is automatically extracted.
Performance 90% performance enhancement for UI and calculation processing in jobs. This enhancement will make the UI more responsive and make all jobs execute faster.
Please note that the transfer to and retrieval from SharePoint is not affected in any way as that depends on network bandwidth and the SharePoint backend, but everything else runs faster.
DataETL/For-Each Extract from Custom delimited file has a new option to specify if the file has a header line that specifies the column names or not.
For-Each New option for the GroupBy processing
[x] Group all rows into 1 and call Step once
Each [<#MyCol>] Variable will contain the column values from all rows separated by ;#
Use the GroupBy functionality for high performance File-Cherry-Pickling and for combining values for use with multivalued SharePoint columns.
For-Each Variable assignments: Drag&Drop the sequence of the Variable assignments. This is useful when Variables are cross referencing each other and the calculation order matters. Also useful when using the GroupBy processing where the grouping assignments must be the last assignments.
Documentation New article how to use of the PowerShell Connect-PnPOnline CMDLET
Documentation New article with some simple PowerShell samples
PowerShell Simplified management of PowerShell references to SharePoint connections
Job Server Windows service account: enhanced error handling of missing account permissions.
Database Support for Microsoft SQLServer 2022 and all it’s new features. Please make sure to use v19 of the Microsoft OLEDB Driver
Windows Support for Windows Server 2022 and Windows 11 2022 update.
Installer Facelifted and upgraded installer
Version 2.56 July 2022
Job Server Simplified UI for automatic job rescheduling when a fatal error happens. [x] On-error reschedule job This option is default checked.
Documentation New article that describes the For-Each GroupBy feature
SQL Support for the SQL keyword WITH in conjunction with SELECT. This enables advanced result stet generation like a path hierarchy derived from a linked list of records. Useful for migration of folder structures from legacy Document Management Systems (DMS) like IBM FileNet.
See syntax examples here.
This article describes more detail about selecting Parent-child relational rows.
File Management UNC file specification support added for GroupBy arrays
FileManagement When uploading files to SharePoint Online the log rows now initially has the values “Pending create” or “Pending replace”. This makes it clear what actions are pending to be completed by Azure
FileManagement Support for hardcoded destination setup
including Rename File and Alternative Path
when using GroupBy values from a ForEach Step
Data ETL Load into Database: Using value mappings is now significantly faster.
Value mapping Paste values copied from Excel is now x50 faster and an hour glass is displayed during the import
Value mapping New keywords you can mix with text you type in the replacement value.
  • <today> Converts to todays date. You can use a format string to force a specific format like: <today:yyyy-mm-dd>
  • <now> Converts to current time. You can use a format string to force a specific format like: <now:hh:mm:ss>
  • <tab> Converts to TAB character
  • <newline> Converts to carriage return + line fed
    Char(13) + Char(11)
  • <cr> Converts to carriage return character
    Char(13)
  • <lf> Converts to line feed character
    Char(11)
  • <space> Converts to space character
  • <semicolon> Converts to semicolon character
  • <pipe> Converts to pipe character
Version 2.49 July 2022
UI Filter as you type in dropdown menus for Variables and columns
FileManagement M365 – Apply users that do not exist to [ModifiedBy] or [CreatedBy]: Performance enhancement of x50 when >100 users do not exist.
Documentation Updated documentation for the consolidated reporting views:
Run job You can now specify Session name and Custom info when manually starting a job. This information is useful when selecting different job sessions and when querying the the consolidated reporting views.
Extract data Tooltip on the result table generated from the “Extract data button” (upper left toolbar). Useful when querying the the consolidated reporting views
Session Log The default retention schedule is set to 3000 so that you will by default store 3000 job sessions in the consolidated reporting views.
SharePoint Enhanced CAML support for query of nested Managed Metadata tags.
SharePoint Enhanced support for creating Document Sets from a For-Each Step using inherited content-types and variables.
DataETL CSV/Excel source: If the MS Access runtime engine is not installed a massage now offers to download from Microsoft and install it.
System Steps are now auto reopened when starting the program
Version 2.45 June 2022
Selective
FirstTimeMigration
ReMigration
RertryMigration
Major new feature to selectively: ReMigrate, RetryMigrate, FirstTimeMigrate
The selective migration feature is based on the compressive For-Each Step that has access to the new consolidated migration reporting data source that contains reference information and results for all processed files from all SharePoint tenants, farms, sites and fileshares across all jobs and sessions. The selective migration features include:
  • ReMigration of incorrectly migrated files for example when metadata calculations and user mappings needs to be revised.
  • RetryMigration files that for some reason failed for example network issues, faulty provisioned sites, bad URLs etc.
  • FirsTimeMigration of files based on advanced criteria.
  • Ultra high transfer performance using parallel transfer
  • Job results are automatically “merged” into the consolidated reporting data source for incremental and successive actions either for reporting or further migration. There are no useless CSV or Excel sheets floating around.
  • Run a job in test-mode to simulate the transfer and thereafter investigate the results and selectively decide what files to process for real. And in the process make changes for dynamic metadata, URLs, users and mappings etc.
  • Selection is done using the simple UI in the For-Each Step (source) and you can use raw parameterized SQL if you desire.
  • Embedding of the selective migration feature in any Step flow (automation) since it is based on a standard For-Each Step
  • Any data source can be used for the selective migration. Not only the internal consolidated reporting data sources.
General Revised Tabbed UI for easier navigation between open Steps.
SharePoint Managed metadata sub term query support. You can now query for a specific Managed Metadata sub term using the PnP ParentTerm|SubTerm syntax. Also direct term GUID is supported using {my-term-guid}.
Sub term criteria and GUID criteria works using the
operators: = (Equal) and <> Not Equal
The GUID based query is activated only when specifying | (pipe). So if you just specify a term label then all terms matching the label regardless of position in the term hierarchy are considered.
SharePoint Managed metadata sub term path extraction support. When you retrieve values from Managed Metadata columns the full term path is now returned.
For-Each GroupBy in combination with FileManagement now support subfolders. That enables you to GroupBy an entire Library. So you can use a grouped variable (multi value) in the destination Dynamic URL/UNC. To rename a file use the assignment to “rename file” as in all other scenarios.
General New button Computer performance index located in the Account screen that will give you an index value for how fast the computer and database is performing. Performance index value of 1.0 is minimum required for running Migration Flow with acceptable speed.
FileManagement SharePoint Online: New Move folders option allow you to move folders around super fast within the same Document Library or between libraries on the same site. The MoveCopyUtil api is used and it is super fast as no binary transfer is needed. Also, any Check-out status is retained and the file GUID does not change. When working within the same library there are no limitations but moving folders between libraries on the same site is subject to the SharePoint API ListViewThreshold limitation of 5000 items (subfolders + files being moved in 1 operation).
Data types Set data types in DataETL and For-Each now has a new button to reload the original data types from the source table. This is useful when you want to reset what ever changes you have made to the data types or the schema for the table. Useful also when the schema has been changed externally.
SharePoint New Variable support for retrieving items starting at a given folder level (right-click menu in the folder column). This is useful for example when dynamically retrieving all folders at a given level to move them elsewhere.
PowerShell New default script that includes Try Catch and session logging. This makes it easier to get started.
PowerShell New CMDLET for connection to SharePoint
Migrated files report Performance enhancement x50
Database View support for the Internal repository
Session Log New column [Custom Info] to display the custom logging information defined either on the Options-Tab or assigned to the system Variable [<SessionLog-CustomInfo>] from a For-Each Step.
Session Log Find and Filter now also considers the new column [Custom Info].
File reporting Ability to cancel a running report.
File reporting New filter on Custom info
File reporting New filter on Copied or Moved
File reporting New ability to do reporting on files subject to “Test run”
File reporting New database reporting view report_migratedfiles located on the internal repository that enables you to do reporting across all jobs/steps/sessions/individual For-Each iterations. The output can be loaded into Excel, CSV, SP-List or database just like any other data source. Then the reporting can become become part of flow setup.
SharePoint user validation reporting New database reporting view report_validatedsharepointusers located on the internal repository that enables you to do reporting across all jobs/steps/sessions/individual For-Each iterations. The output can be loaded into Excel, CSV, SP-List or database just like any other data source. Then the reporting can become become part of flow setup.
Data validation reporting New database reporting view report_datavalidationwarnings” located on the internal repository that enables you to do reporting across all jobs/steps/sessions/individual For-Each iterations. The output can be loaded into Excel, CSV, SP-List or database just like any other data source. Then the reporting can become become part of flow setup.
DataETL/ForEach SharePoint: Override URL no longer needs | (pipe character) to specify subfolder, you can just use / as you would normally do in a SharePoint URL
DataETL Load into Database: New progress information added to the Session log. This is nice when for example doing 1 million inserts.
DataETL/For-Each Extract data: Added limit result set to nn rows for SQLServer, CSV, Excel (SharePoint was already supported)
Preview Enhanced interactive filter in the preview windows for DataETL, For-Each and FileManagement. The filter now automatically follows the column you sort by and applies the filter.
Job Available memory is now monitored and if the threshold of 2 GB available memory is reached the job will issue a warning
Transformation Replace action has 2 new replacement options: All but first occurrence and All but last occurrence
Version 2.23 April 2022
Session Log Performance enhancement (x3) when loading large sessions on machines that have background Accessibility software running. Most windows 10 & 11 have that
SQL Server INSERT/UPDATE raw SQL statements can now use the SQLServer specific syntax N'my string value' to support UNICODE character set for columns of type NVARCHAR. Notice tables columns created from the IDE will be of type VARCHAR. So, to support UNICODE you must create/alter your table using embedded SQL or use SQL Server Management Studio.
Job Variable values can now be returned into the IDE from a job. This is very convenient for debugging and testing jobs that generates a dynamic filenames or other values dynamic values that the next Step to test is using.
Job SharePoint as source: new option to specify how many items to retrieve from the source. This is very convenient for debugging/testing
PowerShell Enhanced longing of PS exception trace:
The log generated when an exception is thrown in the above example:
M365 Azure Migration API – package retry. This feature catches the errors that Azure randomly throws for no reason when transferring data from blob storage to SPO and then performs async retry of the Azure job without having to reupload the binary content and metadata. Wen a retry occurs it is indicated on the AzureJobLog.txt file (Retry)
Job notification You can now run a Step or series of connected Steps to perform job notification. The flow can for example update a database or SP list with notification relevant data.
You can invoke your notification Step from any Step in your job also within in a For-Each loop thus enabling individual notification for each iteration in the loop
Note that the notification will not be invoked synchronously when there are files pending in Azure (i.e.. files that are uploaded successfully but not yet moved by Azure from temp blob storage to SharePoint).
When all files processed within the the Step is accounted for by Azure then the notification is triggered. If an error occurs asynchronously in the Azure migration import API, it is reflected in the notification even though the upload of the file succeeded. The entire Variable context is restored when the notification is triggered asynchronously.
The below variables designated for the notification are also populated:
  • [<NotificationType>] The job result result since the last notification
  • [<NotificationMessage>] The job message issued since the last notification. The first 10k of log data for warnings and errors is accumulated. If you update a database you need to make sure the column width is large enough or truncate the value.
  • [<NotificationFileCount>] The number of files processed within the Step the notification is linked to.
  • [<NotificationContext>] The information provided from the Step setup by you.
    This is particularly useful when you use your own job tracking setup with for example a SharePoint list that specifies parameter for the job as a whole or for For-Each iterations.
SharePoint Create Folder/DocumentSet has been redesigned to avoid throttling. This may reduce the performance in terms of number of folders/DocumentSets created per minute in order to comply with the Microsoft throttling policy change.
FileManagement [x] Include subfolders is now visible also on the Home tab
Session Log Dynamic colour change for expanded levels
SharePoint Override URL no longer needs | (pipe) to separate Library from folder structure. This makes it significantly easier to use/understand
FileManagement The FileShare attribute Last Accessed can now be referenced in filters and be used for metadata assignment and calculation.
Version 2.04 March
Database Drop down menu now shows database VIEWS
SharePoint Generic DMS version history migration: x17 performance enhancement for major/minor version calculation.
Job Server New error tracking for Windows Service account permission issues. This makes it easier to figure out what is wrong with a given Windows Service account.
UX multi monitor Step windows and multitasking jobs now open on the same monitor as the parent window. Also the resize logic now use the screen resolution for the monitor in context
Data ETL/ForEach “File and Directory scan” now includes the [LastAccessed] attribute for files. This enables a variety of applications for example only migrating the most recent used files or creating PowerBI models that shows what files are actively being used and what files are ROT (Redundant Obsolete and Trivial).
Data ETL/ForEach “File and Directory scan” Performance optimized for retrieving the first level of folders. (no longer runs through the entire hierarchy)
Version 2.01 February 2022
Data ETL Edit table New button to edit a database table’s column layout
Data ETL List settings New button to open the SharePoint List settings in a browser
File Management Library settings New button to open the SharePoint Library settings in a browser
File Management Migration of legacy DMS document major/minor version history.
Supports Alfresco, OpenText, FileNet or any DMS where the metadata for document versions is stored in a database, CSV or Excel.
For-Each Group By: New functionality to perform data grouping. It is essential for transforming multiple rows into 1 value per. column based on one or more GroupBy columns. Use cases include:
  • Create values for SharePoint choice-columns or multi value Manged Metadata and Lookup columns.
  • Apply the multi value Variables in SharePoint and Database filters using the IN filter clause.
  • Document Management System migration where you can use a single result set (that you can create/prepare in various ways) as Source to a FOR-EACH and automate “FileManagement”. Then you GroupBy Site/Folder and the FileManagement step will optimize and “batch” files from various sources into 1 azure session per destination. That will effectively increase performance anywhere from x2 to x50 compared to single file processing and moreover make metadata assignment and calculation much easier and transparent.This feature is extremely useful for distributing content to many sites based on a result set that defines the destination Sites, Source storage location and metadata.
SharePoint Managed metadata tags in a hierarchy can now be referenced in compliance with the PnP framework using the pipe character |
SharePoint Online File Management: The version for a file can now be specified even the file has no version history. use the new [Set version] column in the metadata assignment.
Session log Auto-refresh of log for running job
Session log Enhanced filtering. Also works real-time on running job. Option to chose to view errors only.
Session log x15 faster purge of outdated log sessions. Deleting specific sessions is done instantly.
Session Log New viewer for analysing embedded reports like SourceFiles.txt This is convenient when running on a machine that does not have Excel installed. Instead of opening the report in Notepad or Excel a new tab opens within the Step where the report can be analysed. The content can be sorted, filtered and exported to Notepad or Excel.
Session log New r-click menu option to view File Version History
Version 1.95 January 2022
Language support Added support for the following languages:
  • Thai (Thailand)
  • Vietnamese (Vietnam)
SharePoint Handling of columns level validation rules in SharePoint Lists and Libraries. The defined validation message from the actual SharePoint rule is also returned to the session log along with the data value for your context understanding.
SharePoint Enhanced handling of corrupted SharePoint List column definitions.
MS SQL Server Support for having the repository in a SQL Server database rather than locally in the program-public folder. This is useful for compliance with data protection policies and also gives better performance for large session logs.
Transformation New function IndexValue
Variables The expression test value for [data] is now reloaded when the Variable properties window is opened
Session Log Ctrl-A will now select the message text
File Management Optimized file copy operation between FileShares
Session Log Load/Expand 35% faster
Version 1.93 December 2021
Run Job interactively Columns are now sized automatically to make it easier to view data
Validated users SharePoint: New report window to view all validated users. Useful to se how users are mapped and how they resolve against the AD
Public preview These are the features not yet ready for GA but included for you to preview
Version 1.86
SharePoint Online “Refresh Schema” is now x2-x80 faster (The bigger Termstore and number of content-types the faster)
SharePoint Online “Connection” has new option to include hidden lists and libraries
Database New option to create a connection to CSV/Excel Sheet directly from the Database connection window.
Functions New system function RemoveIllegalcharactersFromFileName
Data Transformation Test transformation for SharePoint user columns: When transforming values the user mapping list is now included in the test so the actual mapped result is shown. Also, you can edit the mapping list from within the transformation window
SharePoint user validation User name validation can now be performed in “Data Transformation” and “Edit user mapping”
Data Transformation New “Paste value” button for SharePoint columns of type Managed Metadata, Lookup, Content-Type, Choice
Job log New right-click “Open SharePoint version history” in Notepad or Excel”
General Mouse wheel now scrolls the options tab pane on all Steps
Version 1.79
General Digitally signed executables and Setup
FileManagement New method for excluding folders based on partial string matching on the columns [Path] and [URL path]. For example *forms*;*system* will exclude/include/include+ BOTH files and folders if the path string contains “forms” or “system”. It is coded to work specifically on the column [URL path] for SharePoint source and [Path] for FileShare source.
Version 1.77
DataETL & FileManagement New Preview filter. When running a job visually with preview you can now filter the content to process based on the sort column.
DataETL & FileManagement SharePoint user validation: The default user can now specify additional domain(s) that will be used for user validation.
The syntax app@sharePoint;mydomain.com will try to use mydomain.com applied to a user name if a given user do not validate.
  • Example1: user JOHN will first be validated using the name as it is. If that fails then validation is attempted using JOHN@mydomain.com
  • Example2: user ANN@abc.com be validated using the name as it is. If that fails then validation is attempted using ANN@mydomain.com
Transformations Now the value mapping list name is displayed along side the transformation instance to give better overview. Also the mapping list can be opened directly without having to open the transformation.
Transformations New Final result column to show the result of the expression combined with the transformation actions.
Job tab New visual indication on the Job tab when a running job has completed. Job completed successfully etc.
Job tab New button on the Job tab Goto preview job will take you to the running preview session that might be hidden behind other windows. This makes it easier to find visual a running job on the machine when there are many windows open.
Version 1.67
DataETL & ForEach SQLServer: Enhanced SQL parsing of tables and columns having all sorts of characters in the names. Added support for the UNION keyword without a FROM clause.
Job scheduling New toolbar button added on the Job ribbon tab: Managed scheduled jobs for easy access to the job scheduler. Also the job scheduler opens immediately after a job has been submitted for easy review and status monitoring.
Scheduled jobs New toolbar button to easily select a Step and schedule it to run without first having to open the Step and go to the Job tab.
Other changes
DataETL & ForEach SQL Source: The editor window is now dynamically resized to give more editor space.
Reporting Performance enhancement: Retrieve file migration report is now x3 faster.
Data ETL & FileManagement Support for SharePoint system Libraries SitePages and SiteAssets
Session Log Load time performance enhancement when having >75000 log rows in one session.
Variables & Assignments Visual redesign of “Edit” window and new column Description added.
Variables New “Referenced” list box where all references to the selected Variable are listed. You can se the Step name, Step type and the location where the variable is being used.
Variables Multi-select is now supported for deleting. (Ctrl-A, Ctrl-Click, Ctrl-Shift-Click)
Variables New check box “Show only unreferenced”. Applies a filter so only Variables that are not being used anywhere are listed. This is very useful for cleaning up by deleting unused variables
All Steps New option to specify a Custom log message (Options tab). Useful for adding context specific information to the Session Log for job tracking and analysis. The information is globally searchable across all sessions and all Steps from the Dashboard.
All Steps New option to specify what to do when an error occurs. valid options are:
  • Log a warning and continue (the default)
  • Log a warning and exit current Group Step
  • Stop the job
DataETL & ForEach & FileManagement New option to specify what to do when no data or files are retrieved from the Source. Valid options are:
  • Ignore and continue (The default)
  • Log a warning message
  • Log a message and exit the current Group Step
  • Log a warning and exit the current Group Step
Group Step New hyper link to open embedded Step.
Group Step New condition return value keywords:
  • "ExitGroup" , "ExitGroupWarning" to not call the current Step or any subsequent steps within the Group.
  • "StopJob" , "StopJobWarning" to stop the job immediately.
Data ETL SharePoint Item Update: New option to react when not all source keys matches a destination Item. You can trigger a warning, exit the current Group Step or stop the job. This is very useful when your business rules dictate that every source key must match a destination Item. Note that the condition is detected before any action is taken so no data is harmed.
Data ETL SharePoint Item Update: New option to react when multiple items are updated based on a single source key. You can trigger a warning, exit the current Group Step or stop the job. This is very useful when your business rules dictate that only 1 item must be updated for each source key. Note that the condition is detected before any action is taken so no data is harmed.
Data ETL & FileManagement New option to react when an error occurs. You can trigger a warning (the default), exit the current Group Step or stop the job. This is very useful when for example a dynamic URL is wrong and it makes no sense to continue processing the current iteration in a For-Each or the entire job.
Session Log New hyper link to open nested Step. The link has tooltip that shows the storage location of the Step
Session Log Message can now be 8k (previously 2k) and is globally searchable across all sessions and all Steps from the Dashboard.
PowerShell Add-FlowLogRow new arguments to -LogTypes:
  • -LogType StopJob Stop the job
  • -LogType ExitGroup Exit the Group Step from where the PowerShell Step is called.
Calculation New functions:
  • RemoveIllegalCharactersFromFileName()
  • ReadTextFile()
  • FileExist()
  • GetFileNamePart()
Data ETL SharePoint: Support for extraction of custom list item version history to database(Insert/Update), Excel, csv, ForEach.
You can specify criteria for the extraction: Date of oldest version to include, hard coded or using a variable. When previewing the retrieved SharePoint Item data you have the option to visually also show the version history.
(Please note that the version history cannot yet be applied to new SharePoint custom list items. Version history is 100% supported in File Management)
File Management SharePoint version history: You can now specify criteria for the extraction: Date of oldest version to include, hard coded or using a variable.
Column mapping Data-ETL & File Management: New Ribbon bar button to automatically map all columns & reset mappings
Documentation New article SharePoint user mapping. There is a link from Options user mapping to the article within the product.
FileManagement Allow any user name (Only O365) – New validation message to verify that the conditions for this option to function are met
FileManagement The width of Override URL has been extended to 300 characters to allow for long URLs consisting of multiple Variables
Data ETL new link to open CSV folder and Database connection properties
Data ETL & FileManagement New visual indication when a “From” or “To” filter is defined. The indication also serves as a shortcut link to the filter definition
File ribbon Now navigates to the location of the Step in context.
SharePoint Data-ETL & File Management: Emoji support (❤️ Resource for copy/paste Emojis ). Item properties (including folder names) can now contain Emojis. You can even query for items having a specific Emoji.
Job Log New ribbon bar button to open the Dashboard
Job Log Find/Filter: Enhanced visual highlighting and keyboard navigation from the find field.
Job Log New filter option to apply “find criteria” as a filter to limit the rows visible.
Job Log Find and filter criteria is now applied immediately when you type.
Job Log Message column is now permanently expanded but only when clicked.
Job Log New “Open button” to open the Step the current log line belongs to.
Data ETL To SharePoint Document Library: New action Check out will check out files to the account defined in the SharePoint Connection.
Data ETL To SharePoint Document Library: New action Discard check out will remove the checked out status from on files.
Data ETL Preview window: New maximize button to expand the preview Window
File Migration From FileShare: Automatic conversion of file names that are URL encoded. For example My%20file.docx is converted into My file.docx before transfer to SharePoint
File Migration Preview window: New maximize button to expand the preview Window
Notification PowerAutomate and E-Mail: Delayed notification for SharePoint Online. Notification is not sent until all files have been verified by Azure.
Notification PowerAutomate and E-Mail: New field “FileSummary” that shows the number of files Ok, Warning, Error
File Migration Azure/SPO destination. Notification (if enabled) is not triggered until all files for a given step has been processed by Azure.
Account New “Diagnostics” setting to enable logging of the raw data sent to SharePoint and databases.
Step Revised toolbar at top right corner. All commands from TOOLS are now here also.
Step Ctrl+S in all steps does a SAVE
Data-ETL New input type “Variables”. Useful for SQL DDL statements and operation that are entirely based on variables hence no input result set.
File Reporting 3x Faster retrieval of reports. (Revised indexing and SQL WHERE syntax)
File Migration Revised detection of the condition where the computer system is out of memory and many things begin to fail.
File Migration Azure pending files: Optimized visual status for how many files are pending asynchronously in Azure. New article that describes that scenario
Migration OnDemand New article that describes what you can do with the template plus a comparison to MS Mover and MS Migration Manager
Data transformation New UI for data transformations in Variables and column assignments
Dashboard Visual indication of files awaiting Azure Transfer to SharePoint
Dashboard New column Job PID that show the Windows Task Manager PID
Expressions New mechanism to test expressions

Feedback

Was this helpful?

Yes No
You indicated this topic was not helpful to you ...
Could you please leave a comment telling us why? Thank you!
Thanks for your feedback.

Post your comment on this topic.

Post Comment