| Version 5.20 | April 22, 2024 |
|---|---|
| SharePoint | The column [Property Bag] is now supported for retrieval. |
| FileManagement | Enhancement to Pending files gives quicker detection of when the files are in fact completed by the associated Azure Migration Job. |
| FileManagement | SharePoint >> FileShare with version history now includes all metadata columns in the associated csv file. |
| Teams | Support for empty folders in fie Discovery |
| Version 5.14 | March 21, 2024 |
|---|---|
| File Management | New options for alternative modified detection |
| PowerShell | SharePoint Online: Support for Connect-PnPOnline using Entra App |
| Date ETL | File Share Discovery: Support for unlimited File Share size. |
| Date ETL | File Share Discovery: performance enhancement when scanning FileShares >10.000 files. The destination is populated in parallel. So while the scanning takes place chunks of 10.000 file names are applied to the destination in parallel. |
| SQL Server | All templates are now using the UTF8 codepage. Latin1_General_100_CI_AS_SC_UTF8 |
| Version 5.10 | March 12, 2024 |
|---|---|
| Teams | SMART Migration for Teams: Multiple Teams channels are now processed automatically (Enterprise+ license required) |
| Teams | SMART Migration for Teams: Enhanced layout in SMART Migration Job |
| Teams | SMART Migration for Teams: Enhanced Color coding in views |
| Teams | SMART Migration for Teams: Enhanced issue tracking with direct link to associated Dashboard item from an issue item. |
| Teams | SMART Migration for Teams: Discovery performance enhancement x5. Now creates up to 80.000 FileProxy items per hour. |
| FileManagement | SharePoint Online: 30-150% performance enhancement |
| FileManagement | SharePoint Online: Folders are now created on demand along with the files. |
| FileManagement | FileShare source: The column [File Type] now has the value <folder> when the item being processed is a folder. |
| Version 5.0 | February 16, 2024 |
|---|---|
| Teams | SMART Migration for Teams initial release |
| SharePoint | ZERO THROTTELING implemented by using the Rate-Limit headers. This effectively eliminates 429 Exceptions |
| UI | Numerous enhancement to the UI including new module to do interactive query and ETL without having to create a dedicated Step |
| Installer | All prerequisites are now installed automatically such as .NET 6.0 etc. |
| Repository | SQL Server is now the repository |
| Repository | New ability to mange SQL Server : create new databases etc. (Account tab) |
| Job Server | Ability to reconnect to SQL Server if it has been temporarily unavailable. |
| FileManagement | Ability to auto-shorten too long path before upload to SharePoint |
| FileManagement | New property [<IgnoreOnlyNewer>] to force a file overwrite. To activate set the property to 1 |
| FileManagement | Ability to update metadata on folders (Options tab) |
| SharePoint | [Approval-status] and [Approval comments] are now migrated as is (if mapped). Previously all files would get the status Approved. |
| Functions |
|
| ETL | Database query sow supports the Skip! keyword |
| Version 4.28 | December 29, 2023 |
|---|---|
| Dashboard | Search for [SessionLog-Id] |
| DataETL | SharePoint Online: Emoji support. Use Windows key and . (dot) to insert emoji |
| SharePoint | Support for: [Folder Child Count] & [Item Child Count] |
| FileManagement | SharePoint: Upload is now returning the GUID for the uploaded file and apply it to the Report_MigratedFiles view. Notice the GUID can be used in SharePoint search for find the file within a site collection. It takes up to 24 hours before the GUID is searchable within SharePoint. |
| FileManagement | Validation including complex data types and handling of identical values accomplished by multiple edits of multi-value values. |
| FileManagement | SharePoint Online: Folder color support |
| Transformation | New function: ExtractMetadataColumn()String data,String column, String Default value |
| SharePoint | Session log: New format for Managed Metadata that show the assigned tag much more clearly and can be used directly in assignments from the New function: ExtractMetadataColumn() |
| Version 4.01 | September 16, 2023 |
|---|---|
| DataETL | SharePoint: Selecting a different list or library is now keeping settings. |
| FileManagement | SharePoint: Selecting a different library is now keeping settings. |
| FileManagement | SharePoint: Upload is now returning the GUID for the uploaded file and apply it to the Report_MigratedFiles view. Notice the GUID can be used in SharePoint search for find the file within a site collection. It takes up to 24 hours before the GUID is searchable within SharePoint. |
| DataETL | You can now validate that files have been successfully migrated. |
| Transformation | New function: ExtractMetadataColumn()String data,String column, String Default value |
| SharePoint | Session log: New format for Managed Metadata that show the assigned tag much more clearly and can be used directly in assignments from the New function: ExtractMetadataColumn() |
| Dashboard | Search for [SessionLog-Id] |
| DataETL | SharePoint Online: Emoji support. Use Windows key and . (dot) to insert emoji |
| Version 3.99 | August 17, 2023 |
|---|---|
| Session Log | SharePoint: New format [Display name<Internal name>] |
| Duplicates detection | New Function FileCheckSum that allows you to calculate the file check sum (or hash key) that you can use for file duplicates detection |
| Repository | Latest Microsoft MS SQL Server support OLEDB 19.3 May 18, 2023 C:\Windows\SysWOW64\msoledbsql19.dll |
| Variables | New Variable SessionLog-Name. Sets the session name visible in the “Job session drop down menu” |
| SharePoint | Automatic time zone detection. The site no longer has to be in the same time zone as the local machine. |
| FileManagement | New metadata assignment:[<Ignore «Only newer» condition>]When True a file that already exist on the destination is overwritten with the matching source file regardless of it having a newer or identical timestamp compared to the source file. Use this option to force remigration of specific files. Valid values: Unassigned/True/False |
| File Management | New option: [x] Transfer files having data validation warning(s)Files are transferred despite any warnings like: Wrong Content Type Wrong username Invalid managed metadata Invalid dates etc. Please note that the files having data validation issues are inconsistent and will not be considered successfully migrated. (Default it is [x]Checked)
IMPORTANT: if you [ ]Uncheck this option you get the benefit of never uploading files that have inconsistent metadata. This makes it significantly easier to fix your data issues on the specific files and then run an incremental job. |
| Version 3.94 | July 17, 2023 |
|---|---|
| UI | Reengineered Step-Tab system with close buttons on the Step-Tabs etc. |
| UI | Reengineered dropdown menus and many other minor UI enhancements |
| DataETL | SharePoint: Support for creating folders also in Document Libraries. |
| FileManagement | Override Variables |
| Version 3.80 | May 15, 2023 |
|---|---|
| DataETL | Load into SharePoint now support Create Folder for Custom Lists. To create a folder set the column [Item Type] = 1 (The value 1 is the SharePoint identifier for a folder.) The folder name you can assign to the column [Title] or include it in the folder path that you assign to the columns [Path].The path columns supports both / (forward slash) and \ (back slash) for hierarchy folder level separation.
Features include:
|
| Repository | The SharePoint connections and Database connections now remember the latest used connation and setup when you close and restart the UI. |
| Repository | The SharePont connections Data Tab has 3 new button to refresh the result set. |
| Repository | The Database connections Data Tab has new buttons to refresh the result set, export to excel and Notepad. |
| File Management | Filter now support * (wildcard) for the IN operator and the = operator. |
| OnDeman | The OnDemand configuration wizard now supports AzureAD App authentication. |
| Version 3.78 | May 4, 2023 |
|---|---|
| Job Log | New Export to Excel and Export to Notepad of all or selected rows (Multi select of rows is supported). Also, when positioned on an embedded file like Source.txt the report will be opened in the same way as if you r-click the report row. This combined with the global search across all sessions and jobs and also combined with the filter gives you additional analysis capability. |
| Call tree | New buttons in the variables tab to : Copy , Notepad , Refresh |
| UI | Many minor refinements. |
| Version 3.73 | April 26, 2023 |
|---|---|
| SharePoint | New system requirement: We are now using PnP.Core and the MS Graph and that requires .Net 6.0 to be installed on the computer. |
| SharePoint | Support for download of files > 2GB |
| SharePoint | Redesigned “CAML query” encapsulation to bypass the SharePoint Online 5000 list item threshold limitation. The redesign is done due the the fact that Microsoft is now much more likely to throw the fatal List View Threshold exception when querying indexed columns. |
| Dashboard | New server status check to inform you if the Job server is not running correctly. |
| Windows Event Log | New fatal error event reporting : The Job Azure “pending files” scanner was unable to connect to the database. No migration results from pending files will be retrieved. Notice that the Azure scanner runs within the Job Server. This event fires when the job server is incorrectly configured, typically the Windows Service account |
| Version 3.70 | March 29, 2023 |
|---|---|
| Mapping lists | Redesigned to be flat (no dialog box) |
| Version Control | Redesigned to be flat (no dialog box) |
| Account | Redesigned to be flat (no dialog box) |
| Version 3.65 | March |
| Dashboard | Query across all job session can now run in parallel so you can start a long-running query and continue doing other tasks while it runs. |
| File reporting | Query for migrated files run in parallel so you can start a long-running query and continue doing other tasks while it runs. |
| SharePoint | Connections: New interactive data CRUD tool |
| Database CSV Excel |
Connections: New interactive data CRUD tool.
|
| System UI | The bottom menu have has been moved to the “Repository Tab” |
| Account | New button to backup the local database repository. |
| Version 3.57 | March |
| Transformations | 5 new expression functions to return NULL in SQL Insert/Update Right-click menu Paste function. Use these functions to force a NULL value into a database column when performing INSERT/UPDATE.Example: If( [mycol] = 0 , NullNumber() , [data] )
|
| DataETL | Load into Excel: New option to use an Excel file as template when creating new Workbook reports. Use this feature to make report files that has predefined layout. The layout options includes setting fonts and colors, defining calculations, adding graphics, adding pivot tables, reordering of table columns and much more. This article shows how to reorder table columns Note that you can populate an Excel workbook report with multiple sets of data to different tables and set single values to individual cells. And when populated then copy/move the Workbook report somewhere including to SharePoint. |
| Version 3.54 | March |
| Dashboard | Querying for session matching a criteria now supports || (OR) and && (AND) operators in the query string. |
| Dashboard | Querying can now be canceled so you are not forced to wait for the query to complete. |
| Dashboard | Opening and querying for sessions in the Dashboard is now x3 faster when using local repository (non SQL Server) |
| Dashboard | Sessions: New filter as you type option |
| Dashboard | Window Event Log: New filter as you type option along options for filtering event types such as your own or system events |
| SessionLog | New GLOBAL query option allows you to query the entire repository across all jobs and sessions. This makes it significantly easier to locate log rows not having to know what job it was and to examine logs across entirely different jobs. You can cancel a query so you do not have to wait for it to complete. |
| System | In the account window you can now reset the status for Do not stow again messages so they appear again. |
| Version 3.52 | March |
| SharePoint | Managed Metadata: Browse terms via the Select Value button now show the Termstore anchor tag GUID.This is useful when troubleshooting sites referenced via dynamic URLs where a given MangedMetadata tag cannot be resolved. |
| SharePoint | Managed Metadata: When a Term cannot be resolved the column’s TermStore anchor tag (GUID) is now part of the error message making it easier to troubleshoot. Use the SelectValue on the button on the columns assignment to make sure the anchor tags match. |
| PowerShell | New option to disable the PowerShell command line argument -NoProfile.Disabling this argument will instruct PowerShell.exe to load whatever is configured in your user profile. |
| PowerShell | New option to disable automatic TryCatch code encapsulation.Disabling automatic TryCatch encapsulation is NOT recommended unless you are debugging code and want to eliminate any outside interference. |
| Expressions | New function: IsServerJob() It retunes true if the job is running on the Job Server and false if it is running in the UI. This enables you to use different values for accessing external resources such as FileShares when the Job Server runs as a Windows Service on a different account than the UI. |
| Version 3.51 | February, 2023 |
|---|---|
| Notification | New option When error to run the notification ONLY if an error happens. |
| DataETL | Load into Database – new option:Update records. If nothing is changed then exit Group Step.When the DataETL Step is placed in a Group Step and the database update do not change any data then the flow exits from the Group. This enables you to make sure the current job made the update to the database table, not another parallelly running job. This is useful for designing “Thread Safe” applications where multiple simultaneously running jobs, pick tasks from a database table for example by querying a column [ReadyForProcessing]=1 and then update the column to 0 signaling that “This job has that task”. |
| Job Server | You can now Hold a job while it is running.The job status will change to Running to Hold and continue to run and when finished put it self on Hold. |
| Job Server | New scheduling option Continuously that forces the job to repeat it self after it finishes. Typically the computer overhead is 1-2 seconds for the repeated job start. |
| Job Server | Faster job start. The jobs are now started and fast as possible. This becomes important when you run multiple jobs that take advantage of the new Continuously scheduling option. |
| Job Server | Account window: New configuration of job server job start.[x] Delay each job start by 10 secondsIf checked, the machine workload associated with job start will be reduced. Also, flows can be designed to take advantage of a delayed job start in a multitasking setup. HOWEVER it is strongly recommended to use the new DataETL option: Update records. If nothing is changed then exit Group Stepto achieve a thread safe setup. |
| For-Each | GroupBy: When having large groups (>10000 rows) the group call is split into blocks of 10000 to sustain performance for data transformation. |
| System | SMART Migration now writes to the Windows Event Log when lost database connection happens, job termination due to server reboot etc. This feature enables you to better track what is going on and feed information to IT Infrastructure Monitoring systems. |
| Notification | You can now enable the notification to write to the Windows Event Log. This feature enables you to better track what is going on and feed information to IT Infrastructure Monitoring systems. |
| Dashboard | New tab “Windows Event Log” where you can view the information written to the Windows Event Log eighter by the system or by the job notifications. |
| UI | The Job Tree now shows “Notification” on the notification tree node |
| UI | FileManagement & DataETL: The assignment now shows more clearly what columns are not been signed. |
| DevOps Version Control | Separate environments for Development, Test and Production. Make changes only in the Development environment and cerate a version you then deploy to the Test environment and then when approved deploy to the Production environment. |
| DataETL ForEach | SQL data source: New button Refresh drop-down menuIt does a refresh of the columns names from SQL SELECT statement in the destination mapping drop-down menu. |
| Session log | New step-instance scroll buttons in Job tree. This makes much easier to navigate large session logs. |
| Session log | Step icon at all rows to show what Step class the row relates to |
| UI | Refreshed Step-Icons for easier recognition |
| Step browser | New r-click menu to Cut/Copy. This makes it easier to find Steps across all folders and copy/move to different folder. |
| SharePoint | You can now use Variables in SharePoint Connections URL and authentication columns to dynamically point to relevant sites in a DevOps Version Control setup. |
| Transformations | Rearranged UI for easier understanding of the transformation sequence. |
| UI | Job tab new <All sessions> in the session dropdown. This enables you to examine all logs consolidated. You can specify a date range filter and a message filter.The message filter supports AND and OR. This is an example of a criteria that look for this or that and must at the same time contain important: this||that&&important |
| UI | Job tab the FILTER and FIND now supports AND and OR. You can use || to specify OR and use && to specify AND.This is an example of a criteria that look for abc or xyz and must at the same time contain 123: abc||xyz&&123 |
| Version 3.34 | January 2023 |
|---|---|
| DataETL ForEach | FileShare discovery:
|
| UI | Job session log on Local default repository is now x4 faster in session retrieval and incremental refreshSQLServer based repositories do not benefit. They are already optimized. On both local and SQLServer repository, the Session Log does no longer retrieve incremental log information while not visible making the application more responsive especially when running multiple heavy jobs simultaneously. |
| UI | New “Job tree is replacing the previous “Call tree”. The Job tree shows the entire job-structure where a given Step is referenced and lets you easily navigate the structure. Also, you can se the different Variables used for each Step in the Job. It is open by default but you can close it if you like. |
| UI | Job tab status area rearranged. |
| UI | Job tab Redesigned Filter/Find functionality |
| UI | Job tab New icons added to easier log analysis |
| UI | Job tab Indenting to show sublevels by > > > > is replaced by the new Job tree. this makes it the log easier to read. |
| UI | Account window rearranged. |
| UI | New keyboard shortcuts:
|
| DataETL & ForEach | M365: 2 new virtual lists for retrieving user and group information at the tenant level. Requires Azure AD App authentication in the SharePoint connection. Advanced filtering is supported also with Variables.
|
| SharePoint | User mapping now have a drop down menu listing all users.
|
| SharePoint | New button to export all Active Directory users in M365 to Excel or Notepad. |
| SharePoint | User mapping now have hyper link to documentation: SharePoint user mapping |
| FileManagement | GroupBy: Enhanced error handling and reporting when files in the GroupBy data set does not exist or has invalid URL/UNC. |
| FileManagement | GroupBy: New embedded report file named MissingFiles.txt in the job log for the source files referenced in the GroupData that could not be located. |
| FileManagement | GroupBy: New progress information feed in the phase where the GroupBy data set is initially being analyzed so you better can follow what’s going on. |
| FileManagement | Copy from SharePoint or FileShare: Removed dependency on .Net 3.5 |
| FileManagement | Copy from SharePoint or FileShare: New option to Create metadata CSV file |
| Transformation | Enhanced tag extraction from XML |
| Version 3.19 | December 2022 |
|---|---|
| Video | New video that explains about For-Each looping |
| DataETL | Read CSV – automatic detection of UTF8 encoding in CSV files without BOM header. Often CSV files are saved as a standard text files without encoding header (No BOM) even the files contains UTF8 encoded characters. This leads to corrupted international character output when processed by database drivers. |
| DataETL | CSV now supports ¤ as separator character |
| DataETL | Load Into Database using SQL: Enhanced keyword coloring |
| Transformations | New function: ReplaceAll( ‘string1’, ‘searchfor’, ‘replacewith’ ) added to the expression Right-click menu Paste function. |
| Version 3.16 | November 29, 2022 |
|---|---|
| YouTube | New button to open the SMART Migration YouTube channel |
| YouTube | Video: Read CSV files – Output to For-Each or SharePoint list/Database table |
| Variables browse | Auto resize window height according to screen resolution |
| Session dashboard | Auto resize window height according to screen resolution |
| FileManagement | The option [x] Delete folders if empty now also works even if the folders referenced are initially empty e.g. there are no files to process. This feature enables you to clean up empty folders in a hierarchy |
| PowerShell | Scripts can now run in both Restricted and Unrestricted environments. To set the policy on the computer start PowerShell in administrator mode and run the following command: Set-ExecutionPolicy -ExecutionPolicy RestrictedThis is the built-in CMDLETs available to interface your PowerShell scripts |
| Transformation | The Replace action can now process multiple lines having the same “search value” thereby creating a list og values. |
| Data ETL | The Load into Database now has a Delete table toolbar button. Much easier that having to issue a DROP TABLE command |
| version 3.10 | November 2022 |
| Transformation | 3 new powerful transformation actions:
|
| Mapping List | You can now use the DataETL with Mapping lists including the SharePoint users mappings list.(Select/Insert/Update/Delete) |
| For-Each | A Mapping list can now be the source for a For-Each loop |
| Session Log | For-Each/Group-By is now adding a full report to the Session log for each Group call. This enabled detailed tracking and analysis of complex setups. |
| Variables | New data type Date |
| All Steps | The Call-Tree button now show a dropdown window instead of a dialog box. This makes it quicker to use. |
| Session log | M365/File transfer: New button Cancel pending files that becomes visible when the job has completed but Azure has yet to complete the operation. This will signal to Azure to stop the async transfer rom Blob Storage to SharePoint. The status for files not yet confirmed by Azure is updated to “Canceled” and removed from the consolidated reporting. |
| All Steps | New toolbar button in the upper right corner: Open Steps |
| Repository | New button in the Home section: Browse all Steps . A convenient way to find and open a Step without having to navigate the folders |
| Notification | The Options tab now has help text that explains the way notifications work |
| Transformations | Replace-action has been enhanced with an option to output the matched values within the source string separated by ;# This can be useful for creating single-value or multi-value metadata based on keywords located within for example folder name or file name. (Previously you could only replace the matched values within the source string and return the converted string.) |
| M365 | (Preview feature – disabled by default) Asynchronous Metadata Read API (AMR) has been implemented to eliminate throttling caused by reading large Lists & Libraries for example when doing incremental file migration. AMR is an advanced API that Microsoft is enforcing to replace legacy CSOM for bulk read of list item data and library item metadata. According to Microsoft: If this API is not used then the entire Tenant can be at risk of being throttled, thereby severely impacting all users normal online work. |
| M365 | (Preview feature – disabled by default) New connection property: Use Asynchronous Metadata Read APIThe default is Autiomatic however you can disable the API. This very powerful and rather complex Asynchronous Metadata Read API (AMR) is being enforced by Microsoft to avoid throttling. On rare occasions when dealing with lists having few items (<2000) it can make sense to disable it. |
| FileManagement | Dynamic URL/UNC when called from GroupBy (ForEach) can now mix multi-value variables, single-value variables and static text. |
| FileManagement | File duplicate auto-renaming when called from GroupBy (ForEach) For example myfile.docx myfile(1).docx myfile(2).docx ... |
| Documentation | Job notification. How to create a Power Automate notification flow |
| Options on all Steps | Notification using Power Automate now has a link to the article How to create a Power Automate notification flow |
| Version 3.00 | October 2022 |
|---|---|
| File Open | New button to “Browse all Steps”. This makes it easy to quickly locate Steps across folders by start typing the name. |
| Dashboard | New column Custom info that shows the latest assigned value as specified on the Options tab of any Step in your flow setup. This makes it easy to track and find specific sessions based on the data in context. |
| Dashboard | The query now searches the columns Custom info and Message |
| Dashboard | Made larger to display more rows. |
| Variables | The Variable’s values are now saved form UI session to UI session. So when you have a variable having a specific value the this value is reloaded the next time SMART Migration launched. On the Variables window you can disable this behavior. |
| Job notification | Windows notification (sliding window above the system-tray) is triggered when a job is completed. This is not the case for scheduled jobs. |
| Session log | The status “Pending” is now automatically changed when the file has been processed. So you no longer need to “Retrieve latest” |
| General | Windows-F8 is a HOT-Key to bring SMART Migration in focus |
| General | Program icon in the Windows System Tray. Click it to bring SMART Migration in focus. |
| Azure | New real-time statistics columns on the Session log and Session dashboard that show how many migration packages that has been created and submitted to Azure and also how many packages that Azure has picked up for processing. |
| Version 2.92 | October , 2022 |
| Microsoft 365 | SharePoint Connection: Azure AD App authentication type added. The new authentication type enables you to more securely manage the permission given to the SharePoint account used by the connection. Also, here is a step-by-step guide for how to create a self signed certificate, add the Azure AD app and configure it all. The new authentication types are named:
|
| Documentation | New article: How to create & configure Azure AD app for authentication. This article is a step-by-step guide for how to create a self signed certificate, add the Azure AD app and configure it all. It is generic and has no content related specifically to SMART Migration |
| Scheduled jobs | You can now Hold alle jobs from the top panel. |
| Job tab | The buttons layout in the UI have been updated for optimal use |
| Job tab | You can now cancel a job running on the Job Server from within the Job tab |
| Job tab | The job schedule is now showing in the top panel and a hyperlink takes you to the schedule. |
| Dashboard | New column Location that show where the job is running (Window/Job Server). |
| Repository | SQL Server repository: Upgraded to use Microsoft ODBC Driver 18 for SQL Server released by Microsoft August 8, 2022 The driver is automatically installed by the SMART Migration installer |
| Version 2.80 | September 2022 |
|---|---|
| DataETL | FileShare discovery: x2-20 scanning performance enhancement depending on number of files. The more files the more significant the permeance enhancement is. |
| DataETL | FileShare discovery: New option to include permissions. (AccessControlList entries also known as the ACL) |
| DataETL | FileShare discovery: New column [AD owner] |
| DataETL | FileShare discovery: New “Data extraction filter” option in toolbar. The filter allows you to do advanced parameterized filtering identically to the “File Selection Filter” in FileManagement. The filter supports Master-Detail filtering when referencing the permissions columns (AccessControlList/ACL). This allows you to find files having specific ACL entries. |
| Installer | Microsoft OLE DB Driver for SQL Server v19 is now automatically installed |
| Database | Support for Microsoft OLE DB Driver for SQL Server v19. Using that driver you will be able to leverage all the latest SQL language features in SQL Server 2022. You can also use it to connect to older versions of SQLServer. |
| FileShare | The FileShare filter window now has help hyperlink to the documentation for the advanced Match string function that enables “light regular expressions” in the filter criterias |
| Version 2.69 | August 2022 |
|---|---|
| General | New Save All button in the top right toolbar to save all open Steps. |
| General | The Call Tree dialog box is enhanced to also show FROM references. (The Steps from where the current Step is being called from) |
| For-Each | New ability to create a transformation on a GroupBy assignment. This simplifies the process of making calculated grouping values by eliminating the need for a separate Variable assignment with associated and transformation. |
| DataETL | Load into Excel: x11 Performance enhancement |
| SessionLog | x14 Performance enhancement when opening reports in Excel |
| SessionLog | x29 Performance enhancement when storing reports in the session log. |
| SessionLog | New r-click menu item “View report” on embedded reports like Source.txt |
| SessionLog Data tab |
Data set viewer now shows row info: Current row, Filtered rows, Row count |
| Data tab | When changing the limit the new result is automatically extracted. |
| Performance | 90% performance enhancement for UI and calculation processing in jobs. This enhancement will make the UI more responsive and make all jobs execute faster. Please note that the transfer to and retrieval from SharePoint is not affected in any way as that depends on network bandwidth and the SharePoint backend, but everything else runs faster. |
| DataETL/For-Each | Extract from Custom delimited file has a new option to specify if the file has a header line that specifies the column names or not. |
| For-Each | New option for the GroupBy processing[x] Group all rows into 1 and call Step onceEach [<#MyCol>] Variable will contain the column values from all rows separated by ;#Use the GroupBy functionality for high performance File-Cherry-Pickling and for combining values for use with multivalued SharePoint columns. |
| For-Each | Variable assignments: Drag&Drop the sequence of the Variable assignments. This is useful when Variables are cross referencing each other and the calculation order matters. Also useful when using the GroupBy processing where the grouping assignments must be the last assignments. |
| Documentation | New article how to use of the PowerShell Connect-PnPOnline CMDLET |
| Documentation | New article with some simple PowerShell samples |
| PowerShell | Simplified management of PowerShell references to SharePoint connections |
| Job Server | Windows service account: enhanced error handling of missing account permissions. |
| Database | Support for Microsoft SQLServer 2022 and all it’s new features. Please make sure to use v19 of the Microsoft OLEDB Driver |
| Windows | Support for Windows Server 2022 and Windows 11 2022 update. |
| Installer | Facelifted and upgraded installer |
| Version 2.56 | July 2022 |
|---|---|
| Job Server | Simplified UI for automatic job rescheduling when a fatal error happens. [x] On-error reschedule job This option is default checked. |
| Documentation | New article that describes the For-Each GroupBy feature |
| SQL | Support for the SQL keyword WITH in conjunction with SELECT. This enables advanced result stet generation like a path hierarchy derived from a linked list of records. Useful for migration of folder structures from legacy Document Management Systems (DMS) like IBM FileNet.See syntax examples here. This article describes more detail about selecting Parent-child relational rows. |
| File Management | UNC file specification support added for GroupBy arrays |
| FileManagement | When uploading files to SharePoint Online the log rows now initially has the values “Pending create” or “Pending replace”. This makes it clear what actions are pending to be completed by Azure |
| FileManagement | Support for hardcoded destination setup including Rename File and Alternative Pathwhen using GroupBy values from a ForEach Step |
| Data ETL | Load into Database: Using value mappings is now significantly faster. |
| Value mapping | Paste values copied from Excel is now x50 faster and an hour glass is displayed during the import |
| Value mapping | New keywords you can mix with text you type in the replacement value.
|
| Version 2.49 | July 2022 |
|---|---|
| UI | Filter as you type in dropdown menus for Variables and columns |
| FileManagement | M365 – Apply users that do not exist to [ModifiedBy] or [CreatedBy]: Performance enhancement of x50 when >100 users do not exist. |
| Documentation | Updated documentation for the consolidated reporting views: |
| Run job | You can now specify Session name and Custom info when manually starting a job. This information is useful when selecting different job sessions and when querying the the consolidated reporting views. |
| Extract data | Tooltip on the result table generated from the “Extract data button” (upper left toolbar). Useful when querying the the consolidated reporting views |
| Session Log | The default retention schedule is set to 3000 so that you will by default store 3000 job sessions in the consolidated reporting views. |
| SharePoint | Enhanced CAML support for query of nested Managed Metadata tags. |
| SharePoint | Enhanced support for creating Document Sets from a For-Each Step using inherited content-types and variables. |
| DataETL | CSV/Excel source: If the MS Access runtime engine is not installed a massage now offers to download from Microsoft and install it. |
| System | Steps are now auto reopened when starting the program |
| Version 2.45 | June 2022 |
|---|---|
Selective FirstTimeMigrationReMigrationRertryMigration |
Major new feature to selectively:
ReMigrate, RetryMigrate, FirstTimeMigrate The selective migration feature is based on the compressive For-Each Step that has access to the new consolidated migration reporting data source that contains reference information and results for all processed files from all SharePoint tenants, farms, sites and fileshares across all jobs and sessions. The selective migration features include:
|
| General | Revised Tabbed UI for easier navigation between open Steps. |
| SharePoint | Managed metadata sub term query support. You can now query for a specific Managed Metadata sub term using the PnP ParentTerm|SubTerm syntax. Also direct term GUID is supported using {my-term-guid}.Sub term criteria and GUID criteria works using the operators: = (Equal) and <> Not EqualThe GUID based query is activated only when specifying | (pipe). So if you just specify a term label then all terms matching the label regardless of position in the term hierarchy are considered. |
| SharePoint | Managed metadata sub term path extraction support. When you retrieve values from Managed Metadata columns the full term path is now returned. |
| For-Each | GroupBy in combination with FileManagement now support subfolders. That enables you to GroupBy an entire Library. So you can use a grouped variable (multi value) in the destination Dynamic URL/UNC. To rename a file use the assignment to “rename file” as in all other scenarios. |
| General | New button Computer performance index located in the Account screen that will give you an index value for how fast the computer and database is performing. Performance index value of 1.0 is minimum required for running SMART Migration with acceptable speed. |
| FileManagement | SharePoint Online: New Move folders option allow you to move folders around super fast within the same Document Library or between libraries on the same site. The MoveCopyUtil api is used and it is super fast as no binary transfer is needed. Also, any Check-out status is retained and the file GUID does not change. When working within the same library there are no limitations but moving folders between libraries on the same site is subject to the SharePoint API ListViewThreshold limitation of 5000 items (subfolders + files being moved in 1 operation). |
| Data types | Set data types in DataETL and For-Each now has a new button to reload the original data types from the source table. This is useful when you want to reset what ever changes you have made to the data types or the schema for the table. Useful also when the schema has been changed externally. |
| SharePoint | New Variable support for retrieving items starting at a given folder level (right-click menu in the folder column). This is useful for example when dynamically retrieving all folders at a given level to move them elsewhere. |
| PowerShell | New default script that includes Try Catch and session logging. This makes it easier to get started. |
| PowerShell | New CMDLET for connection to SharePoint |
| Migrated files report | Performance enhancement x50 |
| Database | View support for the Internal repository |
| Session Log | New column [Custom Info] to display the custom logging information defined either on the Options-Tab or assigned to the system Variable [<SessionLog-CustomInfo>] from a For-Each Step. |
| Session Log | Find and Filter now also considers the new column [Custom Info]. |
| File reporting | Ability to cancel a running report. |
| File reporting | New filter on Custom info |
| File reporting | New filter on Copied or Moved |
| File reporting | New ability to do reporting on files subject to “Test run” |
| File reporting | New database reporting view report_migratedfiles located on the internal repository that enables you to do reporting across all jobs/steps/sessions/individual For-Each iterations. The output can be loaded into Excel, CSV, SP-List or database just like any other data source. Then the reporting can become become part of flow setup. |
| SharePoint user validation reporting | New database reporting view report_validatedsharepointusers located on the internal repository that enables you to do reporting across all jobs/steps/sessions/individual For-Each iterations. The output can be loaded into Excel, CSV, SP-List or database just like any other data source. Then the reporting can become become part of flow setup. |
| Data validation reporting | New database reporting view report_datavalidationwarnings” located on the internal repository that enables you to do reporting across all jobs/steps/sessions/individual For-Each iterations. The output can be loaded into Excel, CSV, SP-List or database just like any other data source. Then the reporting can become become part of flow setup. |
| DataETL/ForEach | SharePoint: Override URL no longer needs | (pipe character) to specify subfolder, you can just use / as you would normally do in a SharePoint URL |
| DataETL | Load into Database: New progress information added to the Session log. This is nice when for example doing 1 million inserts. |
| DataETL/For-Each | Extract data: Added limit result set to nn rows for SQLServer, CSV, Excel (SharePoint was already supported) |
| Preview | Enhanced interactive filter in the preview windows for DataETL, For-Each and FileManagement. The filter now automatically follows the column you sort by and applies the filter. |
| Job | Available memory is now monitored and if the threshold of 2 GB available memory is reached the job will issue a warning |
| Transformation | Replace action has 2 new replacement options: All but first occurrence and All but last occurrence |
| Version 2.23 | April 2022 |
|---|---|
| Session Log | Performance enhancement (x3) when loading large sessions on machines that have background Accessibility software running. Most windows 10 & 11 have that |
| SQL Server | INSERT/UPDATE raw SQL statements can now use the SQLServer specific syntax N'my string value' to support UNICODE character set for columns of type NVARCHAR. Notice tables columns created from the IDE will be of type VARCHAR. So, to support UNICODE you must create/alter your table using embedded SQL or use SQL Server Management Studio.  |
| Job | Variable values can now be returned into the IDE from a job. This is very convenient for debugging and testing jobs that generates a dynamic filenames or other values dynamic values that the next Step to test is using. |
| Job | SharePoint as source: new option to specify how many items to retrieve from the source. This is very convenient for debugging/testing |
| PowerShell | Enhanced longing of PS exception trace:
The log generated when an exception is thrown in the above example: |
| M365 | Azure Migration API – package retry. This feature catches the errors that Azure randomly throws for no reason when transferring data from blob storage to SPO and then performs async retry of the Azure job without having to reupload the binary content and metadata. Wen a retry occurs it is indicated on the AzureJobLog.txt file (Retry) |
| Job notification | You can now run a Step or series of connected Steps to perform job notification. The flow can for example update a database or SP list with notification relevant data. You can invoke your notification Step from any Step in your job also within in a For-Each loop thus enabling individual notification for each iteration in the loop Note that the notification will not be invoked synchronously when there are files pending in Azure (i.e.. files that are uploaded successfully but not yet moved by Azure from temp blob storage to SharePoint). When all files processed within the the Step is accounted for by Azure then the notification is triggered. If an error occurs asynchronously in the Azure migration import API, it is reflected in the notification even though the upload of the file succeeded. The entire Variable context is restored when the notification is triggered asynchronously. The below variables designated for the notification are also populated:
|
| SharePoint | Create Folder/DocumentSet has been redesigned to avoid throttling. This may reduce the performance in terms of number of folders/DocumentSets created per minute in order to comply with the Microsoft throttling policy change. |
| FileManagement | [x] Include subfolders is now visible also on the Home tab |
| Session Log | Dynamic colour change for expanded levels |
| SharePoint | Override URL no longer needs | (pipe) to separate Library from folder structure. This makes it significantly easier to use/understand |
| FileManagement | The FileShare attribute Last Accessed can now be referenced in filters and be used for metadata assignment and calculation. |
| Version 2.04 | March |
| Database | Drop down menu now shows database VIEWS |
| SharePoint | Generic DMS version history migration: x17 performance enhancement for major/minor version calculation. |
| Job Server | New error tracking for Windows Service account permission issues. This makes it easier to figure out what is wrong with a given Windows Service account. |
| UX multi monitor | Step windows and multitasking jobs now open on the same monitor as the parent window. Also the resize logic now use the screen resolution for the monitor in context |
| Data ETL/ForEach | “File and Directory scan” now includes the [LastAccessed] attribute for files. This enables a variety of applications for example only migrating the most recent used files or creating PowerBI models that shows what files are actively being used and what files are ROT (Redundant Obsolete and Trivial). |
| Data ETL/ForEach | “File and Directory scan” Performance optimized for retrieving the first level of folders. (no longer runs through the entire hierarchy) |
| Version 2.01 | February 2022 |
|---|---|
| Data ETL | Edit table New button to edit a database table’s column layout |
| Data ETL | List settings New button to open the SharePoint List settings in a browser |
| File Management | Library settings New button to open the SharePoint Library settings in a browser |
| File Management | Migration of legacy DMS document major/minor version history. Supports Alfresco, OpenText, FileNet or any DMS where the metadata for document versions is stored in a database, CSV or Excel. |
| For-Each | Group By: New functionality to perform data grouping. It is essential for transforming multiple rows into 1 value per. column based on one or more GroupBy columns. Use cases include:
|
| SharePoint | Managed metadata tags in a hierarchy can now be referenced in compliance with the PnP framework using the pipe character | |
| SharePoint Online | File Management: The version for a file can now be specified even the file has no version history. use the new [Set version] column in the metadata assignment. |
| Session log | Auto-refresh of log for running job |
| Session log | Enhanced filtering. Also works real-time on running job. Option to chose to view errors only. |
| Session log | x15 faster purge of outdated log sessions. Deleting specific sessions is done instantly. |
| Session Log | New viewer for analysing embedded reports like SourceFiles.txt This is convenient when running on a machine that does not have Excel installed. Instead of opening the report in Notepad or Excel a new tab opens within the Step where the report can be analysed. The content can be sorted, filtered and exported to Notepad or Excel. |
| Session log | New r-click menu option to view File Version History |
| Version 1.95 | January 2022 |
|---|---|
| Language support | Added support for the following languages:
|
| SharePoint | Handling of columns level validation rules in SharePoint Lists and Libraries. The defined validation message from the actual SharePoint rule is also returned to the session log along with the data value for your context understanding. |
| SharePoint | Enhanced handling of corrupted SharePoint List column definitions. |
| MS SQL Server | Support for having the repository in a SQL Server database rather than locally in the program-public folder. This is useful for compliance with data protection policies and also gives better performance for large session logs. |
| Transformation | New function IndexValue |
| Variables | The expression test value for [data] is now reloaded when the Variable properties window is opened |
| Session Log | Ctrl-A will now select the message text |
| File Management | Optimized file copy operation between FileShares |
| Session Log | Load/Expand 35% faster |
| Version 1.93 | December 2021 |
|---|---|
| Run Job interactively | Columns are now sized automatically to make it easier to view data |
| Validated users | SharePoint: New report window to view all validated users. Useful to se how users are mapped and how they resolve against the AD |
| Public preview | These are the features not yet ready for GA but included for you to preview |
| Version 1.86 | |
|---|---|
| SharePoint Online | “Refresh Schema” is now x2-x80 faster (The bigger Termstore and number of content-types the faster) |
| SharePoint Online | “Connection” has new option to include hidden lists and libraries |
| Database | New option to create a connection to CSV/Excel Sheet directly from the Database connection window. |
| Functions | New system function RemoveIllegalcharactersFromFileName |
| Data Transformation | Test transformation for SharePoint user columns: When transforming values the user mapping list is now included in the test so the actual mapped result is shown. Also, you can edit the mapping list from within the transformation window |
| SharePoint user validation | User name validation can now be performed in “Data Transformation” and “Edit user mapping” |
| Data Transformation | New “Paste value” button for SharePoint columns of type Managed Metadata, Lookup, Content-Type, Choice |
| Job log | New right-click “Open SharePoint version history” in Notepad or Excel” |
| General | Mouse wheel now scrolls the options tab pane on all Steps |
| Version 1.79 | |
|---|---|
| General | Digitally signed executables and Setup |
| FileManagement | New method for excluding folders based on partial string matching on the columns [Path] and [URL path]. For example *forms*;*system* will exclude/include/include+ BOTH files and folders if the path string contains “forms” or “system”. It is coded to work specifically on the column [URL path] for SharePoint source and [Path] for FileShare source. |
| Version 1.77 | |
|---|---|
| DataETL & FileManagement | New Preview filter. When running a job visually with preview you can now filter the content to process based on the sort column. |
| DataETL & FileManagement | SharePoint user validation: The default user can now specify additional domain(s) that will be used for user validation. The syntax app@sharePoint;mydomain.com will try to use mydomain.com applied to a user name if a given user do not validate.
|
| Transformations | Now the value mapping list name is displayed along side the transformation instance to give better overview. Also the mapping list can be opened directly without having to open the transformation. |
| Transformations | New Final result column to show the result of the expression combined with the transformation actions. |
| Job tab | New visual indication on the Job tab when a running job has completed. Job completed successfully etc. |
| Job tab | New button on the Job tab Goto preview job will take you to the running preview session that might be hidden behind other windows. This makes it easier to find visual a running job on the machine when there are many windows open. |
| Version 1.67 | |
|---|---|
| DataETL & ForEach | SQLServer: Enhanced SQL parsing of tables and columns having all sorts of characters in the names. Added support for the UNION keyword without a FROM clause. |
| Job scheduling | New toolbar button added on the Job ribbon tab: Managed scheduled jobs for easy access to the job scheduler. Also the job scheduler opens immediately after a job has been submitted for easy review and status monitoring. |
| Scheduled jobs | New toolbar button to easily select a Step and schedule it to run without first having to open the Step and go to the Job tab. |
| Other changes | |
|---|---|
| DataETL & ForEach | SQL Source: The editor window is now dynamically resized to give more editor space. |
| Reporting | Performance enhancement: Retrieve file migration report is now x3 faster. |
| Data ETL & FileManagement | Support for SharePoint system Libraries SitePages and SiteAssets |
| Session Log | Load time performance enhancement when having >75000 log rows in one session. |
| Variables & Assignments | Visual redesign of “Edit” window and new column Description added. |
| Variables | New “Referenced” list box where all references to the selected Variable are listed. You can se the Step name, Step type and the location where the variable is being used. |
| Variables | Multi-select is now supported for deleting. (Ctrl-A, Ctrl-Click, Ctrl-Shift-Click) |
| Variables | New check box “Show only unreferenced”. Applies a filter so only Variables that are not being used anywhere are listed. This is very useful for cleaning up by deleting unused variables |
| All Steps | New option to specify a Custom log message (Options tab). Useful for adding context specific information to the Session Log for job tracking and analysis. The information is globally searchable across all sessions and all Steps from the Dashboard. |
| All Steps | New option to specify what to do when an error occurs. valid options are:
|
| DataETL & ForEach & FileManagement | New option to specify what to do when no data or files are retrieved from the Source. Valid options are:
|
| Group Step | New hyper link to open embedded Step. |
| Group Step | New condition return value keywords:
|
| Data ETL | SharePoint Item Update: New option to react when not all source keys matches a destination Item. You can trigger a warning, exit the current Group Step or stop the job. This is very useful when your business rules dictate that every source key must match a destination Item. Note that the condition is detected before any action is taken so no data is harmed. |
| Data ETL | SharePoint Item Update: New option to react when multiple items are updated based on a single source key. You can trigger a warning, exit the current Group Step or stop the job. This is very useful when your business rules dictate that only 1 item must be updated for each source key. Note that the condition is detected before any action is taken so no data is harmed. |
| Data ETL & FileManagement | New option to react when an error occurs. You can trigger a warning (the default), exit the current Group Step or stop the job. This is very useful when for example a dynamic URL is wrong and it makes no sense to continue processing the current iteration in a For-Each or the entire job. |
| Session Log | New hyper link to open nested Step. The link has tooltip that shows the storage location of the Step |
| Session Log | Message can now be 8k (previously 2k) and is globally searchable across all sessions and all Steps from the Dashboard. |
| PowerShell | Add-FlowLogRow new arguments to -LogTypes:
|
| Calculation | New functions:
|
| Data ETL | SharePoint: Support for extraction of custom list item version history to database(Insert/Update), Excel, csv, ForEach. You can specify criteria for the extraction: Date of oldest version to include, hard coded or using a variable. When previewing the retrieved SharePoint Item data you have the option to visually also show the version history. (Please note that the version history cannot yet be applied to new SharePoint custom list items. Version history is 100% supported in File Management) |
| File Management | SharePoint version history: You can now specify criteria for the extraction: Date of oldest version to include, hard coded or using a variable. |
| Column mapping | Data-ETL & File Management: New Ribbon bar button to automatically map all columns & reset mappings |
| Documentation | New article SharePoint user mapping. There is a link from Options user mapping to the article within the product. |
| FileManagement | Allow any user name (Only O365) – New validation message to verify that the conditions for this option to function are met |
| FileManagement | The width of Override URL has been extended to 300 characters to allow for long URLs consisting of multiple Variables |
| Data ETL | new link to open CSV folder and Database connection properties |
| Data ETL & FileManagement | New visual indication when a “From” or “To” filter is defined. The indication also serves as a shortcut link to the filter definition |
| File ribbon | Now navigates to the location of the Step in context. |
| SharePoint | Data-ETL & File Management: Emoji support (❤️ Resource for copy/paste Emojis ). Item properties (including folder names) can now contain Emojis. You can even query for items having a specific Emoji. |
| Job Log | New ribbon bar button to open the Dashboard |
| Job Log | Find/Filter: Enhanced visual highlighting and keyboard navigation from the find field. |
| Job Log | New filter option to apply “find criteria” as a filter to limit the rows visible. |
| Job Log | Find and filter criteria is now applied immediately when you type. |
| Job Log | Message column is now permanently expanded but only when clicked. |
| Job Log | New “Open button” to open the Step the current log line belongs to. |
| Data ETL | To SharePoint Document Library: New action Check out will check out files to the account defined in the SharePoint Connection. |
| Data ETL | To SharePoint Document Library: New action Discard check out will remove the checked out status from on files. |
| Data ETL | Preview window: New maximize button to expand the preview Window |
| File Migration | From FileShare: Automatic conversion of file names that are URL encoded. For example My%20file.docx is converted into My file.docx before transfer to SharePoint |
| File Migration | Preview window: New maximize button to expand the preview Window |
| Notification | PowerAutomate and E-Mail: Delayed notification for SharePoint Online. Notification is not sent until all files have been verified by Azure. |
| Notification | PowerAutomate and E-Mail: New field “FileSummary” that shows the number of files Ok, Warning, Error |
| File Migration | Azure/SPO destination. Notification (if enabled) is not triggered until all files for a given step has been processed by Azure. |
| Account | New “Diagnostics” setting to enable logging of the raw data sent to SharePoint and databases. |
| Step | Revised toolbar at top right corner. All commands from TOOLS are now here also. |
| Step | Ctrl+S in all steps does a SAVE |
| Data-ETL | New input type “Variables”. Useful for SQL DDL statements and operation that are entirely based on variables hence no input result set. |
| File Reporting | 3x Faster retrieval of reports. (Revised indexing and SQL WHERE syntax) |
| File Migration | Revised detection of the condition where the computer system is out of memory and many things begin to fail. |
| File Migration | Azure pending files: Optimized visual status for how many files are pending asynchronously in Azure. New article that describes that scenario |
| Migration OnDemand | New article that describes what you can do with the template plus a comparison to MS Mover and MS Migration Manager |
| Data transformation | New UI for data transformations in Variables and column assignments |
| Dashboard | Visual indication of files awaiting Azure Transfer to SharePoint |
| Dashboard | New column Job PID that show the Windows Task Manager PID |
| Expressions | New mechanism to test expressions |
Last modified:
22 April 2024