February
Last updated
Last updated
Over the last 6 months, we have been working on platform stability as one of the core focus areas for the Daton platform engineering team. We set out to achieve three overarching goals by end of 2022 –
Flexible pipelines with a high degree of user controls
Identify and reduce all barriers to fast data transfers
Improve visibility and transparency in the data pipeline
Our first major milestone as a part of this plan was the release of an upgraded multi-stage pipeline that was launched on December 21; we covered the details of that upgrade in our earlier update. The following enhancements to the platform are now in production.
Upgraded capability to reduce delay in propagating configuration setups and updates to the warehouse.
Improved visibility into any configuration errors in the UI.
Extension of “Sync Now” capability to also handle configuration changes without disrupting the pipeline.
Benefits
Tables will be configured as soon as the integration is set up, and the first job will start immediately.
Updates to tables will no longer get stuck in the pipelines. This tended to happen when users made changes to the configuration when jobs were in progress.
Visibility into configuration errors, if any.
Users can also use “Sync Now” to push through configurations without needing to wait for the scheduler to kick in.
"Sync Now" enables users to run replication jobs in an integration on-demand. As useful as this feature is, the challenge is that it was implemented at the Integration level. When a user hits “Sync Now”, all the tables in that integration would be replicated again even when the user just wanted one of the tables to be re-synced. This problem would become more glaring when there are 100s of tables in an integration.
With this release, we now support the “Sync Now” functionality at the table level.
Better control on:
data replication, a user who wants to sync only a single table in the integration can now do so without triggering all the tables in the integration.
costs, users can now sync a single table instead of replicating data for other tables when they didn’t need to.
It is common to create integrations to test the service and later delete the integration and start on a clean slate.
However, users couldn’t re-use the deleted integration names.
The functionality to allow the re-use of the integration name after deleting is now made available.
Benefits
Data models built of test integrations can be re-used by dropping and creating integrations with the same names.
Column hashing: A new feature to handle PII data has been released as a core platform capability. The UI will be updated in the upcoming releases to allow users to select columns to hash.
Until the UI changes are made, please reach out to our support team to discuss your column hashing needs and get the integrations set up accordingly.
Improvements in the UI for multi-account selection in the source configuration page
BigQuery Projects list size increased so that the list is visible to add a new destination.
Daton connector- ETA 26 Feb 2022
We understand and appreciate the demands that the data teams are receiving. This results in data delays, putting pressure on those involved in building the data infrastructure.
As a part of our ongoing effort to become a more transparent data pipeline, we plan to launch a new connector, as step 1, to sync job-related data that Daton is generating in your Data warehouse. Future releases will be built on this capability as we work towards achieving our aim of complete transparency.
The development of the Daton connector is ready, but we had to push this release to the last week of Feb 22. There are some prerequisites that we need to handle before exposing this connector to the users that we couldn’t cover in this release.
Enhanced our S3 source and have added support for the following:
Folder and Sub-folder level processing
Regular expression-based file search
If you have files with different types of names in a folder, then you can use REGEX to highlight patterns of files to look for and process data
Added support for gz, zip, rar compressed file processing
Fix for clone integration regex field
Added Turkey as a marketplace.
Added support for the VCPM fields.
Added support for some additional fields.
Added the GetMatchingProduct table.
Fixed the errors in the ListInventorySupply table.
Fixed the issue of not loading data with FBAStorageFeesReport.
Added the 'tagIds' column in Contacts table.
Fixed the missing data in the conversations table.
Fixed the token renewal issues for some accounts.
Fixed the issue of data not loading till the present day.
Fixed the missing data in the sales order table
Fix for missing data in the orders table.
Fixed the issue with data replication.
Error fixed in the Campaign Ads table.
Fixed the intermittent java.lang.Arithmetic exception in campaign ads table.
Daily Exchange Rates (Finance)
Snapchat (Advertising)
Pinterest (Advertising)
Gorgias (Customer Support)
There will be a planned maintenance being performed on 26 Feb.
During this time, the system will not be processing any new jobs.
All the processes will automatically resume once the maintenance is complete.
The exact time of the planned maintenance will be communicated via an email update.
