10.21.2021 Release Notes

Here are the release notes for October 21, 2021.

🚀 FEATURES🚀

• Add a separate component state of waiting “Cluster Capacity” to indicate when component is waiting to process due to insufficient cluster capacity, instead of keeping components in “Ready” state
• Site Admins can view the Spark UI for the Spark cluster that powers the ‘Query’ panel in the Ascend interface

✨ ENHANCEMENTS✨

• [Snowflake Write Connector] Improve speed in cases where the cluster has capacity to process additional Ascend partitions in parallel
• [Snowflake Write Connector] If the desired schema doesn’t exist in the database and the ‘alter’ option is chosen, automatically create the Snowflake schema
• Improve performance of schema inference for Redshift and Snowflake write connections for large Dataframes
• Unblock writing large strings into Redshift by coercing string type in Redshift to varchar(65535) instead of the Spark default varchar(256)
• Add automatic schema management options for the new Redshift Write Connection
• Improve clarity of running partition count while a component is running

🔧 BUGFIXES🔧

• Ensure that Write Connector tasks are always failed if any of the write subtasks fail
• Fix edge case in checking for schema matches in Snowflake and Redshift connections
• Fix MS SQL Test Connection bug where Access check on Alter Table fails incorrectly
• Fix edge case where saving legacy Write Connectors would fail
• Fix edge case where details fields don’t load while editing a connector
• Handle edge cases in S3 Test Read Connection where an exception arises during object read, hiding the actual failure during “Test Connection”
• Fix “Text” data source for Spark jobs running in clustered mode
• Fix MS SQL statistics collection for tables larger than 2B records by using COUNT_BIG instead of COUNT
• Fixed case where not all components showed up as available as inputs in Additional Input Config for transforms
• Fix errors with Object Aggregation where there’s an empty directory at the root of the aggregation
• Reduce cases of Failed to process task: exceeded task submission deadline by increasing timeout