11.18.2021 Release Notes
about 3 years ago
Here are the release notes for November 18, 2021.
🚀 FEATURES🚀
- Custom Spark Parameters (Driver Size, Number of Executors, etc) for Read Connectors and Write Connectors
✨ ENHANCEMENTS✨
- Enrich lost Spark worker/ failed to connect with pod failure reason or full pod status json
- Disambiguate "Ready" state when Ascend is running Schema Inference (into waiting for resources for schema inference and running schema inference); read more about Schema Inference at https://developer.ascend.io/docs/schema-inference
- Add an "Analyzing Upstream" state for Transforms when they are completing analysis of an upstream Transform. This state would most likely occur downstream of a Legacy Read Connector or a Custom Python Connector
- Utilize a user's default schema for MS SQL Server if the schema field is not set in a connector
- Include all Spark logs from cluster in logs query, not just logs from job-labeled threads; adding these logs can make it easier to debug especially when working with third-party libraries
- Add BigQuery schema management support (Alter, drop+recreate, or error on schema mismatch)
- Ignore failed task statistics from UTD/Running partitions for most calculations, for failure metrics use basis of 1 task rather than sum across all
🔧 BUGFIXES🔧
- Always coerce schema when reading from blob storage to handle a case with Avro and object aggregation creating errors on unioned schemas
- Handle a case of reading/writing from Redshift with a staging directory throwing an error of missing credentials
- Automatically recover from a case of a partition's output data profile getting garbage collected by recomputing the statistic