A full-text search is now available for lineage and all underlying SQL queries. It is a fantastic feature for narrowing down lineage complexity and showing only the relevant parts.
Consider this scenario. Let’s have a table PROFITABILITY populated by 30 queries with 100 tables involved. Ordinary lineage would be very hard to understand as all tables will be visualised at once.
Now let’s apply a full-text search and restrict the profitability lineage for a loan products only. Having the knowledge about your SQL queries you know that you can search for prod_type = ‘loan’. What happens with the lineage now is that all underlying SQL queries will be checked against the search condition. And of course resulting lineage will be displayed for the loan products only.
Another strong use case is when using comments in SQL code. The full-text search is applied to it as well. For example you can further restrict the lineage by a specific developer who made the change or name of the project requesting the change.
Feel free to upload a sample of your ETL and give it a go!