Data Mesh Architecture

[next-mdx-remote] error compiling MDX: Expected the closing tag `</span>` either after the end of `emphasis` (107:1157) or another opening tag after the start of `emphasis` (107:667) 105 | From an implementation perspective, these preprocessing steps can be implemented as simple SQL views that project the raw data. The queries may be organized through [common table expressions](https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax#with_clause)  (CTEs) and may be enhanced with [user-defined functions](https://cloud.google.com/bigquery/docs/reference/standard-sql/user-defined-functions)  (UDFs), e.g., for JSON processing. As an alternative, the cleaning steps can be implemented as lambda functions that operate on topics. More complex pipelines can be built with frameworks like [dbt](https://www.getdbt.com/)  or [Apache Beam](https://beam.apache.org/)  that offer an advanced programming model, but also require more skills to master. 106 | > 107 | <table data-tab-size="8" data-paste-markdown-skip="" data-tagsearch-lang="SQL" data-tagsearch-path="entities__inventory_history.sql"><tbody><tr><td id="file-entities__inventory_history-sql-L1" data-line-number="1"></td><td id="file-entities__inventory_history-sql-LC1"><span><span>--</span> Step 1: Deduplicate</span></td></tr><tr><td id="file-entities__inventory_history-sql-L2" data-line-number="2"></td><td id="file-entities__inventory_history-sql-LC2">WITH inventory_deduplicated <span>AS</span> (</td></tr><tr><td id="file-entities__inventory_history-sql-L3" data-line-number="3"></td><td id="file-entities__inventory_history-sql-LC3"><span>SELECT</span> <span>*</span></td></tr><tr><td id="file-entities__inventory_history-sql-L4" data-line-number="4"></td><td id="file-entities__inventory_history-sql-LC4">EXCEPT (row_number)</td></tr><tr><td id="file-entities__inventory_history-sql-L5" data-line-number="5"></td><td id="file-entities__inventory_history-sql-LC5"><span>FROM</span> (</td></tr><tr><td id="file-entities__inventory_history-sql-L6" data-line-number="6"></td><td id="file-entities__inventory_history-sql-LC6"><span>SELECT</span> <span>*</span>,</td></tr><tr><td id="file-entities__inventory_history-sql-L7" data-line-number="7"></td><td id="file-entities__inventory_history-sql-LC7">ROW_NUMBER() OVER (PARTITION BY id <span>ORDER BY</span> <span>time</span> <span>DESC</span>) row_number</td></tr><tr><td id="file-entities__inventory_history-sql-L8" data-line-number="8"></td><td id="file-entities__inventory_history-sql-LC8"><span>FROM</span> <span><span>`</span>datameshexample-fulfillment.raw.inventory<span>`</span></span>)</td></tr><tr><td id="file-entities__inventory_history-sql-L9" data-line-number="9"></td><td id="file-entities__inventory_history-sql-LC9"><span>WHERE</span> row_number <span>=</span> <span>1</span></td></tr><tr><td id="file-entities__inventory_history-sql-L10" data-line-number="10"></td><td id="file-entities__inventory_history-sql-LC10">),</td></tr><tr><td id="file-entities__inventory_history-sql-L11" data-line-number="11"></td><td id="file-entities__inventory_history-sql-LC11"><span><span>--</span> Step 2: Parse JSON to columns</span></td></tr><tr><td id="file-entities__inventory_history-sql-L12" data-line-number="12"></td><td id="file-entities__inventory_history-sql-LC12">inventory_parsed <span>AS</span> (</td></tr><tr><td id="file-entities__inventory_history-sql-L13" data-line-number="13"></td><td id="file-entities__inventory_history-sql-LC13"><span>SELECT</span></td></tr><tr><td id="file-entities__inventory_history-sql-L14" data-line-number="14"></td><td id="file-entities__inventory_history-sql-LC14">json_value(data, <span><span>"</span>$.sku<span>"</span></span>) <span>AS</span> sku,</td></tr><tr><td id="file-entities__inventory_history-sql-L15" data-line-number="15"></td><td id="file-entities__inventory_history-sql-LC15">json_value(data, <span><span>"</span>$.location<span>"</span></span>) <span>AS</span> location,</td></tr><tr><td id="file-entities__inventory_history-sql-L16" data-line-number="16"></td><td id="file-entities__inventory_history-sql-LC16">CAST(json_value(data, <span><span>"</span>$.available<span>"</span></span>) <span>AS</span> int64) <span>AS</span> available,</td></tr><tr><td id="file-entities__inventory_history-sql-L17" data-line-number="17"></td><td id="file-entities__inventory_history-sql-LC17">CAST(json_value(data, <span><span>"</span>$.updated_at<span>"</span></span>) <span>AS</span> <span>timestamp</span>) <span>AS</span> updated_at,</td></tr><tr><td id="file-entities__inventory_history-sql-L18" data-line-number="18"></td><td id="file-entities__inventory_history-sql-LC18"><span>FROM</span> inventory_deduplicated</td></tr><tr><td id="file-entities__inventory_history-sql-L19" data-line-number="19"></td><td id="file-entities__inventory_history-sql-LC19">)</td></tr><tr><td id="file-entities__inventory_history-sql-L20" data-line-number="20"></td><td id="file-entities__inventory_history-sql-LC20"><span><span>--</span> Step 3: Actual Query</span></td></tr><tr><td id="file-entities__inventory_history-sql-L21" data-line-number="21"></td><td id="file-entities__inventory_history-sql-LC21"><span>SELECT</span> sku, location, available, updated_at</td></tr><tr><td id="file-entities__inventory_history-sql-L22" data-line-number="22"></td><td id="file-entities__inventory_history-sql-LC22"><span>FROM</span> inventory_parsed</td></tr><tr><td id="file-entities__inventory_history-sql-L23" data-line-number="23"></td><td id="file-entities__inventory_history-sql-LC23"><span>ORDER BY</span> sku, location, updated_at</td></tr></tbody></table> | ^ 108 | 109 | #### Analytics[](https://www.datamesh-architecture.com/#analytics) More information: https://mdxjs.com/docs/troubleshooting-mdx