Release 5.0 - Welcome Marketing! 📺

Author: Lsubatin

.gitmodules

@@ -14,3 +14,7 @@
 	path = src/SFDC
 	url = ../cortex-salesforce
 	branch = main
+[submodule "src/marketing"]
+	path = src/marketing
+	url = ../cortex-marketing
+	branch = main

1_click.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+#
+# Copyright 2023 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# 1-Click Deployment Launcher
+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+pushd "${SCRIPT_DIR}" 1> /dev/null
+echo -n "Please wait..."
+git submodule update --init --recursive &> /dev/null || true
+echo -e -n "\r                \r"
+src/utils/interactive/interactive.sh "${1}" "${2}"
+popd 1> /dev/null

README.md

@@ -1,11 +1,37 @@
+## July 2023 - Release 5.0
+*   **New Marketing models:** A new repository, [Cortex for Marketing](https://github.com/GoogleCloudPlatform/cortex-marketing), has been added to the Data Foundation. This repository starts with data ingestion and data processing DAGs for Cloud Composer and Dataflow and predefined data models for Google Ads and Campaign Manager 360. This accelerates Ads reporting scenarios like keyword performance insights across campaigns and audience insights across display campaigns directly in BigQuery. Please check the [ERDs in the docs](https://github.com/GoogleCloudPlatform/cortex-data-foundation/tree/main/docs) folder.
+*   **Quick demo deployment**: For those looking for a frictionless demo deployment experience, we have created a button that will guide them through an automated process to create sample datasets with test data and enable APIs and permissions. This is available in the [README](https://github.com/GoogleCloudPlatform/cortex-data-foundation/tree/main#quick-demo-setup).
+*   **Cross-workload and source reusable models** (a.k.a, **K9**, where the DAGs 🐶 live): Reusable models, such as time dimensions or external sources like Weather, are now available through a deployment mechanism that is shared across all datasets. This allows for cross workload reporting, like joining SAP and Google Ads data too. Some DAGs that used to be deployed as workload-specific, like holiday calendar to SAP, is not migrated to the dataset Please check the migration guide in [the docs folder](https://github.com/GoogleCloudPlatform/cortex-data-foundation/tree/main/docs/external_dag_migration). Weather and Trends extraction DAGs are now disabled by default. More information about the K9 module in the [README](https://github.com/GoogleCloudPlatform/cortex-data-foundation/tree/main#configure-k9-deployments).
+*   [Optional materialization and performance optimization](https://github.com/GoogleCloudPlatform/cortex-data-foundation/tree/main#performance-optimization-for-reporting-views) functionality is enabled for all modules and views. This impacts the SQL templates previously containing a `CREATE` statement. 🚨🚨 We strongly recommend checking the new default configurations as this will attempt to replace views with tables. 🚨🚨 You will need to delete the existing views in Reporting before you can deploy a table with the same name. See the documentation for more details.
+*   [CATGAP](https://github.com/GoogleCloudPlatform/cortex-data-foundation/tree/main/src/k9/src/catgap) has moved to K9. By default, the deployment is disabled.
+*   The test harness data has now moved from buckets into BigQuery datasets available in all relevant regions. Test harness data is still not provided with any warranty of quality, but this change simplifies and speeds up the deployment for those exploring the framework.
+*   Substitutions are removed, except for Logs GCS bucket. Configuration files are now normalized into `config.json`. Legacy Dotenv files have been removed. Check [the updated documentation](https://github.com/GoogleCloudPlatform/cortex-data-foundation/tree/main#configure-deployment) to understand parameters.
+    *   deployCDC is now specific to each workload
+    *   generateExtraData is now removed, the control of this deployment is owned by K9.
+### **SAP**
+- SAP-specific DAGs, like currency conversion, inventory management or hierarchy reader, were moved to the reporting module and are executed by default. This prevents downstream errors that used to happen when _GEN_EXT was flagged false.
+- Please check the [reporting configuration for materialization](https://github.com/GoogleCloudPlatform/cortex-reporting/blob/main/reporting_settings_ecc.yaml), these have been modified to better match customer expectations but may not apply to your business at all.
+*   UNION deployment logic has been removed, but supported views will continue to have the UNION template until further notice.
+- The reporting module relies on the time dimension deployment in K9, it can be executed manually if deploying from the [cortex-reporting submodule](https://github.com/GoogleCloudPlatform/cortex-reporting/).
+
+### **Salesforce**
+- New currency conversion for Salesforce. Since these are optional components for Salesforce deployments, they can be created through the placeholder settings. If you don't have the currency types and conversion in Salesforce, comment out or remove the calls in `SFDC/config/ingestion_settings.yaml`. The configuration in `cdc_placeholder_settings.yaml` will take care of the dependencies in the reporting views.
+- The reporting module relies on the time dimension deployment in K9, it can be executed manually if deploying from the [Salesforce submodule](https://github.com/GoogleCloudPlatform/cortex-salesforce).
+
+### Known issues and limitations
+-   SAP Reporting and Salesforce Reporting requires execution of K9 pre-processing if used as a separate submodule.
+-   ML Models repository still allows for substitutions.
+-   CM360 and Ads integration scripts are in `Experimental` mode while we gather early feedback from customers on usage. We would like to hear what you think through GitHub issues.
+-   Google Ads DAG is supported on Airflow v2 only.
+
 ## March 2023 - Release 4.2
 *   **New Supply Chain and Finance views**: These views support KPIs for Inventory management, Vendor Performance, Accounts Payable and Spend Analysis. Please check the updated [ECC ERD](https://github.com/GoogleCloudPlatform/cortex-data-foundation/blob/main/docs/erd_ecc.pdf) and the [S/4 ERD](https://github.com/GoogleCloudPlatform/cortex-data-foundation/blob/main/docs/erd_s4.pdf) The new objects are:
     -  **Reusable DAG scripts for inventory snapshots**: ([SAP_CDC/external_dag/inventory_snapshots](https://github.com/GoogleCloudPlatform/cortex-dag-generator/tree/main/src/external_dag/inventory_snapshots)): Initializer and periodic update for weekly and monthly inventory snapshots and aggregations, slow moving threshold, and stock characteristic configuration. Please check for CORTEX-CUSTOMER comments for potential updates needed for custom thresholds and material movements. These scripts are in PREVIEW and will be moved into another structure in deployment in the next major release. ⚠️⚠️ These scripts fill the base tables and need to be scheduled for the reporting views to function properly.⚠️⚠️
     -   **New views for reporting**: AccountsPayableOverview, DaysPayableOutstanding, InventoryByPlant, InventoryKeyMetrics, MaterialLedger, MaterialMovementTypesMD
     MaterialsBatchMD, POScheduleLine, PurchaseDocumentsHistory, SalesOrderStatus, SlowMovingThreshold, StockCharacteristicsConfig, StockInHand, StockMonthlySnapshots, StockWeeklySnapshots.
     -   **Overview views**: VendorLeadTimeOverview, VendorPerformanceOverview. These views show the reporting logic used in Looker in case you want to replicate them in another tool or a microservice. These views are not deployed by default.
 *   **Materializer PREVIEW** 🫶: By default, the new views that will require a lot of computation are now deployed as materializing DAGs. This helps improve performance and reduce costs and is configurable. This configuration is optional and the generated SQL can be ported into a scheduler of choice if you are not using Cloud Composer or Airflow. The next major release will follow this deployment approach for all views. See the [documentation](https://github.com/GoogleCloudPlatform/cortex-data-foundation/blob/main/README.md#optional-sap-only_performance-optimization-for-reporting-views) for more details.
-*   **Cortex Analytics Templates - Google Ads Pipelines (CATGAP)** 🐈: This new experimental feature uses Natural Language Processing machine learning models to intelligently map product categories from Google Ads to SAP's product hierarchy. We'd love to know what you think. CATGAP is not deployed by default. Please check [the documentation](https://github.com/GoogleCloudPlatform/cortex-dag-generator/tree/main/src/external_dag/catgap/README.md) for details and further setup.
+*   **Cortex Analytics Templates - Google Ads Pipelines (CATGAP)** 🐈: This new experimental feature uses Natural Language Processing machine learning models to intelligently map product categories from Google Ads to SAP's product hierarchy. We'd love to know what you think. CATGAP is not deployed by default.Please check [the documentation](https://github.com/GoogleCloudPlatform/cortex-dag-generator/tree/main/src/external_dag/catgap/README.md) for details and further setup.
 *   ⚠️⚠️NOTE⚠️⚠️ Reporting views that expect parameters from currencies in config/config.json will produce the same result as many times as currencies are set as targets. Currency conversion in newer views is no longer commented out for convenience. However, the target currency needs to be passed as a filter from the reporting view. 🙏🙏 Please check for `CORTEX-CUSTOMER` comments for specific guidance if you deploy the data foundation with more than one currency.🙏🙏
 *   Parameters that control runtime (e.g, "DEPLOY_SAP", "DEPLOY_SFDC", "DEPLOY_CDC" and "TEST_DATA") are now also read from the config file. These are still defaulted in cloudbuild.yaml substitutions. If you want to use the values in the file, the [substitutions section in cloudbuild.yaml](https://github.com/GoogleCloudPlatform/cortex-data-foundation/blob/main/cloudbuild.yaml#L107) needs to be commented out. Substitutions from the command line will be phased out.
 *   Compatibility for Airflow v1 and v2 updated for currency_conversion DAG.
@@ -36,7 +62,7 @@
 *   Salesforce integration DAGs have been tested in Airflow 1.0. Airflow 2.0 may require library updates or use of backwards compatible libraries to be tested and confirmed in the next release.
 *   Finance views for SAP are good candidates for partial or total materialization. Check BigQuery's execution details to identify opportunities to create materialization processes and further optimizations that fit your data best.
 ## November 2022 - Release 3.1
-*   **New partitioning and clustering configuration for CDC deployment:** Configurable partitioning and clustering on deployment of CDC landing tables and scripts. See example in [setting.yaml](https://github.com/GoogleCloudPlatform/cortex-dag-generator/blob/main/setting.yaml) and [the README instructions](https://github.com/GoogleCloudPlatform/cortex-data-foundation#performance-optimization-for-cdc-tables).
+*   **New partitioning and clustering configuration for CDC deployment:** Configurable partitioning and clustering on deployment of CDC landing tables and scripts. See example in [cdc_settings.yaml](https://github.com/GoogleCloudPlatform/cortex-dag-generator/blob/main/cdc_settings.yaml) and [the README instructions](https://github.com/GoogleCloudPlatform/cortex-data-foundation#performance-optimization-for-cdc-tables).
 *   **New date dimension 📅** generated from external DAGs through `_GEN_EXT=true` to allow for more flexibility in reporting. This table has been incorporated into views POSchedule, MaterialsValuation, PurchaseDocuments, Deliveries, Billing, SalesOrderScheduleLine, AccountingDocuments, InvoiceDocuments_Flow, POOrderHistory and SalesOrders_V2. The straucture of the table will be generated without data if `_GEN_EXT=false`.
   **Note:** If you do not want to execute the generation of other DAGs for external sources, remove them from the list in the [generation script](https://github.com/GoogleCloudPlatform/cortex-dag-generator/blob/main/generate_external_dags.sh#L4).
 *   Performance improvements to AccountingDocuments, InvoiceDocuments_Flow,Billing, Deliveries, MaterialsValuation, PurchaseDocuments and POOrderHistory, SalesOrders_V2 when using currency conversion generated from DAG materializing results (see CORTEX-CUSTOMER tags and differences when merging).
@@ -136,7 +162,7 @@ This release brings the following changes Cortex Data Foundation.
 
 
 **Bug Fixes:**
-* Fix for Runtime views not generated from `setting.yaml`
+* Fix for Runtime views not generated from `settings.yaml`
 * Deployment dependencies causing errors on view creation