|
| 1 | +# Integration options for Salesforce |
| 2 | + |
| 3 | +## Deployment Configuration for Salesforce |
| 4 | + |
| 5 | +| Parameter | Meaning | Default Value | Description | |
| 6 | +| ------------------ | ------------- | ---------------------- | ------------------| |
| 7 | +| `SFDC.deployCDC` | Deploy CDC | `true` | Generate CDC processing scripts to run as DAGs in Cloud Composer. See the documentation for different ingestion options for Salesforce. | |
| 8 | +| `SFDC.createMappingViews` | Create mapping views | `true` | The provided DAGs to fetch new records from the Salesforce APIs update records on landing. This value set to **true** will generate views in the CDC processed dataset to surface tables with the "latest version of the truth" from the Raw dataset. If **false** and `SFDC.deployCDC` is `true`, DAGs will be generated with change data capture processing based on SystemModstamp. See details on [CDC processing for Salesforce](./README_SFDC.md#configure-api-integration-and-cdc-for-salesforce). | |
| 9 | +| `SFDC.createPlaceholders` | Create Placeholders | `true` | Create empty placeholder tables in case they are not generated by the ingestion process to allow the downstream reporting deployment to execute without failure. | |
| 10 | +| `SFDC.datasets.raw` | Raw landing dataset | - | Used by the CDC process, this is where the replication tool lands the data from SFDC. If using test data, create an empty dataset. | |
| 11 | +| `SFDC.datasets.cdc` | CDC Processed Dataset | - | Dataset that works as a source for the reporting views, and target for the records processed DAGs. If using test data, create an empty dataset. | |
| 12 | +| `SFDC.datasets.reporting` | Reporting Dataset SFDC| `"REPORTING_SFDC"` | Name of the dataset that is accessible to end users for reporting, where views and user-facing tables are deployed. | |
| 13 | + |
| 14 | + |
| 15 | +## Loading Salesforce data into BigQuery |
| 16 | + |
| 17 | +We provide a replication solution based on Python scripts scheduled in [Apache Airflow](https://airflow.apache.org/) and [Salesforce Bulk API 2.0](https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/bulk_api_2_0.htm). These Python scripts can be adapted and scheduled in your tool of choice. |
| 18 | + |
| 19 | +There are three sets of processing options for data integration: |
| 20 | +- API call and load into Raw datasets, updating existing records if needed |
| 21 | +- Source-to-target structure mapping views |
| 22 | +- CDC processing scripts |
| 23 | + |
| 24 | +If you have datasets already loaded through a different tool in append-always mode, the CDC processing scripts contain mapping files to map the schema of the tables as generated by your tool into the names and data types of the structure required by the reporting views in Cortex Data Foundation. You can also add custom fields in the schema definition so they are incorporated in the CDC processing. |
| 25 | + |
| 26 | +> **Note**: For CDC scripts to work, the **Id** for each API (e.g., `Account Id`) and the [**SystemModStamp**](https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/system_fields.htm) need to be present in the source table. These fields should either have their original name (`Id`, `SystemModstamp`) or be mapped respectively to the `{object_name}` `Id` and `SystemModstamp`. |
| 27 | +> |
| 28 | +> For example, the source table with data of Account object should have original `Id` and `SystemModstamp` fields. If these fields have different names, then `src/SFDC/src/table_schema/accounts.csv` file must be updated with id field's name mapped to `AccountId` and whatever system modification timestamp field mapped to `SystemModstamp`. |
| 29 | +
|
| 30 | +If you already have the replication and CDC working for Salesforce APIs and only need the mapping, you can edit the [mapping files](https://github.com/GoogleCloudPlatform/cortex-salesforce/tree/main/src/table_schema) to generate views that translate the structure generated by the integration tool to the structure expected by Cortex Data Foundation reporting views. |
| 31 | + |
| 32 | +## Salesforce data requirements |
| 33 | +* The structure of the source tables follows *snake_case* naming in plural, i.e., `some_objects`. The columns have the same data types as how Salesforce represents them internally. Some fields have been renamed for better readability in the reporting layer. |
| 34 | +* Any required tables that did not exist within the raw dataset will be created as empty tables during deployment. This is to ensure the CDC deployment step runs correctly. |
| 35 | +* If required, for CDC scripts to work, the **Id** for each API (e.g., `Account Id`) and the [**SystemModStamp**](https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/system_fields.htm) need to be present in the source table. The provided Raw processing scripts fetch these fields automatically from the APIs and update the target replication table. |
| 36 | +* The provided Raw processing scripts do not require additional change data capture processing. This behavior is set during deployment by default. |
| 37 | + |
| 38 | + |
| 39 | +### **Source tables for Currency Conversion in Salesforce** |
| 40 | + |
| 41 | +The currency conversion functionality of Salesforce relies on the existence of the objects `CurrencyTypes` and `DatedConversionRates` within the source Salesforce system, which are available only if [Advanced Currency Management](https://help.salesforce.com/s/articleView?id=sf.administration_about_advanced_currency_management.htm) is enabled. If not, you may want to remove relevant entries from `src/SFDC/config/ingestion_settings.yaml` to avoid running into errors during Salesforce to Raw extraction. |
| 42 | + |
| 43 | +If these tables are not available, we will automatically create empty placeholder tables for them during deployment to avoid break Reporting logic. |
| 44 | + |
| 45 | +## Configure Salesforce integration with Cortex provided ingestion templates</summary> |
| 46 | + |
| 47 | +### Configure API integration and CDC for Salesforce |
| 48 | + |
| 49 | +Following a principle of openness, customers are free to use the provided replication scripts for Salesforce or a data replication tool of their choice as long as data meets the same structure and level of aggregation as provided by the Salesforce APIs. If you are using another tool for replication, this tool can either append updates as new records (_[append always](https://cloud.google.com/bigquery/docs/migration/database-replication-to-bigquery-using-change-data-capture#overview_of_cdc_data_replication)_ pattern) or update existing records with changes when landing the data in BigQuery. If the tool does not update the records and replicates any changes as new records into a target (Raw) table, Cortex Data Foundation provides the option to create change-data-capture processing scripts. |
| 50 | + |
| 51 | +To ensure the names of tables, names of fields, and data types are consistent with the structures expected by Cortex regardless of the replication tool, you can modify the mapping configuration to map your replication tool or existing schemata. This will generate mapping views compatible with the structure expected by Cortex Data Foundation. |
| 52 | + |
| 53 | + |
| 54 | + |
| 55 | +You can use the configuration in [`setting.yaml`](https://github.com/GoogleCloudPlatform/cortex-salesforce/blob/main/config/setting.yaml) to configure the generation of scripts to call the salesforce APIs and replicate the data into the Raw dataset (section `salesforce_to_raw_tables`) and the generation of scripts to process changes incoming into the Raw dataset and into the CDC processed dataset (section `raw_to_cdc_tables`). |
| 56 | + |
| 57 | +By default, the scripts provided to read from APIs update changes into the Raw dataset, so CDC processing scripts are not required, and mapping views to align the source schema to the expected schema are created instead. |
| 58 | + |
| 59 | +The generation of CDC processing scripts is not executed if `SFDC.createMappingViews` in the [config.json](https://github.com/GoogleCloudPlatform/cortex-data-foundation/blob/main/config/config.json#L29) file remains true (default behavior). If CDC scripts are required, set `SFDC.createMappingViews` to false. This second step also allows for mapping between the source schemata into the required schemata as required by Cortex Data Foundation. |
| 60 | + |
| 61 | +The following example of a `setting.yaml` configuration file illustrates the generation of mapping views when a replication tool updates the data directly into the replicated dataset, as illustrated in `option 3` (i.e., no CDC is required, only re-mapping of tables and field names). Since no CDC is required, this option executes as long as the parameter `SFDC.createMappingViews` in the config.json file remains `true`. |
| 62 | + |
| 63 | + |
| 64 | + |
| 65 | +In this example, removing the configuration for a base table or all of them from the sections will skip the generation of DAGs of that base table or the entire section, as illustrated for `salesforce_to_raw_tables`. For this scenario, setting the parameter `deployCDC : False` has the same effect, as no CDC processing scripts need to be generated. |
| 66 | + |
| 67 | +The following example illustrates the mapping of the field `unicornId` as landed by a replication tool to the name and type expected by Cortex Data Foundation, `AccountId` as a `String`. |
| 68 | + |
| 69 | + |
| 70 | + |
| 71 | +Make any changes to the [DAG templates for CDC](https://github.com/GoogleCloudPlatform/cortex-salesforce/tree/main/src/cdc_dag_generator/templates) or for [Raw](https://github.com/GoogleCloudPlatform/cortex-salesforce/tree/main/src/raw_dag_generator/templates) as required by your instance of Airflow or Cloud Composer. You will find more information in the [Appendix - Gathering Cloud Composer settings](./README.md#gathering-cloud-composer-settings). |
| 72 | + |
| 73 | +If you do not need any DAGs for Raw data generation from API calls or CDC processing, set [parameter](#deployment-configuration-for-salesforce) `deployCDC` to `false`. Alternatively, you can remove the contents of the sections in [`setting.yaml`](https://github.com/GoogleCloudPlatform/cortex-salesforce/blob/main/config/setting.yaml). If data structures are known to be consistent with those expected by Cortex Data Foundation, you can skip the generation of mapping views with [parameter](#deployment-configuration-for-salesforce) `SFDC.createMappingViews` set to `false`. |
| 74 | + |
0 commit comments