Release/0.3.0 (#144)

* Feature/sfe 3067 proxy (#140)

* Fixed data type (#132)

Co-authored-by: arunpamulapati <arunpamulapati>

* Updated the SBP content to make sure the checks are clearly noted with mutual exclusive nature (#133)

Co-authored-by: arunpamulapati <arunpamulapati>

---------

Co-authored-by: Arun Pamulapati <[email protected]>

* Sfe 3029 remove dependency on pat dashboards (#141)

* Fixed data type (#132)

Co-authored-by: arunpamulapati <arunpamulapati>

* Updated the SBP content to make sure the checks are clearly noted with mutual exclusive nature (#133)

Co-authored-by: arunpamulapati <arunpamulapati>

* Removed dependency on PAT token

* Added proxies to config

---------

Co-authored-by: Arun Pamulapati <[email protected]>
Co-authored-by: arunpamulapati <arunpamulapati>

* Feature/sfe 3028 configure schema through setup (#142)

* Made schema configurable

* fixed catalog name doc

---------

Co-authored-by: arunpamulapati <arunpamulapati>

* Update SAT configuration to include proxy support DABs, WIP TF

* Update TF code

* Add proxy support to SAT configuration

* delete previous initialize.py

* added condition to check if database already has  security_best_practices

* update SAT configuration to include catalog and security analysis schema

* Change proxy type for TF

* removed comments cell.

* Updated to not use PAT for testing

* removed PAT in the print

* Updated TF to make sure we have analysis_schema_name working

* updated to not use PAT for testing

* Update README.md to fix borken terrform links

* Add analysis_schema_name variable to GCP provider configuration

* moved away from PATs for GCP to use temp tokens

* Made GCP TF  to go with the current code

* Update setup documentation and remove PAT references

* Update setup.md

added note about SAT install  rule for a given account and subscription.

* Updated best practices file

* Update setup.md

Added dashboard new location for the legacy dashboards.

* Update setup documentation to include support for Unity Catalog and proxies

* Update security_best_practices.csv

merged from main

---------

Co-authored-by: David Leiva <[email protected]>
Co-authored-by: arunpamulapati <arunpamulapati>

Author: arunpamulapati

README.md

@@ -9,7 +9,10 @@
 
 Refer to specific use-case:
 - [Standard setup guide](./docs/setup.md)
-- [Terraform](./terraform/README.md)
+- Terraform
+   - [AWS](./terraform/aws/TERRAFORM_AWS.md)
+   - [Azure](./terraform/azure/TERRAFORM_Azure.md)
+   - [GCP](./terraform/gcp/TERRAFORM_GCP.md)
 - [Deprecated: Manual setup](./docs/deprecated_old_setup.md)
 
 ## Introduction

configs/security_best_practices.csv

@@ -43,13 +43,37 @@ def form():
             ignore=lambda x: not x["enable_uc"],
             default="hive_metastore",
         ),
+        Text(
+            name="security_analysis_schema",
+            message="Schema name for SAT",
+            default="security_analysis",
+        ),
         List(
             name="warehouse",
             message="Select warehouse",
             choices=loading(get_warehouses, client=client),
         ),
     ]
-    questions = questions + cloud_specific_questions(client)
+    proxies = [
+        Confirm(
+            name="use_proxy",
+            message="Want to use a proxy?",
+            default=False,
+        ),
+        Text(
+            name="http",
+            message="HTTP Proxy",
+            ignore=lambda x: not x["use_proxy"],
+            default="",
+        ),
+        Text(
+            name="https",
+            message="HTTPS Proxy",
+            ignore=lambda x: not x["use_proxy"],
+            default="",
+        ),
+    ]
+    questions = questions + cloud_specific_questions(client) + proxies
     return client, prompt(questions), profile
 
 
@@ -116,15 +140,6 @@ def generate_secrets(client: WorkspaceClient, answers: dict, cloud_type: str):
 
     client.secrets.create_scope(scope_name)
 
-    token = client.tokens.create(
-        lifetime_seconds=86400 * 90,
-        comment="Security Analysis Tool",
-    )
-    client.secrets.put_secret(
-        scope=scope_name,
-        key=f"sat-token-{client.get_workspace_id()}",
-        string_value=token.token_value,
-    )
     client.secrets.put_secret(
         scope=scope_name,
         key="account-console-id",
@@ -135,6 +150,29 @@ def generate_secrets(client: WorkspaceClient, answers: dict, cloud_type: str):
         key="sql-warehouse-id",
         string_value=answers["warehouse"]["id"],
     )
+    client.secrets.put_secret(
+        scope=scope_name,
+        key="analysis_schema_name",
+        string_value=f'{answers["catalog"]}.{answers["security_analysis_schema"]}',
+    )
+
+    if answers["use_proxy"]:
+        client.secrets.put_secret(
+            scope=scope_name,
+            key="proxies",
+            string_value=json.dumps(
+                {
+                    "http": answers["http"],
+                    "https": answers["https"],
+                }
+            ),
+        )
+    else:
+        client.secrets.put_secret(
+            scope=scope_name,
+            key="proxies",
+            string_value="{}",
+        )
 
     if cloud_type == "aws":
         client.secrets.put_secret(

dabs/dabs_template/initialize.py.tmpl

@@ -6,8 +6,6 @@ config_file=$3
 
 
 cp -r ../configs ../notebooks ../dashboards ./dabs_template/template/tmp
-rm ./dabs_template/template/tmp/notebooks/Utils/initialize.py
-cp ./dabs_template/initialize.py.tmpl ./dabs_template/template/tmp/notebooks/Utils/initialize.py.tmpl
 
 databricks bundle init ./dabs_template -p $profile --config-file $config_file
 rm -rf $config_file

dabs/sat/config.py

@@ -206,6 +206,7 @@ Please gather the following information before you start setting up:
      * Set the PAT token value for the workspace_id 
      * Set the value for the account_id 
      * Set the value for the sql_warehouse_id
+     * Set the value for the analysis_schema_name to store SAT analysis results (for UC use catalog.schema name , for hive use schema name)
     
 
        ```
@@ -219,6 +220,10 @@ Please gather the following information before you start setting up:
        ```
        databricks --profile e2-sat secrets put-secret sat_scope sql-warehouse-id
        ```  
+        
+       ```
+       databricks --profile e2-sat secrets put-secret sat_scope analysis_schema_name 
+       ```  
     
 
    * In your environment where you imported SAT project from git (Refer to Step 4 in Prerequisites) Open the \<SATProject\>/notebooks/Utils/initialize notebook and modify the JSON string with :  
@@ -230,7 +235,8 @@ Please gather the following information before you start setting up:
          ```
               {
                  "account_id": dbutils.secrets.get(scope="sat_scope", key="account-console-id"),   
-                 "sql_warehouse_id": dbutils.secrets.get(scope="sat_scope", key="sql-warehouse-id")
+                 "sql_warehouse_id": dbutils.secrets.get(scope="sat_scope", key="sql-warehouse-id"),
+                 "analysis_schema_name": dbutils.secrets.get(scope="sat_scope", key="analysis_schema_name"),
                  "verbosity":"info"
               }
 

dabs/setup.sh

@@ -1,5 +1,7 @@
 # Setup Guide
 
+> **SAT v0.2.0 or higher** brings full support for Unity Catalog. Now you can pick your catalog instead of hive_metastore. Plus, you get to choose your own schema name.
+
 Follow this guide to setup the Security Analysis Tool (SAT) on your Databricks workspace.
 
 ## Prerequisites
@@ -27,12 +29,13 @@ SAT creates a new security_analysis database and Delta tables. If you are an exi
 ### Unity Catalog based schema
 
 ```sql
-  drop  database <uc_catalog_name>.security_analysis cascade;
+  drop  database <uc_catalog_name>.<schema_name> cascade;
 ```
 
 ## Setup
 
-> SAT is a productivity tool to help verify security configurations of Databricks deployments, it's not meant to be used as certification or attestation of your deployments. SAT project is regularly updated to improve the correctness of checks, add new checks, and fix bugs. Please send your feedback and comments to [email protected]. 
+> SAT is a productivity tool to help verify security configurations of Databricks deployments, it's not meant to be used as certification or attestation of your deployments. SAT project is regularly updated to improve the correctness of checks, add new checks, and fix bugs. You will need a single SAT install per Databricks account in AWS and GCP and a single install per azure subscription in Azure. Add the Service principle as mentioned in the detailed steps to analyze the rest of the workspaces from the workspace where SAT is installed. You can choose not to add SP to a given workspace if you wish to ignore a given workspace.
+> Please send your feedback and comments to [email protected]. 
 
 SAT can be setup on any of the cloud providers where Databricks is hosted. Follow the setup guide for the cloud provider you are using:
 
@@ -70,10 +73,10 @@ You now have two jobs (SAT Initializer Notebook & SAT Driver Notebook). Run SAT
 
 ###  2. Access Databricks SQL Dashboards
 
-   > **Note:** You can also use Lakeview Dashboards to view the results, instead of classic Dashboards.
+   > **Note:** You can also use Lakeview Dashboards to view the results.
 
 
-In DBSQL find "SAT - Security Analysis Tool" dashboard  to see the report. You can filter the dashboard by **SAT** tag. 
+In DBSQL find "SAT - Security Analysis Tool" dashboard  to see the report. You can filter the dashboard by **SAT** tag.  (The old classic legacy dashboard can be found in Workspace -> Home -> SAT_dashboard)
 
    <img src="./images/sat_dashboard_loc.png" width="70%" height="70%">
 

docs/deprecated_old_setup.md

@@ -30,9 +30,9 @@ The first step is to create a Service Principal in Databricks. This will allow S
 - Save the `Secret` and `Client ID`
 - To deploy SAT in a workspace, you must add the Service Principal to the workspace.
 
-![AWS_SP_Workspace](../images/aws_ws.png)
+![AWS_SP_Workspace](../images/gcp_ws.png)
 
-> The Service Principle requires an [Accounts Admin role](https://docs.databricks.com/en/admin/users-groups/service-principals.html#assign-account-admin-roles-to-a-service-principal), [Admin role](https://docs.databricks.com/en/admin/users-groups/service-principals.html#assign-a-service-principal-to-a-workspace-using-the-account-console) for **each workspace** and needs to be a member of the [metastore admin group](https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/admin-privileges.html#who-has-metastore-admin-privileges) is required to analyze many of the APIs
+> The Service Principle requires an [Accounts Admin role](https://docs.gcp.databricks.com/en/admin/users-groups/service-principals.html#assign-account-admin-roles-to-a-service-principal), [Admin role](https://docs.gcp.databricks.com/en/admin/users-groups/service-principals.html#assign-a-service-principal-to-a-workspace-using-the-account-console) for **each workspace** and needs to be a member of the [metastore admin group](https://docs.gcp.databricks.com/en/data-governance/unity-catalog/manage-privileges/admin-privileges.html#who-has-metastore-admin-privileges) is required to analyze many of the APIs
 
 ## Installation
 
@@ -61,6 +61,7 @@ To execute the SAT follow these steps on your workstation or a compatible VM tha
       ./install.sh
     ```
 
+> **Proxies are now supported as part of SAT. You can add your HTTP and HTTPS links to use your proxies.**
 
 ![](../gif/terminal-aws.gif)
 

docs/images/gcp_ws.png

@@ -16,7 +16,7 @@ We created diagnosis notebooks for respective clouds to help troubleshoot your S
 * [SAT Azure troubleshooting notebook](https://github.com/databricks-industry-solutions/security-analysis-tool/blob/main/notebooks/diagnosis/sat_diagnosis_azure.py)
 * [SAT GCP troubleshooting notebook](https://github.com/databricks-industry-solutions/security-analysis-tool/blob/main/notebooks/diagnosis/sat_diagnosis_gcp.py)
 
-### 1. Incorrectly configured secrets
+### Incorrectly configured secrets
 
 * **Error:**
 
@@ -31,19 +31,7 @@ We created diagnosis notebooks for respective clouds to help troubleshoot your S
     databricks --profile e2-sat secrets list-secrets sat_scope
     ```
 
-### 2. Invalid access token
-
-* **Error:**
-
-    ```
-    Error 403 Invalid access token.
-    ```
-
-* **Resolution:**
-
-    Check your PAT token configuration for the `workspace_pat_token` key.
-
-### 3. Firewall blocking Databricks accounts console
+### Firewall blocking Databricks accounts console
 
 * **Error:**
 
@@ -86,7 +74,7 @@ We created diagnosis notebooks for respective clouds to help troubleshoot your S
 
     If you don’t see a JSON with a clean listing of workspaces, you are likely having a firewall issue that is blocking calls to the accounts console. Please have your infrastructure team add `accounts.cloud.databricks.com` to the allow-list. Ensure that the private IPv4 address from the NAT gateway is added to the IP allow list.
 
-### 4. Offline install of libraries in case of no PyPI access
+### Offline install of libraries in case of no PyPI access
 
 * **Steps:**