#2 Adds support for JSON files

Author: bytebutcher

README.md

@@ -10,87 +10,74 @@
 </div>
 <br>
 
-Wireshark-like display filter for python dictionaries.
+A Wireshark-like display filter for Python dictionaries. This tool allows you to easily filter, analyze, and 
+manipulate data in Python dictionaries. It offers a range of features including comparison operators, 
+combining operators, membership operators, and more. 
+
+## Table of Contents
+1. [Quick Start](#quick-start)
+2. [Usage](#usage)
+3. [Features](#features)
+4. [Examples](#examples)
+5. [Acknowledgements](#acknowledgements)
+
+## Quick Start
+
+Here's a simple example to get you started. First, install the package:
 
-## Setup
 ```commandline
 pip3 install python-dict-display-filter
 ```
 
-## Usage
+Then, use it to filter a list of dictionaries:
+```
+from pydictdisplayfilter import DictDisplayFilter
+actors = [
+    {"name": ["Laurence", "Fishburne"], "age": {"born": "1961"}, "gender": "male"},
+    {"name": ["Keanu", "Reeves"], "age": {"born": "1964"}, "gender": "male", "power": ["flight", "bullet-time"]},
+    {"name": ["Joe", "Pantoliano"], "age": {"born": "1951"}, "gender": "male"},
+    {"name": ["Carrie-Anne", "Moss"], "age": {"born": "1967"}, "gender": "female"}
+]
+ddf = DictDisplayFilter(actors)
 
-The basics and the syntax of the display filter are described in the 
-<a href="https://github.com/bytebutcher/python-dict-display-filter/blob/main/docs/USER_GUIDE.md">User Guide</a>.
+# This will filter the list to show only male actors born between 1960 and 1965 whose names end with 'e'
+filtered_actors = ddf.filter("gender == male and (age.born > 1960 and age.born < 1965) and name matches .*e$")
 
-If you want to see some advanced examples of how ```python-dict-display-filter``` can be put to use checkout the 
-<a href="https://github.com/bytebutcher/python-dict-display-filter/blob/main/docs/EXAMPLES.md">Examples</a>.
+print(list(filtered_actors))
+[{'name': ['Laurence', 'Fishburne'], 'age': {'born': '1961'}, 'gender': 'male'}]
+```
 
-If you want to use ```python-dict-display-filter``` in your own application and customize it to your needs 
-check out the 
-<a href="https://github.com/bytebutcher/python-dict-display-filter/blob/main/docs/DEVELOPER_GUIDE.md">Developer Guide</a>.
+For more details, please refer to the 
+<a href="https://github.com/bytebutcher/python-dict-display-filter/blob/main/docs/USER_GUIDE.md">User Guide</a>.
 
 ## Features
-The following overview shows all the supported features of the display filter:
+
+Python Dictionary Display Filter supports a wide range of features, including:
 * **Comparison Operators:** ```==```, ```!=```, ```<=```, ```<```, ```>=```, ```>```, ```~=```, ```~```, ```&```
 * **Combining Operators:** ```and```, ```or```, ```xor```, ```not``` 
 * **Membership Operators:** ```in```
 * **Types:** ```Text```, ```Number```, ```Date & Time```, ```Ethernet-```, ```IPv4-```, ```IPv6-Address```
 * **Slicing:** ```Text```, ```Ethernet-```, ```IPv4-```, ```IPv6-Address```
 * **Functions:** ```upper```, ```lower```, ```len```
+* **Data Sources**: ```CSV```, ```Dictionaries```, ```JSON```, ```SQLite```
 
 For a detailed description of the individual features check out the
 <a href="https://github.com/bytebutcher/python-dict-display-filter/blob/main/docs/USER_GUIDE.md">User Guide</a>.
 
 ## Examples 
 
-Initialize ```DictDisplayFilter``` with a list of dictionaries:
-```
-> from pydictdisplayfilter import DictDisplayFilter
-> actors = [
-    {"name": ["Laurence", "Fishburne"], "age": {"born": "1961"}, "gender": "male"},
-    {"name": ["Keanu", "Reeves"], "age": {"born": "1964"}, "gender": "male", "power": ["flight", "bullet-time"]},
-    {"name": ["Joe", "Pantoliano"], "age": {"born": "1951"}, "gender": "male"},
-    {"name": ["Carrie-Anne", "Moss"], "age": {"born": "1967"}, "gender": "female"}
-]
-> ddf = DictDisplayFilter(actors)
-```
-
-Show only actors with some kind of super-power:
-```
-> ddf.filter("power")
-```
-
-Show only actors which were born before 1965:
-```
-> ddf.filter("age.born < 1965")
-```
-
-Show only female actors:
-```
-> ddf.filter("gender == female")
-```
-
-Show all male actors which are born between 1960 and 1965:
-```
-> ddf.filter("gender == male and (age.born > 1960 and age.born < 1965)")
-```
-
-Show all actors which name contain the character 'e':
-```
-> ddf.filter("name contains e")
-```
-
-Show all actors which name matches a regular expression:
-```
-> ddf.filter("name matches .*e$")
-```
+For detailed examples of how the display filter can be utilized, please refer to the following:
 
-## Inspired by
+* [CSV Display Filter](https://github.com/bytebutcher/python-dict-display-filter/blob/main/docs/EXAMPLES.md#csv-display-filter)
+* [JSON Display Filter](https://github.com/bytebutcher/python-dict-display-filter/blob/main/docs/EXAMPLES.md#json-display-filter)
+* [SQLite Display Filter](https://github.com/bytebutcher/python-dict-display-filter/blob/main/docs/EXAMPLES.md#sqlite-display-filter)
+* [Nmap Display Filter](https://github.com/bytebutcher/python-dict-display-filter/blob/main/docs/EXAMPLES.md#nmap-display-filter)
 
-* <a href="https://wiki.wireshark.org/DisplayFilters">Wireshark Display Filter</a>
+## Acknowledgements
 
-## Powered by
+This project wouldn't be possible without these awesome projects:
 
+* <a href="https://wiki.wireshark.org/DisplayFilters">wireshark display filter</a>: Display filter for filtering network packages
 * <a href="https://github.com/wolever/parameterized">parameterized</a>: Parameterized testing with any Python test framework
 * <a href="https://github.com/pyparsing/pyparsing/">pyparsing</a>: Creating PEG-parsers made easy
 * <a href="https://github.com/bytebutcher/ipranger/">ipranger</a>: Parsing and matching IPv4-addresses

data/csv_example.csv

@@ -1,5 +1,5 @@
-"name","age","gender","killed"
-"Morpheus",38,"male",False
-"Neo",35,"male",False
-"Cipher",48,"male",True
-"Trinity",32,"female",False
+"name","actor","age","gender","killed"
+"Morpheus","Laurence Fishburne",38,"male",False
+"Neo","Keanu Reeves",35,"male",False
+"Cipher","Joe Pantoliano",48,"male",True
+"Trinity","Carrie-Anne Moss",32,"female",False

data/json_example.json

@@ -0,0 +1,6 @@
+[
+    {"name": "Morpheus","actor": ["Laurence", "Fishburne"], "age": {"born": "1961"}, "gender": "male"},
+    {"name": "Neo","actor": ["Keanu", "Reeves"], "age": {"born": "1964"}, "gender": "male", "power": ["flight", "bullet-time"]},
+    {"name": "Cipher","actor": ["Joe", "Pantoliano"], "age": {"born": "1951"}, "gender": "male"},
+    {"name": "Trinity","actor": ["Carrie-Anne", "Moss"], "age": {"born": "1967"}, "gender": "female"}
+]

data/sqlite_example.sqlite

@@ -34,5 +34,7 @@ The ```python-dict-display-filter``` defines a set of helpers which may be used
 
 | Helper Class                 | Description                                                                                                                     |
 |------------------------------|---------------------------------------------------------------------------------------------------------------------------------|
+| ```DisplayFilterShell```     | A command line loop which allows to filter data on a provided data store. Uses Table to print the result in a pretty table.     |
 | ```DictDisplayFilterShell``` | A command line loop which allows to filter data on a provided data store. Uses DictTable to print the result in a pretty table. |
+| ```Table```                  | Initialized with a data store, provides a filter method, prints results in a pretty table.                                      | 
 | ```DictTable```              | Initialized with a data store, provides a filter method, prints results in a pretty table.                                      | 

docs/DEVELOPER_GUIDE.md

@@ -5,16 +5,24 @@
 <h1 align="center" style="margin-top: 0px;">Examples</h1>
 <br>
 
+## Table of Contents
+
+1. [CSV Display Filter](#csv-display-filter)
+2. [JSON Display Filter](#json-display-filter)
+3. [Nmap Display Filter](#nmap-display-filter)
+4. [SQLite Display Filter](#sqlite-display-filter)
+
 ## CSV Display Filter
 
-This example shows how to create a display filter for CSV-files:
+This example shows how to use the display filter to query CSV-files:
 ```commandline
 python3 examples/csv_display_filter.py data/example.csv
 # Enter ?help for a list of commands.
 ```
 ```
 > fields
 name
+actor
 age
 gender
 killed
@@ -29,38 +37,36 @@ Neo  | 35  | male   | False
 1 row in set (0.01 secs)
 ```
 
-See <a href="https://github.com/bytebutcher/python-dict-display-filter/raw/main/examples/csv_display_filter.py">examples/csv_display_filter.py</a> for the actual source code.
+See <a href="https://github.com/bytebutcher/python-dict-display-filter/raw/main/examples/csv_display_filter.py">examples/csv_display_filter.py</a> for implementation details.
 
 
-# SQLite Display Filter
+## JSON Display Filter
 
-This example shows how to create a display filter for a sqlite database:
+This example shows how to use the display filter to query JSON files:
 ```commandline
-python3 examples/sqlite_display_filter.py data/example.sqlite
+python3 examples/json_display_filter.py data/example.json
 # Enter ?help for a list of commands.
 ```
 ```
 > fields
-name
-age
 gender
-killed
+name
+power
+actor
+age.born
 ```
 ```
 > filter name == Neo
-
-name | age | gender | killed
----- | --- | ------ | ------
-Neo  | 35  | male   | 0 
+{"name": "Neo", "actor": ["Keanu", "Reeves"], "age": {"born": "1964"}, "gender": "male", "power": ["flight", "bullet-time"]}
 
 1 row in set (0.01 secs)
 ```
 
-See <a href="https://github.com/bytebutcher/python-dict-display-filter/raw/main/examples/sqlite_display_filter.py">examples/sqlite_display_filter.py</a> for the actual source code.
+See <a href="https://github.com/bytebutcher/python-dict-display-filter/raw/main/examples/json_display_filter.py">examples/json_display_filter.py</a> for implementation details.
 
 ## Nmap Display Filter
 
-This example shows how to create a display filter for nmap-xml-files:
+This example shows how to use a display filter to query Nmap XML files:
 ```commandline
 python3 examples/nmap_display_filter.py data/nmap_example.xml
 # Enter ?help for a list of commands.
@@ -98,4 +104,40 @@ host           | port | protocol | status | service
 6 rows in set (0.01 secs)
 ```
 
-See <a href="https://github.com/bytebutcher/python-dict-display-filter/raw/main/examples/nmap_display_filter.py">examples/nmap_display_filter.py</a> for the actual source code.
+See <a href="https://github.com/bytebutcher/python-dict-display-filter/raw/main/examples/nmap_display_filter.py">examples/nmap_display_filter.py</a> for implementation details.
+
+## SQLite Display Filter
+
+This example shows how to use the display filter to query a SQLite database:
+```commandline
+python3 examples/sqlite_display_filter.py data/example.sqlite
+# Enter ?help for a list of commands.
+```
+```
+> tables
+People
+```
+```
+> use People
+Database changed
+```
+```
+> fields
+name
+actor
+age
+gender
+killed
+```
+
+```
+> filter name == Neo
+
+name | actor        | age | gender | killed
+---- | ------------ | --- | ------ | ------
+Neo  | Keanu Reeves | 35  | male   | 0 
+
+1 row in set (0.01 secs)
+```
+
+See <a href="https://github.com/bytebutcher/python-dict-display-filter/raw/main/examples/sqlite_display_filter.py">examples/sqlite_display_filter.py</a> for implementation details.

docs/EXAMPLES.md

@@ -0,0 +1,103 @@
+# vim: ts=8:sts=8:sw=8:noexpandtab
+#
+# This file is part of python-dict-display-filter.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+import argparse
+import json
+import logging
+import os.path
+import sys
+import traceback
+from typing import List, Set
+
+from pydictdisplayfilter import DictDisplayFilter
+from pydictdisplayfilter.helpers import Table, DisplayFilterShell
+
+
+def read_json_file(json_file):
+    try:
+        with open(json_file, mode='r') as infile:
+            return json.load(infile)
+    except Exception as err:
+        logger.debug(err)
+        raise Exception("{}: Error reading file".format(json_file))
+
+
+class JSONTable(Table):
+    """ Data store with filter capabilities and pretty table printout. """
+
+    def __init__(self, data_store: List[dict]):
+        """ Initializes the DictTable with a data store. """
+        self._data_store = data_store
+        super().__init__(DictDisplayFilter(data_store))
+
+    def _make_table(self, data_store: List[dict]) -> List[str]:
+        """ Creates a view for nested data items. """
+        return [json.dumps(data) for data in data_store]
+
+    def _extract_keys(self, data_store: List[dict]) -> List[str]:
+        """ Extracts the keys from a nested data store. """
+        def extract_keys(data: dict, parent_key: str='') -> Set[str]:
+            keys = set()
+            if isinstance(data, dict):
+                for k, v in data.items():
+                    new_key = f"{parent_key}.{k}" if parent_key else k
+                    keys |= extract_keys(v, new_key)
+            else:
+                keys.add(parent_key)
+            return keys
+
+        all_keys = set()
+        for data in data_store:
+            all_keys |= extract_keys(data)
+        return list(all_keys)
+
+    def fields(self, thorough: bool = True) -> List[str]:
+        """ Returns the field names used in the data store. """
+        if not self._data_store:
+            # No items in data store, so there are no fields to query either.
+            return list()
+        return self._extract_keys(self._data_store)
+
+
+class JSONDisplayFilterShell(DisplayFilterShell):
+    """ A little shell for querying a list of dictionaries using the display filter. """
+
+    def __init__(self, data_store: List[dict]):
+        """ Initializes the DictDisplayFilterShell with a data store. """
+        super().__init__(JSONTable(data_store))
+
+
+if __name__ == '__main__':
+    logging.basicConfig(format='%(message)s')
+    logger = logging.getLogger(__name__)
+    parser = argparse.ArgumentParser(description='Display filter for json file.')
+    parser.add_argument('json_file', action='store', metavar='FILE', help='JSON-file to load')
+
+    # Parse all arguments
+    arguments = parser.parse_args()
+
+    json_file = arguments.json_file
+    if not os.path.isfile(json_file):
+        logger.error("{}: No such file".format(json_file))
+        sys.exit(1)
+
+    try:
+        data_store = read_json_file(json_file)
+        JSONDisplayFilterShell(data_store).cmdloop()
+    except Exception as err:
+        logger.error(str(err))
+        traceback.print_exc()
+        sys.exit(1)

examples/json_display_filter.py

@@ -20,7 +20,7 @@
 import sys
 import traceback
 from collections import defaultdict
-from typing import List, Dict
+from typing import List
 
 from pydictdisplayfilter.helpers import DictDisplayFilterShell