Release 1.8.0 (#244)

* adapt for 16 channel

* temp filter fix

* add data recovery from bin feature, and 16 channel bin2csv

* linting APIS-709

* fix for 32 channel APIS-709

* save settings on disconnection

* fix linting

* delete blank space

* apply isort

* remove thread lock in subscriber for mixed sps fix

* replace unused variable with -

* remove unused imports

* Feature channel 32 APIS-729 (#241)

* adapt for 16 channel

* temp filter fix

* add data recovery from bin feature, and 16 channel bin2csv

* linting APIS-709

* fix for 32 channel APIS-709

* save settings on disconnection

* fix linting

* delete blank space

* apply isort

* remove thread lock in subscriber for mixed sps fix

* replace unused variable with -

* remove unused imports

Co-authored-by: Andrea Escartin <[email protected]>

* round recovery data timestamp APIS-663

* remove debug print

* add support for external string marker

* Feature channel 32 (#243)

* adapt for 16 channel

* temp filter fix

* add data recovery from bin feature, and 16 channel bin2csv

* linting APIS-709

* fix for 32 channel APIS-709

* save settings on disconnection

* fix linting

* delete blank space

* apply isort

* remove thread lock in subscriber for mixed sps fix

* replace unused variable with -

* remove unused imports

* round recovery data timestamp APIS-663

* remove debug print

* Made recovering from bin a little faster

* APIS-663 adjust with time offset

* add support for external string marker (#242)

* add support for external string marker

* add example script to demonstrate integration with LSL markers

* add clock sync and thread safety for inlet variable

* delete test script

---------

Co-authored-by: Andrea Escartin <[email protected]>
Co-authored-by: Sonja Stefani <[email protected]>

* fix linting

* fix isort

* Update docs 1.8.0 (#245)

* update documentation

* bumpversion

* Update author list and python versions

* remove debug prints

* Update installer with new deps

* Update deps (#246)

* switch to primitive python float for new numpy

* undo fixed numpy and pyedflib versions

* adjust time offset for external marker

* Unit tests for EEG packets (#247) (#248)

* Unit tests for EEG packets (#247)

* Added unignore lines for test files in gitignore

* Added some unit tests for filters.py

* Refactored filters.py some and added unit test(s) to test_filters.py

* Added tests for packet

* Added packet test cases

* Added README for tests

* Fixed wrong patching in test_packet.py, added test

* Added stubs

* Added test_packet.py script, changed workflows script

* yaml changed

* Made tiny change in eeg data for tests

* Added Jenkinsfile, cleaned up testscripts and filters.py

* Renamed Jenkinsfile

* Adjusted Jenkinsfile

* Added Dockerfile for Linux container, adapted Jenkinsfile

* Adapted Jenkinsfile to record test results, made sure image is built from Dockerfile properly

* Changed testing.yml

* Converted unittest functions to use pytest and pytest-mock

* Added pytest-mock as requirement

* Added fixtures for data reading and passing to tests

* Changed gitignore to allow tests/, added test and expected output files as well as test fixtures

* Added file for expected outcome from fake data

* Changed test res structure to have input and output test data, added more (bin) test packets

* Parametrized fixture that provides eeg in and output data, changed workflow to create html report

* Changed workflow to store artifact, removed windows and macos for now to reduce time

* Added clause that archiving runs on success and failure (sigh)

* Changed html creation to self contained

* Changed workflow back to run on pr on master

* Added new test input and output files, added parametrized tests for int24to32, tests for abstract (Packet and EEG), test for timestamp init (Packet)

* Added tests for EEG base packet and more expected output data to res/out/

* Edited ReadMe to mention res folders and formats

* Added more res files in conftests.py

* Removed unnecessary packet

* Removed unnecessary files / unfinished tests

* Fixed formatting so linter is happy

* Sorted imports for isort...

* Fixed testing workflow (how did that bracket get there?)

* Changed testing workflow to trigger on PR to develop as well

---------

Co-authored-by: Sonja Stefani <[email protected]>
Co-authored-by: Sonja Stefani <[email protected]>

* Update deps (#249)

* switch to primitive python float for new numpy

* undo fixed numpy and pyedflib versions

* replace np.float with float (#250)

* Fix numpy chnagelog (#251)

* replace np.float with float

* remove np namespace var and unused var

* Fix numpy chnagelog (#252)

* replace np.float with float

* remove np namespace var and unused var

* remove more numpy var

* edit changelog for release

---------

Co-authored-by: Andrea Escartin <[email protected]>
Co-authored-by: Sonja Stefani <[email protected]>
Co-authored-by: Sonja Stefani <[email protected]>

Author: 3 people

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 1.7.0
+current_version = 1.8.0
 commit = False
 tag = False
 

.github/workflows/testing.yml

@@ -1,13 +1,13 @@
 name: testing
 on:
     pull_request:
-        branches: [master]
+        branches: [master, develop]
 jobs:
     test:
         strategy:
             matrix:
                 os: [ubuntu-latest, windows-latest, macos-latest]
-                python: [3.8.10]
+                python: [3.9.12]
         runs-on: ${{ matrix.os }}
         timeout-minutes: 10
         steps:
@@ -17,7 +17,7 @@ jobs:
                 fetch-depth: 0
                 clean: false
 
-            - name: checkout to current branch
+            - name: Checkout to current branch
               run: git checkout ${{ env.BRANCH }}
 
             - name: Install non-python dependencies on Ubuntu
@@ -30,7 +30,14 @@ jobs:
               with:
                   python-version: ${{ matrix.python }}
                   conda-channels: anaconda, conda-forge
-            - run: |
+            - name: Install liblsl from conda-forge
+              run: |
                 conda install -c conda-forge liblsl
                 pip install -e .[test]
-                python -m pytest --import-mode=append
+                python -m pytest --import-mode=append --html=pytest_report.html --self-contained-html
+            -   name: Archive test results
+                if: success() || failure()
+                uses: actions/upload-artifact@v3
+                with:
+                    name: pytest-results
+                    path: pytest_report.html

.gitignore

@@ -32,13 +32,8 @@ pip-log.txt
 nosetests.xml
 coverage.xml
 htmlcov
-tests/
 src/explorepy/example.py
 
-
-
-
-
 # Translations
 *.mo
 

CHANGELOG.rst

@@ -1,6 +1,17 @@
 
 Changelog
 =========
+
+
+1.8.0 (27.2.2023)
+------------------
+* Add support for explore+ 16 ch device
+* Unit test integration with CI/CD
+* Repair CSV from binary data
+* Save current device settings on disconnection, restore them on demand
+* Receive external triggers from LSL
+
+
 1.7.0 (21.12.2022)
 ------------------
 * Add suppport for new explore+ 32 ch device
@@ -10,7 +21,7 @@ Changelog
 
 1.6.3 (25.10.2022)
 ------------------
-* Add new 8 channel Explore+ device 
+* Add new 8 channel Explore+ device
 
 
 1.6.2 (7.09.2022)

README.rst

@@ -17,9 +17,9 @@
     :target: https://pypi.org/project/explorepy
 
 
-.. |commits-since| image:: https://img.shields.io/github/commits-since/Mentalab-hub/explorepy/v1.7.0.svg
+.. |commits-since| image:: https://img.shields.io/github/commits-since/Mentalab-hub/explorepy/v1.8.0.svg
     :alt: Commits since latest release
-    :target: https://github.com/Mentalab-hub/explorepy/compare/v1.7.0...master
+    :target: https://github.com/Mentalab-hub/explorepy/compare/v1.8.0...master
 
 
 .. |wheel| image:: https://img.shields.io/pypi/wheel/explorepy.svg
@@ -60,7 +60,7 @@ For other operating systems, or to build the package manually on Windows, please
 Requirements
 ------------
 
-* Python 3.7 to Python 3.10.
+* Python 3.7 to Python 3.11.
 * Visual Studio 2015 community edition (Windows only. For package building).
 * Bluetooth header files (Linux only. Use: ``sudo apt-get install libbluetooth-dev``).
 
@@ -137,6 +137,7 @@ Authors
 - `Mohamad Atayi`_
 - `Salman Rahman`_
 - `Andrea Escartin`_
+- `Sonja Stefani`_
 - `Alex Platt`_
 - `Andreas Gutsche`_
 - `Masooma Fazelian`_
@@ -148,6 +149,7 @@ Authors
 .. _Mohamad Atayi: https://github.com/bmeatayi
 .. _Salman Rahman: https://github.com/salman2135
 .. _Andrea Escartin: https://github.com/andrea-escartin
+.. _Sonja Stefani: https://github.com/SonjaSt
 .. _Alex Platt: https://github.com/Nujanauss
 .. _Andreas Gutsche: https://github.com/andyman410
 .. _Masooma Fazelian: https://github.com/fazelian

docs/conf.py

@@ -29,7 +29,7 @@
 year = '2018-2022'
 author = 'Mentalab GmbH.'
 copyright = '{0}, {1}'.format(year, author)
-version = release = '1.7.0'
+version = release = '1.8.0'
 
 pygments_style = 'trac'
 templates_path = ['.']

docs/installation.rst

@@ -4,14 +4,14 @@ Installation
 
 Minimal Requirements
 ------------
-* Python 3.6 to Python 3.10
+* Python 3.7 to Python 3.11
 * Microsoft Build Tools for Visual Studio 2019 (only Windows)
 * 6GB RAM (minimum 1GB *free* RAM during the session)
 * Intel i5 or higher (2x2.5GHz) CPU
 
 Recommended Requirements
 ------------
-* Python 3.6 to Python 3.10
+* Python 3.7 to Python 3.11
 * Microsoft Build Tools for Visual Studio 2019 (only Windows)
 * 8GB RAM
 * Intel i7 or higher CPU

examples/receive_external_marker.py

@@ -0,0 +1,42 @@
+import time
+from threading import Thread
+
+import explorepy
+
+
+def modify_variable(explore_instance):
+    from pylsl import StreamInlet, resolve_stream
+    print("looking for a marker stream...")
+    streams = resolve_stream('type', 'Markers')
+
+    # create a new inlet to read from the stream
+    # adds processing flags for clocksync and thread safety
+    # Reference: https://labstreaminglayer.readthedocs.io/info/faqs.html#lsl-local-clock
+    inlet = StreamInlet(streams[0], processing_flags=1 | 8)
+
+    while True:
+        sample, timestamp = inlet.pull_sample()
+        print("got %s at time %s" % (sample[0], timestamp))
+        explore_instance.set_external_marker(timestamp, str(sample[0]))
+        time.sleep(1)
+
+
+def main():
+    # Create an Explore object
+    explore = explorepy.Explore()
+    explore.connect(device_name='Explore_XXXX')
+    explore.record_data(file_name='test_event_gen', file_type='csv', do_overwrite=True, block=False)
+    t = Thread(target=modify_variable, args=(explore,))
+    t.start()
+
+    n_iteration = 100
+    interval = 2
+
+    for i in range(n_iteration):
+        time.sleep(interval)
+
+    explore.stop_recording()
+
+
+if __name__ == "__main__":
+    main()

installer/windows/installer.cfg

@@ -1,12 +1,12 @@
 [Application]
 name=MentaLab ExplorePy
-version=1.7.0
+version=1.8.0
 entry_point=explorepy.cli:cli
 console=true
 icon=mentalab.ico
 
 [Python]
-version=3.7.9
+version=3.8.0
 bitness=64
 
 
@@ -16,32 +16,49 @@ entry_point=explorepy.cli:cli
 
 [Include]
 pypi_wheels =
-    explorepy==1.7.0
+    #explorepy==1.8.0
     appdirs==1.4.3
 #   bokeh==2.2.3 # No wheel available
     certifi==2020.12.5
-    Click==7.0
-	eeglabio==0.0.2.post2
-    Jinja2==2.11.2
-    MarkupSafe==1.1.1
-	mne==1.1.1
-    numpy==1.19.2
-    packaging==20.4
-    Pillow==7.2.0
-    pyEDFlib==0.1.15
-    pylsl==1.13.1
-    pyparsing==2.4.7
-    python-dateutil==2.8.1
-    PyYAML==5.3.1
-    scipy==1.5.2
+    click==7.1.2
+    colorama==0.4.6
+    contourpy==1.0.7
+    cycler==0.11.0
+    decorator==5.1.1
+    eeglabio==0.0.2.post3
+    fonttools==4.38.0
+    idna==3.4
+    Jinja2==3.0.0
+    kiwisolver==1.4.4
+    MarkupSafe==2.1.2
+    matplotlib==3.6.3
+    mne==1.3.0
+    numpy==1.21.4
+    packaging==23.0
+    pandas==1.5.3
+    Pillow==9.4.0
+    pipdeptree==2.3.3
+    pooch==1.6.0
+    pyEDFlib==0.1.25
+    pylsl==1.16.0
+    pyparsing==3.0.9
+    python-dateutil==2.8.2
+    pytz==2022.7.1
+    PyYAML==6.0
+    requests==2.28.2
+    scipy==1.10.0
     sentry-sdk==1.0.0
-    six==1.15.0
-    tornado==6.0.4
-    typing-extensions==3.7.4
-	urllib3==1.26.4
+    six==1.16.0
+    tornado==6.2
+    tqdm==4.64.1
+    typing_extensions==4.4.0
+    urllib3==1.26.14
+    wincertstore==0.2
+    setuptools==67.1.0
 
 local_wheels =
     local-wheels/bokeh-2.2.3-py3-none-any.whl
+    local-wheels/explorepy-1.8.0-cp38-cp38-win_amd64.whl
 
 # Other files and folders that should be installed
 files = ../../licenses

lib/windows/test.py

@@ -32,8 +32,10 @@ def read(*names, **kwargs):
         return fh.read()
 
 
-my_req = ['numpy==1.21.4', 'scipy', 'pyedflib==0.1.25', 'click==7.1.2', 'appdirs==1.4.3', 'sentry_sdk==1.0.0', 'mne', 'eeglabio', 'pandas']  # noqa: E501
+my_req = ['numpy', 'scipy', 'pyedflib', 'click==7.1.2', 'appdirs==1.4.3', 'sentry_sdk==1.0.0', 'mne', 'eeglabio', 'pandas']  # noqa: E501
 test_requirements = ["pytest==6.2.5",
+                     "pytest-mock==3.10.0",
+                     "pytest-html==3.2.0",
                      "flake8==4.0.1",
                      "isort==5.10.1"]
 extras = {"test": test_requirements}
@@ -80,7 +82,7 @@ def read(*names, **kwargs):
         os.system('cp  lib/mac/exploresdk.py  src/explorepy')
 setup(
     name='explorepy',
-    version='1.7.0',
+    version='1.8.0',
     license='MIT license',
     description='Python API for Mentalab biosignal aquisition devices',
     long_description_content_type="text/markdown",

setup.py

@@ -22,7 +22,7 @@
 
 
 __all__ = ["Explore", "Dashboard", "command", "exploresdk", "tools", "log_config"]
-__version__ = '1.7.0'
+__version__ = '1.8.0'
 
 this = sys.modules[__name__]
 this._bt_interface = 'sdk'

src/explorepy/__init__.py

@@ -17,7 +17,6 @@
 import os
 import re
 import time
-from datetime import datetime
 from threading import Timer
 
 import numpy as np
@@ -170,14 +169,15 @@ def record_data(
                                                     file_type=file_type,
                                                     do_overwrite=do_overwrite)
 
+        #  TODO: make sure older timestamp in meta file was not used in any other software!
         if file_type == 'csv':
             self.recorders['marker'] = create_marker_recorder(filename=marker_out_file, do_overwrite=do_overwrite)
             self.recorders['meta'] = create_meta_recorder(filename=meta_out_file,
                                                           fs=self.stream_processor.device_info['sampling_rate'],
                                                           adc_mask=SettingsManager(self.device_name).get_adc_mask(),
                                                           device_name=self.device_name,
                                                           do_overwrite=do_overwrite,
-                                                          timestamp=datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
+                                                          timestamp=str(self.stream_processor.parser._time_offset))  # noqa: E501
             self.recorders['meta'].write_meta()
             self.recorders['meta'].stop()
 
@@ -252,7 +252,9 @@ def convert_bin(self, bin_file, out_dir='', file_type='edf', do_overwrite=False,
         self.mask = self.stream_processor.device_info['adc_mask']
         if 'board_id' in self.stream_processor.device_info:
             if 'PCB_304_801_XXX' in self.stream_processor.device_info['board_id']:
-                self.mask = [1 for i in range(0, 32)]
+                self.mask = [1 for _ in range(0, 32)]
+            if 'PCB_305_801_XXX' in self.stream_processor.device_info['board_id']:
+                self.mask = [1 for _ in range(0, 16)]
 
         self.recorders['exg'] = create_exg_recorder(filename=exg_out_file,
                                                     file_type=self.recorders['file_type'],
@@ -282,8 +284,8 @@ def convert_bin(self, bin_file, out_dir='', file_type='edf', do_overwrite=False,
         def device_info_callback(packet):
             new_device_info = packet.get_info()
             if not self.stream_processor.compare_device_info(new_device_info):
-                new_file_name = exg_out_file + "_" + str(np.round(packet.timestamp, 0))
-                new_meta_name = meta_out_file + "_" + str(np.round(packet.timestamp, 0))
+                new_file_name = exg_out_file[:-4] + "_" + str(np.round(packet.timestamp, 0)) + '_ExG'
+                new_meta_name = meta_out_file[:-4] + "_" + str(np.round(packet.timestamp, 0)) + '_Meta'
                 logger.warning("Creating a new file: " + new_file_name + '.' + self.recorders['file_type'])
                 self.stream_processor.unsubscribe(callback=self.recorders['exg'].write_data, topic=TOPICS.raw_ExG)
                 self.stream_processor.unsubscribe(callback=self.recorders['marker'].set_marker, topic=TOPICS.marker)
@@ -414,6 +416,16 @@ def set_marker(self, code):
         self._check_connection()
         self.stream_processor.set_marker(code=code)
 
+    def set_external_marker(self, time_lsl, marker_string):
+        """Sets a digital event marker while streaming
+
+        Args:
+            time_lsl (timestamp): timestamp from external marker)
+            marker_string (string): string to save as experiment marker)
+        """
+        self._check_connection()
+        self.stream_processor.set_ext_marker(time_lsl, marker_string)
+
     def format_memory(self):
         """Format memory of the device