VirtualFileSystem: detect multiple VFS in one app, provide way to namespace and isolate VFS instances

Author: steve-s

docs/user/Embedding-Build-Tools.md

@@ -12,7 +12,7 @@ required for embedding Python code in Java-based applications:
 - *Python application files* provided by the user, for example, Python sources which are part of the project.
 - *Third-party Python packages* installed by the plugin during the build according to the plugin configuration.
 
-Apart from physically managing and deploying those files, it is also necessary to make them available in Python at runtime by configuring the **GraalPy Context** in your Java code accordingly. 
+Apart from physically managing and deploying those files, it is also necessary to make them available in Python at runtime by configuring the **GraalPy Context** in your Java code accordingly.
 The [GraalPyResources](https://github.com/oracle/graalpython/blob/master/graalpython/org.graalvm.python.embedding/src/org/graalvm/python/embedding/utils/GraalPyResources.java) API provides factory methods to create a Context preconfigured for accessing Python, embedding relevant resources with a **Virtual Filesystem** or from a dedicated **external directory**.
 
 ## Deployment
@@ -22,12 +22,15 @@ There are two modes how to deploy the resources: as Java resources using the Vir
 ### Virtual Filesystem
 
 The Python related resources are embedded in the application file, either in JAR or Native Image generated
-executable, as standard Java resources. 
-The GraalPy Virtual Filesystem accesses resource files as standard Java resources and makes them available to Python code running in GraalPy.
+executable, as standard Java resources.
+The GraalPy Virtual Filesystem internally accesses the resource files as standard Java resources and makes them available to Python code running in GraalPy.
 This is transparent to the Python code, which can use standard Python IO to access those files.
 
-Java resource files in a Maven or Gradle project are typically located in dedicated resources directories.
-All resources subdirectories named _org.graalvm.python.vfs_ are merged and mapped to a configurable Virtual Filesystem mount point at the Python side, by default `/graalpy_vfs`. 
+Java resource files in a Maven or Gradle project are typically located in dedicated resources directories, such as `src/main/resources`.
+Moreover, there can be multiple resources directories and Maven or Gradle usually merges them.
+
+User can choose relative Java resources path that will be made accessible in Python through the virtual filesystem,
+by default it is `org.graalvm.python.vfs`. All resources subdirectories with this path are merged during build and mapped to a configurable Virtual Filesystem mount point at the Python side, by default `/graalpy_vfs`.
 For example, a Python file with the real filesystem path `${project_resources_directory}/org.graalvm.python.vfs/src/foo/bar.py` will be accessible as `/graalpy_vfs/src/foo/bar.py` in Python.
 
 Use the following [GraalPyResources](https://github.com/oracle/graalpython/blob/master/graalpython/org.graalvm.python.embedding/src/org/graalvm/python/embedding/utils/GraalPyResources.java)
@@ -36,20 +39,37 @@ factory methods to create GraalPy Context preconfigured for the use of the Virtu
 * `GraalPyResources.contextBuilder()`
 * `GraalPyResources.contextBuilder(VirtualFileSystem)`
 
+#### Java Resource Path
+Particularly when developing reusable libraries, it is recommended to use custom unique Java resources path for your
+virtual filesystem to avoid conflicts with other libraries on the classpath or modulepath that may also use the
+Virtual Filesystem. The recommended path is:
+  ```
+  GRAALPY-VFS/${project.groupId}/${project.artifactId}
+  ```
+
+The Java resources path must be configured in the Maven and Gradle plugins and must be also set to the same value
+at runtime using the `VirtualFileSystem$Builder#resourceDirectory` API.
+
+*Note regarding Java module system: resources in named modules are subject to the encapsulation rules specified by
+[Module.getResourceAsStream](https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/lang/Module.html#getResourceAsStream(java.lang.String)).
+This is also the case of the default virtual filesystem location. When a resources directory is not
+a valid Java package name, such as the recommended "GRAALPY-VFS", the resources are not subject to
+the encapsulation rules and do not require additional module system configuration.*
+
 ### External Directory
 
-As an alternative to Java resources with the Virtual Filesystem, it is also possible to configure the Maven or Gradle plugin to manage the contents of an external directory, which will **not be embedded** as a Java resource into the resulting application. 
-A user is then responsible for the deployment of such directory. 
+As an alternative to Java resources with the Virtual Filesystem, it is also possible to configure the Maven or Gradle plugin to manage the contents of an external directory, which will **not be embedded** as a Java resource into the resulting application.
+A user is then responsible for the deployment of such directory.
 Python code will access the files directly from the real filesystem.
 
 Use the following [GraalPyResources](https://github.com/oracle/graalpython/blob/master/graalpython/org.graalvm.python.embedding/src/org/graalvm/python/embedding/utils/GraalPyResources.java) factory methods to create GraalPy Context preconfigured for the use of an external directory:
 * `GraalPyResources.createContextBuilder(Path)`
 
 ## Conventions
 
-The factory methods in [GraalPyResources](https://github.com/oracle/graalpython/blob/master/graalpython/org.graalvm.python.embedding/src/org/graalvm/python/embedding/utils/GraalPyResources.java) rely on the following conventions, where the `${root}` is either an external directory, or a Virtual System mount point on the Python side and `${project_resources_directory}/org.graalvm.python.vfs` on the real filesystem:
+The factory methods in [GraalPyResources](https://github.com/oracle/graalpython/blob/master/graalpython/org.graalvm.python.embedding/src/org/graalvm/python/embedding/utils/GraalPyResources.java) rely on the following conventions, where the `${root}` is either an external directory, or a Virtual System mount point on the Python side and Java resources directories, such as `${project_resources_directory}/org.graalvm.python.vfs`, on the real filesystem:
 - `${root}/src`: used for Python application files. This directory will be configured as the default search path for Python module files (equivalent to `PYTHONPATH` environment variable).
-- `${root}/venv`: used for the Python virtual environment holding installed third-party Python packages. 
+- `${root}/venv`: used for the Python virtual environment holding installed third-party Python packages.
 The Context will be configured as if it is executed from this virtual environment. Notably packages installed in this
 virtual environment will be automatically available for importing.
 
@@ -73,8 +93,8 @@ Add the plugin configuration in the `configuration` block of `graalpy-maven-plug
     ...
 </plugin>
 ```
-The **packages** element declares a list of third-party Python packages to be downloaded and installed by the plugin.
-- The Python packages and their versions are specified as if used with `pip`:
+- The **packages** element declares a list of third-party Python packages to be downloaded and installed by the plugin.
+The Python packages and their versions are specified as if used with `pip`:
   ```xml
   <configuration>
       <packages>
@@ -84,11 +104,18 @@ The **packages** element declares a list of third-party Python packages to be do
       ...
   </configuration>
   ```
-- If the **pythonResourcesDirectory** element is specified, then the given directory is used as an [external directory](#external-directory) and no Java resources are embedded.
-Remember to use the appropriate `GraalPyResources` API to create the Context.
+
+- The **resourceDirectory** element can specify the relative [Java resource path](#java-resource-path).
+  Remember to use `VirtualFileSystem$Builder#resourceDirectory` when configuring the `VirtualFileSystem` in Java.
+  ```xml
+  <resourceDirectory>GRAALPY-VFS/${project.groupId}/${project.artifactId}</resourceDirectory>
+  ```
+
+- If the **externalDirectory** element is specified, then the given directory is used as an [external directory](#external-directory) and no Java resources are embedded.
+Remember to use the appropriate `GraalPyResources` API to create the Context. This element and **resourceDirectory** are mutually exclusive.
   ```xml
   <configuration>
-      <pythonResourcesDirectory>${basedir}/python-resources</pythonResourcesDirectory>
+      <externalDirectory>${basedir}/python-resources</externalDirectory>
       ...
   </configuration>
   ```
@@ -118,11 +145,18 @@ The plugin can be configured in the `graalPy` block:
     ...
   }
   ```
-- If the **pythonResourcesDirectory** element is specified, then the given directory is used as an [external directory](#external-directory) and no Java resources are embedded. 
+
+- The **resourceDirectory** element can specify the relative [Java resource path](#java-resource-path).
+  Remember to use `VirtualFileSystem$Builder#resourceDirectory` when configuring the `VirtualFileSystem` in Java.
+  ```
+  resourceDirectory = "GRAALPY-VFS/my.group.id/artifact.id"
+  ```
+
+- If the **externalDirectory** element is specified, then the given directory is used as an [external directory](#external-directory) and no Java resources are embedded.
   Remember to use the appropriate `GraalPyResources` API to create the Context.
   ```
   graalPy {
-    pythonResourcesDirectory = file("$rootDir/python-resources")
+    externalDirectory = file("$rootDir/python-resources")
     ...
   }
   ```

graalpython/com.oracle.graal.python.test.integration/src/GRAALPY-VFS/com.oracle.graal.python.test/integration/SomeFile

@@ -0,0 +1,2 @@
+text1
+text2

graalpython/com.oracle.graal.python.test.integration/src/GRAALPY-VFS/com.oracle.graal.python.test/integration/dir1/extractme

@@ -0,0 +1,2 @@
+text1
+text2

graalpython/com.oracle.graal.python.test.integration/src/GRAALPY-VFS/com.oracle.graal.python.test/integration/dir1/file2

@@ -0,0 +1,24 @@
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/dir1/
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/dir1/extractme
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/dir1/file2
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/emptydir/
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/file1
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/extractme
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/fileslist.txt
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/SomeFile
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/site-packages/
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/site-packages/testpkg/
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/site-packages/testpkg/file.tso
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/site-packages/testpkg.libs/
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/site-packages/testpkg.libs/file1.tso
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/site-packages/testpkg.libs/file2.tso
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/site-packages/testpkg.libs/dir/
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/site-packages/testpkg.libs/dir/file1.tso
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/site-packages/testpkg.libs/dir/file2.tso
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/site-packages/testpkg.libs/dir/nofilterfile
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/site-packages/testpkg.libs/dir/dir/
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/site-packages/testpkg.libs/dir/dir/file1.tso
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/site-packages/testpkg.libs/dir/dir/file2.tso
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/site-packages/testpkg-nolibs/
+/GRAALPY-VFS/com.oracle.graal.python.test/integration/site-packages/testpkg-nolibs/file.tso

graalpython/com.oracle.graal.python.test.integration/src/GRAALPY-VFS/com.oracle.graal.python.test/integration/extractme

@@ -0,0 +1,3 @@
+/GRAALPY-VFS/foo/
+/GRAALPY-VFS/foo/otherdir/
+/GRAALPY-VFS/foo/otherdir/hello.txt

graalpython/com.oracle.graal.python.test.integration/src/GRAALPY-VFS/com.oracle.graal.python.test/integration/file1

@@ -0,0 +1 @@
+world

graalpython/com.oracle.graal.python.test.integration/src/GRAALPY-VFS/com.oracle.graal.python.test/integration/fileslist.txt

@@ -51,6 +51,7 @@
  * Siple copy of GraalPyResourcesTests to test also the deprecated
  * org.graalvm.python.embedding.utils pkg
  */
+@SuppressWarnings("deprecation")
 public class GraalPyResourcesUtilsTests {
 
     @Test