Migrating branches from nonPluginized Scipion to the new Scipion Xmipp structure

Introduction

Scipion was allocating wrappers for the different packages (Xmipp, Relion, Eman...) until the v1.2.1 (included). From the ¿v2.0? (or v1.3) all this wrappers become external plugins allocated in independent repositories and they will be installed as pip modules. Thus, in this new version, the directory <scipion_home>/pyworkflow/em/packages disappears and the installed plugins are added to the python's site-packages folder as the other python modules.

On the other hand, the XMIPP software (C++ and Java sources) was also allocated in the Scipion repository, so far. Now, we have split the XMIPP software in three repositories independent from Scipion (XmippCore, XmippViz and Xmipp) plus the plugin repository (scipion-em-xmipp). All this repositories are in I2PC gitHub.

Getting ready in the new structure

Scipion

This is the easiest part. You just needs to checkout the devel-pluginization branch. However, in order to keep a Scipion/Xmipp backup, we strongly recommend to make a fresh installation of Scipion in a new directory. You can name this directory as you want, in this tutorial we will refer to it as <pluginized_scipion>

cd [where-you-want-to-install-scipion-usually-home]
git clone https://github.com/I2PC/scipion <pluginized_scipion>
cd <pluginized_scipion>
git checkout devel-pluginization

As it is a fresh installation, Scipion needs to be installed from top. You can follow this, but basically

./scipion config
./scipion install -j N

where N is the number of the processors.

After a while, Scipion should be installed. You can lunch it by ./scipion but with a very limited features since no plugin is installed yet.

Xmipp

Xmipp installation

We will deal with four repositories. Thus, in order to have all together we create a directory where to work. You can name this directory as you want, in this tutorial we will refer to it as <xmipp_bundle>

cd [where-you-want-to-install-xmipp-usually-home]
mkdir <xmipp_bundle>
cd <xmipp_bundle>

In order to get all the repositories automatically, we can copy the main xmipp script from here to a new file called xmipp

gedit xmipp  # copy and paste from link above

and save it after copy/paste the code. Change file to be executable. chmod a+x xmipp

Then we are ready to download, compile and install Xmipp in the system by

./xmipp all [N=8]

where N is the number of the processors.

The new version of Xmipp takes the common libraries (scons, fftw3, openmpi, hdf5, opencv...) from the system. Thus, they need to be installed and declared in the PATH. The installer will provide hints if something needed is not found.

After a while, it should be installed. Therefore, the installer have created the src and build directories as follow

   <xmipp_bundle>
      ├── xmipp                   # Main xmipp script
      ├── src                     # Sources directory
      │    ├── xmippCore          # Core of the xmipp                  **git repo**
      │    │     │
      │    │   [ . . . ]          # some coding files and dirs
      │    │
      │    ├── xmippViz           # Vizualization programs (showJ...)  **git repo**
      │    │     │
      │    │   [ . . . ]          # some coding files and dirs
      │    │
      │    ├── xmipp              # EM-reconstruction code of xmipp    **git repo**
      │    │     │
      │    │   [ . . . ]          # some coding files and dirs
      │    │ 
      │    ├── scipion-em-xmipp   # xmipp plugin to scipion            **git repo**
      │          │
      │        [ . . . ]          # some coding files and dirs
      │
      ├── build                   # Compiled files directory
           ├── bin                # Binaries
           │    │
           │  [ . . . ]           # xmipp programs
           │
       [ . . . ]                  # some libs and dirs
           │
           ├── xmipp.bashrc       # bash script to configure the environ

As Xmipp is installed, we can run any test or program after setting the environ variables by

source build/xmipp.bashrc                       # to set the environ
./xmipp test --show                             # to show all program's tests
./xmipp test AngularProjectionMatching          # to run a test
xmipp_reconstruct_significant [some-arguments]  # to run a program

Xmipp plugin

<xmipp_bundle>/src/scipion-em-xmipp contains the Xmipp plugin for Scipion and it is outside of Scipion directory. As we said at the bigining, the em-plugins are recognized by scipion as python modules. Therefore, Scipion needs to discover it in some way. At this point, there are two options:

Making a link from the plugin to the python's site-package in scipion

ln -s <xmipp_bundle>/src/scipion-em-xmipp/xmipp3 <pluginized-scipion>/software/lib/python2.7/site-packages/

or

Adding the path where is the plugin to the PYTHONPATH environ variable. In order to have it in all the sessions, we add the following line to the user's .bashrc
```
$gedit ~/.basrc
(append at file)  >>>  export PYTHONPATH=$PYTHONPATH:<xmipp_bundle>/src/scipion-em-xmipp
```

Let's have a look inside the plugin

   <xmipp_bundle>
      ├── xmipp
      ├── src
      │    ├── xmippCore 
      │    ├── xmippViz
      │    ├── xmipp
      │    │ 
      │    ├── scipion-em-xmipp  > > > XMIPP PLUGIN < < <  **git repo**
      │          │
      │          ├── convert       # dir with convertings
      │          │
      │          ├── protocols     # dir with all the protocols
      │          │
      │          ├── tests         # dir with all the tests
      │          │
      │          ├── viewers       # dir with all the viewers
      │          │
      │      [ . . . ]  # general code files
      │
      ├── build

Notice that now all protocols are inside the dedicated directory, as well as the viewers.

We can set a pyCharm project with the Xmipp as a Scipion plugin. Therefore, open pyCharm and click on File -> New project and choose the location <xmipp_bundle>/src/scipion-em-xmipp. It's preferable (but not necessary) to choose the python from scipion as interpreter by navigating to <pluginized_scipion>/software/bin/python2.7 in the interpreter section. Finally, Create.

It's very useful to include the scipion as a python library to detect the imports from Scipion. To do that, File -> Settings and click on the wheel at right of the interpreter field and show all. With the python from scipion selected, we click on the last icon at right (something like a folders tree) and, there, we add the <pluginized_scipion> directory by the + simbol.

Xmipp software

Scipion needs to find where the XMIPP software is installed, thus we have two option again:

Making a link of the build (recomended)
```
ln -s <xmipp_bundle>/build <pluginized_scipion>/software/em/xmipp
```
Notice that the final directory of the link has to coincide to the XMIPP_HOME variable in ~/.config/scipion/scipion.conf

or

Editing the ~/.config/scipion/scipion.conf to point to the Xmipp build

$gedit ~/.config/scipion/scipion.conf
(update the line)  >>>  XMIPP_HOME = <xmipp_bundle>/build

After that Scipion should normally launch. We can check it by

cd <pluginized_scipion>
./scipion

Some test can be run to check that all work fine. For instance,

./scipion test xmipp3.tests.test_protocols_xmipp_mics.TestXmippExtractParticles

Code migration

If we have code in some branches based on the devel branch of the non-pluginized Scipion, we need to migrate it to the new structure. At this point the old scipion dir that we kept in the system becomes very useful. We refer to that as <OLD_scipion>.

The goal is to bring all the changes from our working branch (in comparison with devel). To clearly see these changes, it is useful to simulate a pull request in GitHub against the devel branch. Thus, go to www.github.com/<your-user>/scipion (if you forked, if not from I2PC) and search the branch that you want to migrate. Then, click on New pull request and change the base to devel. Now, a list of changed files has to be shown.

This files may belong to scipion, scipion-em-xmipp, xmipp, xmippCore or xmippViz in the new structure. Thus, we need to identify where to migrate it.

scipion: Genuine scipion code. Very low-level things, such as config, protocols.config...
scipion-em-xmipp: It is the plugin. The protocol (wrapper), the viewer, the test...
xmipp: EM-reconstraction methods. Usually wrote in C++.
xmippCore: Low-level image-processing and EM-images definitions. Here there are the metadata_label.h/.cpp, metadata.h/.cpp and data.h/.cpp...
xmippViz: Basically ShowJ and Picker viewer. Java code.

First of all you need a new branch in all the repositories where add your changes. We strongly recommend to use the same name in all the repositories! Look at the simulated pull request and evaluate if your changes affect the five repositories or just one or two. Then navigate to the repositories that must be updated and create a new branch with an appropriated name and starting with your initials. In the case of the xmipp bundle, you can do it in one single step by

cd <xmipp_bundle>
./xmipp git checkout -b xx_new-classifying-protocol

and then in the scipion

cd <pluginized_scipion>
git checkout devel-pluginization  # to make sure that your new branch will be based on that
git checkout -b xx_new-classifying-protocol

Returning to the simulated pull request in GitHub, let's suppose that we have this 5 files to migrate: It's very important to merge the files instead of copy/paste it!!

config/templates/protocols.template: This file is genuine from Scipion. Then, the change has to be migrated to the scipion repository. At this point you can use pyCharm (recomended) or meld (or any merging tool). With pyCharm, open the pluginized scipion project and, at the project window (the files tree at left), find the config/templates/protocol.templete and right-click -> compare with -> <OLD_scipion>/config/templates/protocol.templete to compare the two files and take YOUR changes. Be careful to that!
pyworkflow/em/packages/xmipp3/protocol_blablabla.py: This file is the wrapper of the protocol, thus it belongs to the xmipp plugin. This file can be new or can be an improvement of an existing one. If it is completely new, just
```
cp <OLD_scipion>/pyworkflow/em/packages/xmipp3/protocol_blablabla.py <xmipp_bundle>/src/scipion-em-xmipp/protocols/protocol_blablabla.py
```
but if it is an improvement of something existing do not copy/paste, COMPARE!. Open the xmipp plugin project on pyCharm, search the xmipp3/protocols/protocol_blablabla.py at the files tree and right-click -> compare with -> <OLD_scipion>/pyworkflow/em/packages/xmipp3/protocol_blablabla.py and take YOUR changes. Be careful to that, you can use the Annotate tool to check when that lines were included.

The new structure can change the importing statements, thus take it into account.
pyworkflow/em/packages/xmipp3/viewer.py: Probably you have included a new behavior of the viewer for your protocol. Notice that for the viewers is a similar operation than for the protocols, but taking into account that now we have a dedicated viewers directory.

In this case, the new structure issues can be more evident because in the old structure the import was relative (since they was at the same directory) whereas now, we must import the protocol using from xmipp3.protocols import XmippProtBlaBlaBla.

software/em/xmipp/libraries/reconstruction/classification_blablabla.cpp: This belongs to the xmipp repository since it is a EM method. Again, if it is something completely new you can copy

cp <OLD_scipion>/software/em/xmipp/libraries/reconstruction/classification_blablabla.cpp <xmipp_bundle>/src/xmipp/libraries/reconstruction/classification_blablabla.cpp

but if it is an improvement of an existing file it must be merge. Using pyCharm (even being a C++ file) or using the meld program

meld <OLD_scipion>/software/em/xmipp/libraries/reconstruction/classification_blablabla.cpp <xmipp_bundle>/src/xmipp/libraries/reconstruction/classification_blablabla.cpp

software/em/xmipp/libraries/data/metadata_label.cpp: If you have added a new metadata label you must to included in the xmippCore repository. Thus, merge it
```
meld <OLD_scipion>/software/em/xmipp/libraries/data/metadata_label.cpp <xmipp_bundle>/src/xmippCore/core/metadata_label.cpp
```

After that, check that all works fine by running the test

cd <pluginized_scipion>
./scipion test xmipp3.tests.test_protocols_conteining_file.MyTest

and/or

./scipion [project MyTest]

If it is correct and you are happy with your updates, you can push it by

cd <pluginized_scipion>
git status
git add red-file-above-1 red-file-above-2 ...
git commit -m 'some message'
git push

cd <xmipp_bundle>
./xmipp git status    # look if some file is untracked! For instance: classification_blablabla.cpp
cd src/xmipp          # if its belong to another repo change this
git add ..../classification_blablabla.cpp
cd <xmipp_bundle>
./xmipp git commit -am 'some message'    # this will commit all the TRACKED files in red
./xmipp push

Xmipp developers hints

The main xmipp script is an static file since we copied/pasted before. However, it can be linked to the version controlled one (recommended) by

ln -s <xmipp_bundle>/src/xmipp/xmipp .

This script has several functions, ./xmipp --help to show it

$./xmipp --help
Usage: xmipp_installer [options]
   all [N=processors]:   (Default) Retrieve, configure, check, compile N=8, install in build
   get_devel_sources:    Retrieve development sources from github
   clean:                Delete source directories
   cleanBin:             Clean already compiled files
   config:               Configure compilation variables
   check_config:         Check that the configuration is correct
   compile N:            Compile all modules with N processors
   compile N xmippCore:  Compile xmippCore
   compile N xmipp:      Compile xmipp
   compile N xmippViz:   Compile xmippViz
   install dir:          Install at dir (by default, ./build)
   test testName:        Run tests to check Xmipp programs (by default, all).

For developers:
   create_devel_paths:   Create bashrc files for devel
   git ...:              Git command to all 4 repositories
   gitConfig:            Change the git config from https to git
   tar <mode> <version>: Create a bundle of the xmipp
                            <mode> can be 'Sources' or 'Binaries'
                            <version> usually X.YY.MM

Developed by xmipp team

Migrating branches from nonPluginized Scipion to the new Scipion Xmipp structure

Introduction

Getting ready in the new structure

Scipion

Xmipp

Xmipp installation

Xmipp plugin

Xmipp software

Code migration

Xmipp developers hints

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally