Migrating branches from nonPluginized Scipion to the new Scipion Xmipp structure

Introduction

Scipion was allocating wrappers for the different packages (Xmipp, Relion, Eman...) until the v1.2.1 (included). From the ¿v2.0? (or v1.3) all this wrappers become external plugins allocated in independent repositories and they will be installed as pip modules. Thus, in this new version, the directory <scipion_home>/pyworkflow/em/packages disappears and the installed plugins are added to the python's site-packages folder as the other python modules.

On the other hand, the XMIPP software (C++ and Java sources) was also allocated in the Scipion repository, so far. Now, we have split the XMIPP software in three repositories independent from Scipion (XmippCore, XmippViz and Xmipp) plus the plugin repository (scipion-em-xmipp). All this repositories are in I2PC gitHub.

Getting ready in the new structure

Scipion

This is the easiest part. You just needs to checkout the devel branch. However, in order to keep a Scipion/Xmipp backup, we strongly recommend to make a fresh installation of Scipion in a new directory. You can place and name this directory as you want, in this tutorial it is ~/scipion2

cd ~
git clone https://github.com/I2PC/scipion scipion2
cd scipion2
git checkout devel

As it is a fresh installation, Scipion needs to be installed from top. You can follow the Scipion installation instructions, but basically

./scipion config
./scipion install -j N

where N is the number of the processors.

After a while, Scipion should be installed. You can lunch it by ./scipion but with a very limited features since no plugin is installed yet.

Xmipp Installation

See how to install Xmipp for developers.

Code migration

If we have code in some branches based on the devel branch of the non-pluginized Scipion, we need to migrate it to the new structure. At this point the old scipion dir that we kept in the system becomes very useful. We refer to that as <OLD_scipion>.

The goal is to bring all the changes from our working branch (in comparison with devel). To clearly see these changes, it is useful to simulate a pull request in GitHub against the devel branch. Thus, go to www.github.com/<your-user>/scipion (if you forked, if not from I2PC) and search the branch that you want to migrate. Then, click on New pull request and change the base to devel. Now, a list of changed files has to be shown.

This files may belong to scipion, scipion-em-xmipp, xmipp, xmippCore or xmippViz in the new structure. Thus, we need to identify where to migrate it.

scipion: Genuine scipion code. Very low-level things, such as config, protocols.config...
scipion-em-xmipp: It is the plugin. The protocol (wrapper), the viewer, the test...
xmipp: EM-reconstraction methods. Usually wrote in C++.
xmippCore: Low-level image-processing and EM-images definitions. Here there are the metadata_label.h/.cpp, metadata.h/.cpp and data.h/.cpp...
xmippViz: Basically ShowJ and Picker viewer. Java code.

First of all you need a new branch in all the repositories where add your changes. We strongly recommend to use the same name in all the repositories! Look at the simulated pull request and evaluate if your changes affect the five repositories or just one or two. Then navigate to the repositories that must be updated and create a new branch with an appropriated name and starting with your initials. In the case of the xmipp bundle, you can do it in one single step by

cd <xmipp_bundle>
./xmipp git checkout -b xx_new-feature

and then in the scipion

cd <pluginized_scipion>
git checkout devel-pluginization  # to make sure that your new branch will be based on that
git checkout -b xx_new-feature

Returning to the simulated pull request in GitHub, let's suppose that we have this 5 files to migrate: It's very important to merge the files instead of copy/paste it!!

config/templates/protocols.template: This file is genuine from Scipion. Then, the change has to be migrated to the scipion repository. At this point you can use pyCharm (recomended) or meld (or any merging tool). With pyCharm, open the pluginized scipion project and, at the project window (the files tree at left), find the config/templates/protocol.templete and right-click -> compare with -> <OLD_scipion>/config/templates/protocol.templete to compare the two files and take YOUR changes. Be careful to that!
pyworkflow/em/packages/xmipp3/protocol_blablabla.py: This file is the wrapper of the protocol, thus it belongs to the xmipp plugin. This file can be new or can be an improvement of an existing one. If it is completely new, just
```
cp <OLD_scipion>/pyworkflow/em/packages/xmipp3/protocol_blablabla.py <xmipp_bundle>/src/scipion-em-xmipp/protocols/protocol_blablabla.py
```
but if it is an improvement of something existing do not copy/paste, COMPARE!. Open the xmipp plugin project on pyCharm, search the xmipp3/protocols/protocol_blablabla.py at the files tree and right-click -> compare with -> <OLD_scipion>/pyworkflow/em/packages/xmipp3/protocol_blablabla.py and take YOUR changes. Be careful to that, you can use the Annotate tool to check when that lines were included.

The new structure can change the importing statements, thus take it into account.
pyworkflow/em/packages/xmipp3/viewer.py: Probably you have included a new behavior of the viewer for your protocol. Notice that for the viewers is a similar operation than for the protocols, but taking into account that now we have a dedicated viewers directory.

In this case, the new structure issues can be more evident because in the old structure the import was relative (since they was at the same directory) whereas now, we must import the protocol using from xmipp3.protocols import XmippProtBlaBlaBla.

software/em/xmipp/libraries/reconstruction/classification_blablabla.cpp: This belongs to the xmipp repository since it is a EM method. Again, if it is something completely new you can copy

cp <OLD_scipion>/software/em/xmipp/libraries/reconstruction/classification_blablabla.cpp <xmipp_bundle>/src/xmipp/libraries/reconstruction/classification_blablabla.cpp

but if it is an improvement of an existing file it must be merge. Using pyCharm (even being a C++ file) or using the meld program

meld <OLD_scipion>/software/em/xmipp/libraries/reconstruction/classification_blablabla.cpp <xmipp_bundle>/src/xmipp/libraries/reconstruction/classification_blablabla.cpp

software/em/xmipp/libraries/data/metadata_label.cpp: If you have added a new metadata label you must to included in the xmippCore repository. Thus, merge it
```
meld <OLD_scipion>/software/em/xmipp/libraries/data/metadata_label.cpp <xmipp_bundle>/src/xmippCore/core/metadata_label.cpp
```

After that, check that all works fine by running the test

cd <pluginized_scipion>
./scipion test xmipp3.tests.test_protocols_conteining_file.MyTest

and/or

./scipion [project MyTest]

If it is correct and you are happy with your updates, you can push it by

cd <pluginized_scipion>
git status
git add red-file-above-1 red-file-above-2 ...
git commit -m 'some message'
git push

cd <xmipp_bundle>
./xmipp git status    # look if some file is untracked! For instance: classification_blablabla.cpp
cd src/xmipp          # if its belong to another repo change this
git add ..../classification_blablabla.cpp
cd <xmipp_bundle>
./xmipp git commit -am 'some message'    # this will commit all the TRACKED files in red
./xmipp push

Xmipp developers hints

The main xmipp script is an static file since we copied/pasted before. However, it can be linked to the version controlled one (recommended) by

cd <xmipp_bundle>
ln -s <xmipp_bundle>/src/xmipp/xmipp .

This script has several functions, ./xmipp --help to show it

$./xmipp --help
Usage: xmipp_installer [options]
   all [N=processors]:   (Default) Retrieve, configure, check, compile N=8, install in build
   get_devel_sources:    Retrieve development sources from github
   clean:                Delete source directories
   cleanBin:             Clean already compiled files
   config:               Configure compilation variables
   check_config:         Check that the configuration is correct
   compile N:            Compile all modules with N processors
   compile N xmippCore:  Compile xmippCore
   compile N xmipp:      Compile xmipp
   compile N xmippViz:   Compile xmippViz
   install dir:          Install at dir (by default, ./build)
   test testName:        Run tests to check Xmipp programs (by default, all).

For developers:
   create_devel_paths:   Create bashrc files for devel
   git ...:              Git command to all 4 repositories
   gitConfig:            Change the git config from https to git
   tar <mode> <version>: Create a bundle of the xmipp
                            <mode> can be 'Sources' or 'Binaries'
                            <version> usually X.YY.MM

To recompile the code including what you are editing:

./xmipp [all N]  # whitout the [all N], 8 processors are used

To get the newest code:
```
./xmipp git status
```
check that you are in the correct branch of all the repos (master in xmippX and devel for the plugin)
```
./xmipp git pull
./xmipp [all N]  # whitout the [all N], 8 processors are used
```

Developed by xmipp team

Migrating branches from nonPluginized Scipion to the new Scipion Xmipp structure

Introduction

Getting ready in the new structure

Scipion

Xmipp Installation

Code migration

Xmipp developers hints

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally