Pymode failing on opening a file

[blackdwarf]

I have updated my installations of gvim to 8.1 (with Pyhon 3) support as well as Python to 3.7 on my Windows machine. Re-cloned the python-mode repo and initialized the sub-components. However, when I try to open a Python file, I keep getting the following error:

Error detected while processing function pymode#breakpoint#init:
line   29:
Traceback (most recent call last):
  File "<string>", line 2, in <module>
  File "C:\Users\...\vimfiles\bundle\python-mode\pymode\libs\six.py", line 1
    ../../submodules/six/six.py
    ^
SyntaxError: invalid syntax

After I tried to do the debugging/troubleshooting steps from the README.md file, I couldn't get the debug file, but was able to save the following messages that appear:

Error detected while processing function pymode#init:
line    2:
E370: Could not load library python27.dll
E263: Sorry, this command is disabled, the Python library could not be loaded.
line    3:
E370: Could not load library python27.dll
E263: Sorry, this command is disabled, the Python library could not be loaded.
line    4:
E370: Could not load library python27.dll
E263: Sorry, this command is disabled, the Python library could not be loaded.
Error detected while processing C:\Users\...\vimfiles\bundle\python-mode\autoload\pymode\virtualenv.vim:
line    3:
E370: Could not load library python27.dll
E263: Sorry, this command is disabled, the Python library could not be loaded.
Error detected while processing function pymode#breakpoint#init:
line   29:
E370: Could not load library python27.dll
E263: Sorry, this command is disabled, the Python library could not be loaded.
Error detected while processing C:\Users\...\vimfiles\bundle\python-mode\ftplugin\python\pymode.vim:
line   25:
E370: Could not load library python27.dll
E263: Sorry, this command is disabled, the Python library could not be loaded.
line  239:
E370: Could not load library python27.dll
E263: Sorry, this command is disabled, the Python library could not be loaded.

Error detected while processing C:\Users\...\vimfiles\bundle\python-mode\autoload\pymode\lint.vim:
line    1:
E370: Could not load library python27.dll
Error detected while processing C:\Users\...\vimfiles\bundle\python-mode\autoload\pymode\lint.vim:
line    1:
E263: Sorry, this command is disabled, the Python library could not be loaded.

To me, this reads like I need to install Python 2.7? But I don't remember doing this step previously when python-mode was working, that is, I only ever had Python 3.x on my machine.

Does anyone know what is going on here?

[diraol]

Please, take a look at the issue #978

[blackdwarf]

@diraol thanks so much for the quick reply. Indeed, cloning again under administrative privileges with the -c core.symlinks=true solves the issue.

[artkpv]

Thanks for this investigation. Have this issue on Win8. Probably to add at https://github.com/python-mode/python-mode#how-to-install about -c core.symlinks=true for Win users.

[diraol]

Hi @artkpv could you please do a Pull Request with such contribution? You are right, it should be there! =)

[ieglandDRS]

Would put this in #978, but it's locked, so I figured I'd put it here.

Warning! Though this should work, it's not working for me. See below for more info.

Cygwin users will have to ensure that Windows-native symlinks are turned on before running the clone. If native symlinks are available, this should work. If they're not, the git clone will error out.

export CYGWIN=winsymlinks:nativestrict
cd ~/.vim/pack/python-mode/start
git clone --recurse-submodules https://github.com/python-mode/python-mode.git -c core.symlinks=true
cd python-mode

For a nice summary of the different Cygwin symlink modes, see here: https://gist.github.com/karlding/7868aac0c54fe94b7ba0a2061e0a4939
...and here:
https://stackoverflow.com/questions/24222591/check-the-difference-of-symlink-type-in-cygwin
...and here's the official documentation:
https://cygwin.com/cygwin-ug-net/using-cygwinenv.html

And here's some documentation on how git cloning with symlinks works:
https://stackoverflow.com/questions/11662868/what-happens-when-i-clone-a-repository-with-symlinks-on-windows?noredirect=1&lq=1
...and here's the official documentation:
https://github.com/git-for-windows/git/wiki/Symbolic-Links

[diraol]

Hi @ieglandDRS would you mind sending a Pull Request to add this tip into our README? :)

[ieglandDRS]

I'm open to it, but you may not want it. I'm still having troubles with it. When I get things worked out, I'll send one.

[ieglandDRS]

I dunno what else to try here. I've tried each Cygwin symlink mode, and I've verified that I can make a symlink from cygwin and from the command prompt, all under an administrator account. I can clone with the symlinks flag without error, too. But for some reason, Vim still complains about missing python27.dll.

Definitely not pull request ready, but I'm going to leave this and maybe come back to it. If someone else wants to pick it up, I left a link trail above.

[diraol]

@ieglandDRS have you tried what is described in our documentation?

[ieglandDRS]

I have. At least, I think I have.

This (taken from the README) runs in cygwin with no modifications:

cd ~/.vim/pack/python-mode/start
git clone --recurse-submodules https://github.com/python-mode/python-mode.git -c core.symlinks=true
cd python-mode

However, this doesn't exactly match my use case. I'm trying to install pymode for the windows version of Vim. By default, the windows version of vim looks for packages in the following places, is indicated by :set packpath:
packpath=~/vimfiles,C:\Program Files (x86)\Vim/vimfiles,C:\Program Files (x86)\Vim\vim81,C:\Program Files (x86)\Vim/vimfiles/after,~/vimfiles/after

Notice the use of ~/vimfiles in place of ~/.vim.

Additionally, ~ in the Windows Vim context means something different from the Cygwin context.

• In Vim for Windows, ~ is equivalent to the windows environment variable %UserProfile%, which expands to C:\Users\<username> on Windows 10. You can verify this by running :echo expand('~') in vim.
• In Cygwin, ~ means the value of $HOME, like it does in Bash.

So, to adapt for my use case, the above commands from the readme should be:

cd /cygdrive/c/users/user.name/vimfiles/pack/python-mode/start
git clone --recurse-submodules https://github.com/python-mode/python-mode.git -c core.symlinks=true
cd python-mode

Which runs just fine, successfully cloning the submodules. But when I open a python file with Vim for Windows, I get a ton of errors about being unable to load python27.dll... which led me down the whole is-it-using-the-right-symlinks rabbit hole. Unfortunately, I get the same error using any one of the three cygwin symlink settings, except nativestrict, which fails to clone the submodules at all:

export CYGWIN=winsymlinks:nativestrict
export CYGWIN=winsymlinks:native
export CYGWIN=winsymlinks:lnk

(As I write this, I realize that this may be more of a git-on-Cygwin issue than a getting-pymode-running issue.)

[diraol]

Hum.... eventually could this be a problem with your PYTHONPATH env variable? (not sure how things work in windows, but if the issue is related to python27.dll that would be my guess).

[ieglandDRS]

Great tip! I found the answer!

First, I found out how to interrogate vim about whether or not it has python available.

echo has('python')
echo has('python3')

A 1 as an answer here means that python is available. A 0 as an answer here means that python is unavailable. I had 0 for both!

Then I found this stackoverflow answer which says that, basically, if you want to use the 64-bit versions of Python 2 and Python 3 in Vim, then you have to have the 64-bit version of Vim installed.

I checked and found my version of vim to be the 32-bit version!

According to the vim website, if you download the Windows installer for vim you'll get a choice on what version to install. But that's not true for this release! I downloaded the 8.1 installer and it started installing the 32-bit version without giving me another option. So I decided to hunt down the 64 bit version on github.

Unfortunately, the Vim project doesn't tag their major releases, so I had to scroll past all the nightlies to get to the v8.1.0001 release. Lo and behold, there is no 64 bit installer! So I switched to the latest nightly release - v8.1.1826 - and used that 64-bit installer. Had to run it as administrator manually, but otherwise it installed fine.

Now I get 1s as answers to both echo has('python') and echo has('python3')! And the python27.dll errors are gone!

Hooray!

However, things still aren't quite perfect. Opening a python file in vim with pymode installed still produces an error.

Error detected while processing function pymode#breakpoint#init:
line    29:
Traceback (most recent call last):
  File "<string>", line 2, in <module>
  File "C:\Users\user.name\vimfiles\pack\python-mode\start\python-mode\pymode\libs\six.py", line 1
SyntaxError: Non-ASCII character '\xff' in file C:\Users\user.name\vimfiles\pack\python-mode\start\python-mode\pymode\libs\six.py on line 1, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

[ieglandDRS]

It looks like I wasn't crazy when I started exploring the different types of Cygwin symlinks.
The "no encoding declared" error is because pymode\libs\six.py isn't properly symlinked! If you view it in cygwin, it looks fine. If you view it in Windows - which is what Vim for Windows will see - you get this basic symlink stub file:

I'm going to try again with the other symlink types and post back.

[ieglandDRS]

It appears that none of the Cygwin symlink modes are compatible with Vim for Windows, even when it's run with administrator privileges. Every time I try it, I either get the above "no encoding declared" error, or a different error.

More broadly, this may mean that Windows symlinks aren't a viable option for Windows users of Vim.

[ieglandDRS]

Well well well! I think that Vim does support Windows native symlinks because they're supposed to be transparent to the programs that follow them. However, native symlinks are not being created successfully using nativestrict.

$ git clone --recurse-submodules https://github.com/python-mode/python-mode.git -c core.symlinks=true
Cloning into 'python-mode'...
remote: Enumerating objects: 42, done.
remote: Counting objects: 100% (42/42), done.
remote: Compressing objects: 100% (26/26), done.
remote: Total 8044 (delta 14), reused 32 (delta 12), pack-reused 8002
Receiving objects: 100% (8044/8044), 10.67 MiB | 1.25 MiB/s, done.
Resolving deltas: 100% (3234/3234), done.
error: unable to create symlink pymode/autopep8.py: No such file or directory
error: unable to create symlink pymode/libs/astroid: No such file or directory
error: unable to create symlink pymode/libs/logilab: No such file or directory
error: unable to create symlink pymode/libs/mccabe.py: No such file or directory
error: unable to create symlink pymode/libs/pycodestyle.py: No such file or directory
error: unable to create symlink pymode/libs/pydocstyle: No such file or directory
error: unable to create symlink pymode/libs/pyflakes: No such file or directory
error: unable to create symlink pymode/libs/pylama: No such file or directory
error: unable to create symlink pymode/libs/pylint: No such file or directory
error: unable to create symlink pymode/libs/rope: No such file or directory
error: unable to create symlink pymode/libs/six.py: No such file or directory
error: unable to create symlink pymode/libs/snowballstemmer: No such file or directory
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

In native mode, failures like this are coped with by creating a Cygwin link, which creates the "no encoding declared" error. The last mode, lnk just creates Windows shortcut files, which Vim can't reasonably be expected to interpret. So the key is to get The Real McCoy Windows Native Symlinks.

Let the treasure hunt begin!

Or continue...

[ieglandDRS]

Here's another potential interpretation of the "unable to create symlink...No such file or directory error":

When git clones a repository with submodules, it MAY go a little something like this:

1. Clone the respository's files.
2. Create the symlinks to the submodules where necessary
3. Clone the submodules where necessary

According to https://ss64.com/nt/mklink.html and https://ss64.com/bash/ln.html, when you create a symbolic link in windows (cmd) or linux (bash), the target that the link points to doesn't need to actually exist. So the above process is fine!

HOWEVER!

In cygwin, when CYGWIN=winsymlinks:nativestrict, the target must exist!

$ ln -sf /cygdrive/c/notreallythere
ln: failed to create symbolic link './notreallythere': No such file or directory

So it seems like you can have Windows Native Symlinks, but the target must already exist. But git, being built according to the normal usage constraints of ln, doesn't expect that restriction and fails out when it can't make the necessary symlinks.

[ieglandDRS]

And I'm not crazy! Someone else caught this, too!
https://sourceware.org/ml/cygwin/2018-09/msg00115.html

[ieglandDRS]

And before that guy, there was a much longer discussion that included a key difference between POSIX symlinks and NTFS symlinks - that POSIX symlinks can point to either a file or directory, while you have to specify whether an NTFS symlink points to a file or directory upon creation. In any case, however, one should be able to point a symlink to a non-existent target - but in Cygwin that functionality is broken.
https://sourceware.org/ml/cygwin/2016-04/msg00658.html

[ieglandDRS]

I made a stackoverflow post about this: https://superuser.com/questions/1469330/why-cant-i-clone-a-git-repository-with-submodules-in-cygwin-using-windows-nativ/1469332#1469332

[ieglandDRS]

The best way around this limitation seem to be to either:

1. Use this guy's script to augment git functionality. https://stackoverflow.com/a/16754068/5133587
2. Use this guy's script to figure out which files need to be copied and copy them manually, after cloning the repository without the native symlinks flag: https://stackoverflow.com/a/5930443/5133587