Skip to content

Setting the environment #8

Open
@luisfonseca97

Description

@luisfonseca97

Hi, I'm having problems running the bash tests/minimal_test.sh command. I am not sure why this is happening but I believe it starts when I try to set the environment.
I already added execution permissions for the set_environment.sh by running chmod a+x set_environment.sh but when I try to execute it, nothing seems to happen:

(base) luis@luis-BOHK-WAX9X:~/transition-amr-parser$ touch set_environment.sh
(base) luis@luis-BOHK-WAX9X:~/transition-amr-parser$ . set_environment.sh
(base) luis@luis-BOHK-WAX9X:~/transition-amr-parser$

Futhermore, running bash scripts/download_and_patch_fairseq.sh seems that it is also doing nothing:

(base) luis@luis-BOHK-WAX9X:~/transition-amr-parser$ bash scripts/download_and_patch_fairseq.sh
(base) luis@luis-BOHK-WAX9X:~/transition-amr-parser$ 

Everything is installed properly:

base) luis@luis-BOHK-WAX9X:~/transition-amr-parser$ python tests/correctly_installed.py

pytorch 1.8.1+cu102
fairseq 0.7.2
spacy 2.2.3
[OK] correctly installed

But when I run bash tests/minimal_test.sh it does not test (I will leave the full message for convenience).

[normalize rules] months
[normalize rules] units
[normalize rules] cardinals
[normalize rules] ordinals
Read DATA/wiki25.jkaln
25 sentences
216/293 node types/tokens
35/285 edge types/tokens
241/383 word types/tokens
AMR contains 4 duplicate edges
{'ARG1': 4}
2021-05-13 11:33:55 [amr] Processing oracle
2021-05-13 11:33:55 [oracle] Parsing data
computing oracle: 100%|█████████████████████████| 25/25 [00:00<00:00, 92.61it/s]
2021-05-13 11:33:56 [oracle] Done
Not whitelisted actions used e.g. arcs for unconfirmed words
Counter({'LA': 2})
Blacklisted actions used e.g. duplicated edges
Counter({'RA': 1})
There were 18 disconnected nodes (:rel)
2021-05-13 11:33:56 [Totals:] 0.61
2021-05-13 11:33:56 [Totals:] Failed Entity Predictions:
Namespace(alignfile=None, batch_normalize_reward=False, bert_layers=None, bpe=None, cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='DATA.tests/features/wiki25.jkaln/', entity_rules='DATA.tests/oracles/wiki25.jkaln//entity_rules.json', fp16=False, fp16_init_scale=128, fp16_scale_tolerance=0.0, fp16_scale_window=None, gold_annotations=None, gold_episode_ratio=None, joined_dictionary=False, log_format=None, log_interval=1000, lr_scheduler='fixed', machine_rules='DATA.tests/oracles/wiki25.jkaln//train.rules.json', machine_type='AMR', memory_efficient_fp16=False, min_loss_scale=0.0001, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, only_source=False, optimizer='nag', padding_factor=8, pretrained_embed='roberta.base', seed=1, source_lang='en', srcdict=None, target_lang='actions', task='translation', tbmf_wrapper=False, tensorboard_logdir='', testpref='DATA.tests/oracles/wiki25.jkaln//test', tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenize_by_whitespace=False, tokenizer=None, trainpref='DATA.tests/oracles/wiki25.jkaln//train', user_dir=None, validpref='DATA.tests/oracles/wiki25.jkaln//dev', workers=1)
| [en] Dictionary: 247 types
| [en] DATA.tests/oracles/wiki25.jkaln//train.en: 25 sents, 408 tokens, 0.0% replaced by <unk>
| [en] Dictionary: 247 types
| [en] DATA.tests/oracles/wiki25.jkaln//dev.en: 25 sents, 408 tokens, 0.0% replaced by <unk>
| [en] Dictionary: 247 types
| [en] DATA.tests/oracles/wiki25.jkaln//test.en: 25 sents, 408 tokens, 0.0% replaced by <unk>
| [actions] Dictionary: 127 types
| [actions] DATA.tests/oracles/wiki25.jkaln//train.actions: 25 sents, 1327 tokens, 0.0% replaced by <unk>
| [actions] Dictionary: 127 types
| [actions] DATA.tests/oracles/wiki25.jkaln//dev.actions: 25 sents, 1327 tokens, 0.0% replaced by <unk>
| [actions] Dictionary: 127 types
| [actions] DATA.tests/oracles/wiki25.jkaln//test.actions: 25 sents, 1327 tokens, 0.0% replaced by <unk>
Using cache found in /home/luis/.cache/torch/hub/pytorch_fairseq_master
fatal: not a git repository (or any of the parent directories): .git
running build_ext
/home/luis/anaconda3/lib/python3.8/site-packages/torch/utils/cpp_extension.py:369: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
  warnings.warn(msg.format('we could not find ninja.'))
skipping 'fairseq/data/data_utils_fast.cpp' Cython extension (up-to-date)
skipping 'fairseq/data/token_block_utils_fast.cpp' Cython extension (up-to-date)
copying build/lib.linux-x86_64-3.8/fairseq/libbleu.cpython-38-x86_64-linux-gnu.so -> fairseq
copying build/lib.linux-x86_64-3.8/fairseq/data/data_utils_fast.cpython-38-x86_64-linux-gnu.so -> fairseq/data
copying build/lib.linux-x86_64-3.8/fairseq/data/token_block_utils_fast.cpython-38-x86_64-linux-gnu.so -> fairseq/data
copying build/lib.linux-x86_64-3.8/fairseq/libbase.cpython-38-x86_64-linux-gnu.so -> fairseq
copying build/lib.linux-x86_64-3.8/fairseq/libnat.cpython-38-x86_64-linux-gnu.so -> fairseq
loading archive file http://dl.fbaipublicfiles.com/fairseq/models/roberta.base.tar.gz from cache at /home/luis/.cache/torch/pytorch_fairseq/37d2bc14cf6332d61ed5abeb579948e6054e46cc724c7d23426382d11a31b2d6.ae5852b4abc6bf762e0b6b30f19e741aa05562471e9eb8f4a6ae261f04f9b350
| dictionary: 50264 types
Using roberta.base extraction in cpu (slow, wont OOM)

25it [00:00, 84.66it/s]

There were missing actions
Counter({'LA(op1)': 1, 'LA(domain)': 1, 'RA(ARG1)': 1})
Using cache found in /home/luis/.cache/torch/hub/pytorch_fairseq_master
fatal: not a git repository (or any of the parent directories): .git
Traceback (most recent call last):
  File "/home/luis/.cache/torch/hub/pytorch_fairseq_master/hubconf.py", line 49, in <module>
    import fairseq.data.token_block_utils_fast  # noqa
ModuleNotFoundError: No module named 'fairseq.data.token_block_utils_fast'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/luis/anaconda3/lib/python3.8/site-packages/setuptools/sandbox.py", line 152, in save_modules
    yield saved
  File "/home/luis/anaconda3/lib/python3.8/site-packages/setuptools/sandbox.py", line 193, in setup_context
    yield
  File "/home/luis/anaconda3/lib/python3.8/site-packages/setuptools/sandbox.py", line 254, in run_setup
    _execfile(setup_script, ns)
  File "/home/luis/anaconda3/lib/python3.8/site-packages/setuptools/sandbox.py", line 43, in _execfile
    exec(code, globals, locals)
  File "/home/luis/.cache/torch/hub/pytorch_fairseq_master/setup.py", line 268, in <module>
    do_setup(package_data)
  File "/home/luis/.cache/torch/hub/pytorch_fairseq_master/setup.py", line 179, in do_setup
    setup(
  File "/home/luis/anaconda3/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/home/luis/anaconda3/lib/python3.8/distutils/core.py", line 134, in setup
    ok = dist.parse_command_line()
  File "/home/luis/anaconda3/lib/python3.8/distutils/dist.py", line 483, in parse_command_line
    args = self._parse_command_opts(parser, args)
  File "/home/luis/anaconda3/lib/python3.8/site-packages/setuptools/dist.py", line 903, in _parse_command_opts
    nargs = _Distribution._parse_command_opts(self, parser, args)
  File "/home/luis/anaconda3/lib/python3.8/distutils/dist.py", line 546, in _parse_command_opts
    raise DistutilsClassError(
distutils.errors.DistutilsClassError: command class <class 'torch.utils.cpp_extension.BuildExtension'> must subclass Command

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/luis/anaconda3/bin/fairseq-preprocess", line 33, in <module>
    sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')())
  File "/home/luis/transition-amr-parser/fairseq-stack-transformer/fairseq_cli/preprocess.py", line 295, in cli_main
    main(args)
  File "/home/luis/transition-amr-parser/fairseq-stack-transformer/fairseq_cli/preprocess.py", line 212, in main
    make_state_machine(args, src_dict, tgt_dict, tokenize=tokenize)
  File "/home/luis/transition-amr-parser/transition_amr_parser/stack_transformer/preprocess.py", line 300, in make_state_machine
    make_binary_bert_features(args, validpref, outprefix, src_dict.eos_index, src_dict.pad_index, tokenize)
  File "/home/luis/transition-amr-parser/transition_amr_parser/stack_transformer/preprocess.py", line 210, in make_binary_bert_features
    pretrained_embeddings = PretrainedEmbeddings(
  File "/home/luis/transition-amr-parser/transition_amr_parser/stack_transformer/pretrained_embeddings.py", line 180, in __init__
    self.roberta = torch.hub.load('pytorch/fairseq', name)
  File "/home/luis/anaconda3/lib/python3.8/site-packages/torch/hub.py", line 339, in load
    model = _load_local(repo_or_dir, model, *args, **kwargs)
  File "/home/luis/anaconda3/lib/python3.8/site-packages/torch/hub.py", line 365, in _load_local
    hub_module = import_module(MODULE_HUBCONF, hubconf_path)
  File "/home/luis/anaconda3/lib/python3.8/site-packages/torch/hub.py", line 74, in import_module
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/luis/.cache/torch/hub/pytorch_fairseq_master/hubconf.py", line 56, in <module>
    sandbox.run_setup(
  File "/home/luis/anaconda3/lib/python3.8/site-packages/setuptools/sandbox.py", line 257, in run_setup
    raise
  File "/home/luis/anaconda3/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/luis/anaconda3/lib/python3.8/site-packages/setuptools/sandbox.py", line 193, in setup_context
    yield
  File "/home/luis/anaconda3/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/luis/anaconda3/lib/python3.8/site-packages/setuptools/sandbox.py", line 164, in save_modules
    saved_exc.resume()
  File "/home/luis/anaconda3/lib/python3.8/site-packages/setuptools/sandbox.py", line 139, in resume
    raise exc.with_traceback(self._tb)
  File "/home/luis/anaconda3/lib/python3.8/site-packages/setuptools/sandbox.py", line 152, in save_modules
    yield saved
  File "/home/luis/anaconda3/lib/python3.8/site-packages/setuptools/sandbox.py", line 193, in setup_context
    yield
  File "/home/luis/anaconda3/lib/python3.8/site-packages/setuptools/sandbox.py", line 254, in run_setup
    _execfile(setup_script, ns)
  File "/home/luis/anaconda3/lib/python3.8/site-packages/setuptools/sandbox.py", line 43, in _execfile
    exec(code, globals, locals)
  File "/home/luis/.cache/torch/hub/pytorch_fairseq_master/setup.py", line 268, in <module>
    do_setup(package_data)
  File "/home/luis/.cache/torch/hub/pytorch_fairseq_master/setup.py", line 179, in do_setup
    setup(
  File "/home/luis/anaconda3/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/home/luis/anaconda3/lib/python3.8/distutils/core.py", line 134, in setup
    ok = dist.parse_command_line()
  File "/home/luis/anaconda3/lib/python3.8/distutils/dist.py", line 483, in parse_command_line
    args = self._parse_command_opts(parser, args)
  File "/home/luis/anaconda3/lib/python3.8/site-packages/setuptools/dist.py", line 903, in _parse_command_opts
    nargs = _Distribution._parse_command_opts(self, parser, args)
  File "/home/luis/anaconda3/lib/python3.8/distutils/dist.py", line 546, in _parse_command_opts
    raise DistutilsClassError(
distutils.errors.DistutilsClassError: command class <class 'torch.utils.cpp_extension.BuildExtension'> must subclass Command

Any idea why this might be happening?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions