UnicodeDecodeError with Python 3 on Window

[shinstar123]

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 165: character maps to

[elyase]

What was the text fragment that triggered the error?

[shinstar123]

i just ran this example
from geotext import GeoText

places = GeoText("London is a great city")
places.cities

GeoText('New York, Texas, and also China').country_mentions

and that issue comeout

[elyase]

Unfortunately I can't reproduce the issue, can you try installing in a fresh environment?

[snippsat]

My fix for someone else that had this problem.
Run with error:
http://pastebin.com/d0N7Q9cZ
Fix:
with open(filename, 'r') as f:
To:
with open(filename, 'r', encoding='utf-8') as f:
Test:

(geo_test) C:\Python36\geo_test
λ python
Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 07:18:10) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from geotext import GeoText
>>> places = GeoText("London is a great city")
>>> places.cities
['London']
 
>>> GeoText('New York, Texas, and also China').country_mentions
OrderedDict([('US', 2), ('CN', 1)])
 
>>> places = GeoText("Oslo is a great city")
>>> places.cities
['Oslo']

[iShekhar]

Error still seen when installing and running on Python 3.4

import geotext
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Program Files\JetBrains\PyCharm 2017.2.4\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Python34\lib\site-packages\geotext\__init__.py", line 7, in <module>
    from .geotext import GeoText
  File "C:\Program Files\JetBrains\PyCharm 2017.2.4\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Python34\lib\site-packages\geotext\geotext.py", line 87, in <module>
    class GeoText(object):
  File "C:\Python34\lib\site-packages\geotext\geotext.py", line 103, in GeoText
    index = build_index()
  File "C:\Python34\lib\site-packages\geotext\geotext.py", line 77, in build_index
    cities = read_table(get_data_path('cities15000.txt'), usecols=[1, 8])
  File "C:\Python34\lib\site-packages\geotext\geotext.py", line 54, in read_table
    for line in lines:
  File "C:\Python34\lib\site-packages\geotext\geotext.py", line 51, in <genexpr>
    lines = (line for line in f if not line.startswith(comment))
  File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 165: character maps to <undefined>```

[dovinmu]

I'm getting the same error just trying to import geotext, Python 3.6 on Windows 10 with Anaconda. Specifically this error: UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 165: character maps to <undefined> when the IncrementalDecoder tries to open the cities csv.

[dovinmu]

Fixed the problem by using Linux instead of Windows.

[iShekhar]

The question is to solve the issue on WINDOWS!!

[dovinmu]

I may have been being slightly snarky.

[elyase]

Can someone on Windows try again on master:

pip install https://github.com/elyase/geotext/archive/master.zip

?

[Ala1s]

Tried it, didn't solve the problem
snippsat's solution worked for me

[tschlach]

Having the same problem during the module import

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 165: character maps to <undefined>

[iwpnd]

@tschlach and snippsats solution did not work?

[tschlach]

@iwpnd It seems like snippsats solution suggests that the UnicodeDecodeError results from reading a text file without specifying the encoding.

I think the error that most of us are encounter comes on importing the library.

Here's my complete traceback of the error:

Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 12:30:02) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import geotext
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\tschlachter\AppData\Local\Continuum\Miniconda3\envs\language\lib\site-packages\geotext\__init__.py", line 7, in <module>
    from .geotext import GeoText
  File "C:\Users\tschlachter\AppData\Local\Continuum\Miniconda3\envs\language\lib\site-packages\geotext\geotext.py", line 87, in <module>
    class GeoText(object):
  File "C:\Users\tschlachter\AppData\Local\Continuum\Miniconda3\envs\language\lib\site-packages\geotext\geotext.py", line 103, in GeoText
    index = build_index()
  File "C:\Users\tschlachter\AppData\Local\Continuum\Miniconda3\envs\language\lib\site-packages\geotext\geotext.py", line 77, in build_index
    cities = read_table(get_data_path('cities15000.txt'), usecols=[1, 8])
  File "C:\Users\tschlachter\AppData\Local\Continuum\Miniconda3\envs\language\lib\site-packages\geotext\geotext.py", line 54, in read_table
    for line in lines:
  File "C:\Users\tschlachter\AppData\Local\Continuum\Miniconda3\envs\language\lib\site-packages\geotext\geotext.py", line 51, in <genexpr>
    lines = (line for line in f if not line.startswith(comment))
  File "C:\Users\tschlachter\AppData\Local\Continuum\Miniconda3\envs\language\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 165: character maps to <undefined>

[iwpnd]

@tschlach Yes, and if you look closely you see that read_table() is executed which is reading a text.

@elyase
I just created a clean venv and installed geotext via
pip install https://github.com/elyase/geotext/archive/master.zip

on a windows machine with python 3.6 and everything works as expected.