开源软件构建与测试

We truncated the diff of some files because they were too big. If you want to see the full diff for every file, click here.

Changes of Revision 2

_service:tar_scm:python-chardet.spec Changed

@@ -1,22 +1,23 @@
 %global _empty_manifest_terminate_build 0
 
 Name:           python-chardet
-Version:        4.0.0
-Release:        4
+Version:        5.1.0
+Release:        1
 Summary:        Universal encoding detector for Python 2 and 3
 License:        LGPL-2.1-only
 URL:            https://github.com/chardet/chardet
-Source0:        https://files.pythonhosted.org/packages/ee/2d/9cdc2b527e127b4c9db64b86647d567985940ac3698eeabc7ffaccb4ea61/chardet-4.0.0.tar.gz
+Source0:        https://files.pythonhosted.org/packages/source/c/chardet/chardet-%{version}.tar.gz
 BuildArch:      noarch
+
 %description
 Universal encoding detector for Python 2 and 3
 
 %package -n python3-chardet
 Summary:        Universal encoding detector for Python 2 and 3
-Provides:       python-chardet
+Provides:       python-chardet = %{version}-%{release}
 # Base build requires
 BuildRequires:  python3-devel
-BuildRequires:  python3-setuptools
+BuildRequires:  python3-pip python3-wheel python3-flit
 BuildRequires:  python3-pbr
 BuildRequires:  python3-pip
 BuildRequires:  python3-wheel
@@ -32,51 +33,41 @@
 Universal encoding detector for Python 2 and 3
 
 %prep
-%autosetup -n chardet-4.0.0 -p1
+%autosetup -n chardet-%{version} -p1
 
 %build
-%py3_build
+%pyproject_build
 
 %install
-%py3_install
-
+%pyproject_install
 install -d -m755 %{buildroot}/%{_pkgdocdir}
 if  -d doc ; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi
 if  -d docs ; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi
 if  -d example ; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi
-if  -d examples ; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi
-pushd %{buildroot}
-if  -d usr/lib ; then
-    find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst
-fi
-if  -d usr/lib64 ; then
-    find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst
-fi
-if  -d usr/bin ; then
-    find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst
-fi
-if  -d usr/sbin ; then
-    find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst
-fi
-touch doclist.lst
-if  -d usr/share/man ; then
-    find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst
-fi
-popd
-mv %{buildroot}/filelist.lst .
-mv %{buildroot}/doclist.lst .
 
 %check
-%{__python3} setup.py test
-
-%files -n python3-chardet -f filelist.lst
+%{__python3} -m pytest -v
 
-%dir %{python3_sitelib}/*
+%files -n python3-chardet
+%{_bindir}/chardetect
+%{python3_sitelib}/*
 
-%files help -f doclist.lst
+%files help
 %{_docdir}/*
 
 %changelog
+* Mon May 15 2023 jiangxinyu <jiangxinyu@kylinos.cn> - 5.1.0-1
+- Update package to version 5.1.0
+
+* Wed Feb 08 2023 zhuofeng <zhuofeng2@huawei.com> - 5.0.0-2
+- Type:bugfix
+- CVE:NA
+- SUG:NA
+- DESC:make the test check
+
+* Sat Oct 29 2022 zhuofeng <zhuofeng2@huawei.com> - 5.0.0-1
+- update to 5.0.0
+
 * Wed Feb 16 2022 liudabo <liudabo1@huawei.com> - 4.0.0-4
 - enable check

_service Changed

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/compat.py Deleted

@@ -1,36 +0,0 @@
-######################## BEGIN LICENSE BLOCK ########################
-# Contributor(s):
-#   Dan Blanchard
-#   Ian Cordasco
-#
-# This library is free software; you can redistribute it and/or
-# modify it under the terms of the GNU Lesser General Public
-# License as published by the Free Software Foundation; either
-# version 2.1 of the License, or (at your option) any later version.
-#
-# This library is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-# Lesser General Public License for more details.
-#
-# You should have received a copy of the GNU Lesser General Public
-# License along with this library; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
-# 02110-1301  USA
-######################### END LICENSE BLOCK #########################
-
-import sys
-
-
-if sys.version_info < (3, 0):
-    PY2 = True
-    PY3 = False
-    string_types = (str, unicode)
-    text_type = unicode
-    iteritems = dict.iteritems
-else:
-    PY2 = False
-    PY3 = True
-    string_types = (bytes, str)
-    text_type = str
-    iteritems = dict.items

_service:tar_scm:chardet-4.0.0.tar.gz/setup.py Deleted

@@ -1,53 +0,0 @@
-#!/usr/bin/env python
-
-from setuptools import find_packages, setup
-
-
-# Get version without importing, which avoids dependency issues
-def get_version():
-    import re
-    with open('chardet/version.py') as version_file:
-        return re.search(r"""__version__\s+=\s+('")(?P<version>.+?)\1""",
-                         version_file.read()).group('version')
-
-
-def readme():
-    with open('README.rst') as f:
-        return f.read()
-
-
-setup(name='chardet',
-      version=get_version(),
-      description='Universal encoding detector for Python 2 and 3',
-      long_description=readme(),
-      author='Mark Pilgrim',
-      author_email='mark@diveintomark.org',
-      maintainer='Daniel Blanchard',
-      maintainer_email='dan.blanchard@gmail.com',
-      url='https://github.com/chardet/chardet',
-      license="LGPL",
-      keywords='encoding', 'i18n', 'xml',
-      classifiers="Development Status :: 5 - Production/Stable",
-                   "Intended Audience :: Developers",
-                   ("License :: OSI Approved :: GNU Library or Lesser General"
-                    " Public License (LGPL)"),
-                   "Operating System :: OS Independent",
-                   "Programming Language :: Python",
-                   'Programming Language :: Python :: 2',
-                   'Programming Language :: Python :: 2.7',
-                   'Programming Language :: Python :: 3',
-                   'Programming Language :: Python :: 3.5',
-                   'Programming Language :: Python :: 3.6',
-                   'Programming Language :: Python :: 3.7',
-                   'Programming Language :: Python :: 3.8',
-                   'Programming Language :: Python :: 3.9',
-                   ('Programming Language :: Python :: Implementation :: '
-                    'CPython'),
-                   'Programming Language :: Python :: Implementation :: PyPy',
-                   ("Topic :: Software Development :: Libraries :: Python "
-                    "Modules"),
-                   "Topic :: Text Processing :: Linguistic",
-      packages=find_packages(),
-      python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*",
-      entry_points={'console_scripts':
-                    'chardetect = chardet.cli.chardetect:main'})

_service:tar_scm:chardet-4.0.0.tar.gz/tests/iso-8859-5-cyrillic Deleted

_service:tar_scm:chardet-4.0.0.tar.gz/tests/windows-1251-cyrillic Deleted

_service:tar_scm:chardet-4.0.0.tar.gz/LICENSE -> _service:tar_scm:chardet-5.1.0.tar.gz/LICENSE Changed

@@ -1,8 +1,8 @@
-		  GNU LESSER GENERAL PUBLIC LICENSE
-		       Version 2.1, February 1999
+                  GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 2.1, February 1999
 
  Copyright (C) 1991, 1999 Free Software Foundation, Inc.
-     51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
  Everyone is permitted to copy and distribute verbatim copies
  of this license document, but changing it is not allowed.
 
@@ -10,7 +10,7 @@
  as the successor of the GNU Library Public License, version 2, hence
  the version number 2.1.
 
-			    Preamble
+                            Preamble
 
   The licenses for most software are designed to take away your
 freedom to share and change it.  By contrast, the GNU General Public
@@ -112,7 +112,7 @@
 former contains code derived from the library, whereas the latter must
 be combined with the library in order to run.
 &#xc;
-		  GNU LESSER GENERAL PUBLIC LICENSE
+                  GNU LESSER GENERAL PUBLIC LICENSE
    TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
 
   0. This License Agreement applies to any software library or other
@@ -146,7 +146,7 @@
 on the Library (independent of the use of the Library in a tool for
 writing it).  Whether that is true depends on what the Library does
 and what the program that uses the Library does.
-  
+
   1. You may copy and distribute verbatim copies of the Library's
 complete source code as you receive it, in any medium, provided that
 you conspicuously and appropriately publish on each copy an
@@ -432,7 +432,7 @@
 of all derivatives of our free software and of promoting the sharing
 and reuse of software generally.
 
-			    NO WARRANTY
+                            NO WARRANTY
 
   15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO
 WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW.
@@ -455,7 +455,7 @@
 SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
 DAMAGES.
 
-		     END OF TERMS AND CONDITIONS
+                     END OF TERMS AND CONDITIONS
 &#xc;
            How to Apply These Terms to Your New Libraries
 
@@ -485,7 +485,7 @@
 
     You should have received a copy of the GNU Lesser General Public
     License along with this library; if not, write to the Free Software
-    Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+    Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
 
 Also add information on how to contact you by electronic and paper mail.
 
@@ -500,5 +500,3 @@
   Ty Coon, President of Vice
 
 That's all there is to it!
-
-

_service:tar_scm:chardet-4.0.0.tar.gz/MANIFEST.in -> _service:tar_scm:chardet-5.1.0.tar.gz/MANIFEST.in Changed

_service:tar_scm:chardet-4.0.0.tar.gz/PKG-INFO -> _service:tar_scm:chardet-5.1.0.tar.gz/PKG-INFO Changed

@@ -1,99 +1,97 @@
-Metadata-Version: 1.2
+Metadata-Version: 2.1
 Name: chardet
-Version: 4.0.0
-Summary: Universal encoding detector for Python 2 and 3
+Version: 5.1.0
+Summary: Universal encoding detector for Python 3
 Home-page: https://github.com/chardet/chardet
 Author: Mark Pilgrim
 Author-email: mark@diveintomark.org
 Maintainer: Daniel Blanchard
 Maintainer-email: dan.blanchard@gmail.com
 License: LGPL
-Description: Chardet: The Universal Character Encoding Detector
-        --------------------------------------------------
-        
-        .. image:: https://img.shields.io/travis/chardet/chardet/stable.svg
-           :alt: Build status
-           :target: https://travis-ci.org/chardet/chardet
-        
-        .. image:: https://img.shields.io/coveralls/chardet/chardet/stable.svg
-           :target: https://coveralls.io/r/chardet/chardet
-        
-        .. image:: https://img.shields.io/pypi/v/chardet.svg
-           :target: https://warehouse.python.org/project/chardet/
-           :alt: Latest version on PyPI
-        
-        .. image:: https://img.shields.io/pypi/l/chardet.svg
-           :alt: License
-        
-        
-        Detects
-         - ASCII, UTF-8, UTF-16 (2 variants), UTF-32 (4 variants)
-         - Big5, GB2312, EUC-TW, HZ-GB-2312, ISO-2022-CN (Traditional and Simplified Chinese)
-         - EUC-JP, SHIFT_JIS, CP932, ISO-2022-JP (Japanese)
-         - EUC-KR, ISO-2022-KR (Korean)
-         - KOI8-R, MacCyrillic, IBM855, IBM866, ISO-8859-5, windows-1251 (Cyrillic)
-         - ISO-8859-5, windows-1251 (Bulgarian)
-         - ISO-8859-1, windows-1252 (Western European languages)
-         - ISO-8859-7, windows-1253 (Greek)
-         - ISO-8859-8, windows-1255 (Visual and Logical Hebrew)
-         - TIS-620 (Thai)
-        
-        .. note::
-           Our ISO-8859-2 and windows-1250 (Hungarian) probers have been temporarily
-           disabled until we can retrain the models.
-        
-        Requires Python 2.7 or 3.5+.
-        
-        Installation
-        ------------
-        
-        Install from `PyPI <https://pypi.org/project/chardet/>`_::
-        
-            pip install chardet
-        
-        Documentation
-        -------------
-        
-        For users, docs are now available at https://chardet.readthedocs.io/.
-        
-        Command-line Tool
-        -----------------
-        
-        chardet comes with a command-line script which reports on the encodings of one
-        or more files::
-        
-            % chardetect somefile someotherfile
-            somefile: windows-1252 with confidence 0.5
-            someotherfile: ascii with confidence 1.0
-        
-        About
-        -----
-        
-        This is a continuation of Mark Pilgrim's excellent chardet. Previously, two
-        versions needed to be maintained: one that supported python 2.x and one that
-        supported python 3.x.  We've recently merged with `Ian Cordasco <https://github.com/sigmavirus24>`_'s
-        `charade <https://github.com/sigmavirus24/charade>`_ fork, so now we have one
-        coherent version that works for Python 2.7+ and 3.4+.
-        
-        :maintainer: Dan Blanchard
-        
+Project-URL: Documentation, https://chardet.readthedocs.io/
+Project-URL: GitHub Project, https://github.com/chardet/chardet
+Project-URL: Issue Tracker, https://github.com/chardet/chardet/issues
 Keywords: encoding,i18n,xml
-Platform: UNKNOWN
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
-Classifier: License :: OSI Approved :: GNU Library or Lesser General Public License (LGPL)
+Classifier: License :: OSI Approved :: GNU Lesser General Public License v2 or later (LGPLv2+)
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python
-Classifier: Programming Language :: Python :: 2
-Classifier: Programming Language :: Python :: 2.7
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.5
-Classifier: Programming Language :: Python :: 3.6
 Classifier: Programming Language :: Python :: 3.7
 Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: Implementation :: CPython
 Classifier: Programming Language :: Python :: Implementation :: PyPy
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Classifier: Topic :: Text Processing :: Linguistic
-Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*
+Requires-Python: >=3.7
+License-File: LICENSE
+
+Chardet: The Universal Character Encoding Detector
+--------------------------------------------------
+
+.. image:: https://img.shields.io/travis/chardet/chardet/stable.svg
+   :alt: Build status
+   :target: https://travis-ci.org/chardet/chardet
+
+.. image:: https://img.shields.io/coveralls/chardet/chardet/stable.svg
+   :target: https://coveralls.io/r/chardet/chardet
+
+.. image:: https://img.shields.io/pypi/v/chardet.svg
+   :target: https://warehouse.python.org/project/chardet/
+   :alt: Latest version on PyPI
+
+.. image:: https://img.shields.io/pypi/l/chardet.svg
+   :alt: License
+
+
+Detects
+ - ASCII, UTF-8, UTF-16 (2 variants), UTF-32 (4 variants)
+ - Big5, GB2312, EUC-TW, HZ-GB-2312, ISO-2022-CN (Traditional and Simplified Chinese)
+ - EUC-JP, SHIFT_JIS, CP932, ISO-2022-JP (Japanese)
+ - EUC-KR, ISO-2022-KR, Johab (Korean)
+ - KOI8-R, MacCyrillic, IBM855, IBM866, ISO-8859-5, windows-1251 (Cyrillic)
+ - ISO-8859-5, windows-1251 (Bulgarian)
+ - ISO-8859-1, windows-1252, MacRoman (Western European languages)
+ - ISO-8859-7, windows-1253 (Greek)
+ - ISO-8859-8, windows-1255 (Visual and Logical Hebrew)
+ - TIS-620 (Thai)
+
+.. note::
+   Our ISO-8859-2 and windows-1250 (Hungarian) probers have been temporarily
+   disabled until we can retrain the models.
+
+Requires Python 3.7+.
+
+Installation
+------------
+
+Install from `PyPI <https://pypi.org/project/chardet/>`_::
+
+    pip install chardet
+
+Documentation
+-------------
+
+For users, docs are now available at https://chardet.readthedocs.io/.
+
+Command-line Tool
+-----------------
+
+chardet comes with a command-line script which reports on the encodings of one
+or more files::
+
+    % chardetect somefile someotherfile
+    somefile: windows-1252 with confidence 0.5
+    someotherfile: ascii with confidence 1.0
+
+About
+-----
+
+This is a continuation of Mark Pilgrim's excellent original chardet port from C, and `Ian Cordasco <https://github.com/sigmavirus24>`_'s
+`charade <https://github.com/sigmavirus24/charade>`_ Python 3-compatible fork.
+
+:maintainer: Dan Blanchard

_service:tar_scm:chardet-4.0.0.tar.gz/README.rst -> _service:tar_scm:chardet-5.1.0.tar.gz/README.rst Changed

@@ -20,10 +20,10 @@
  - ASCII, UTF-8, UTF-16 (2 variants), UTF-32 (4 variants)
  - Big5, GB2312, EUC-TW, HZ-GB-2312, ISO-2022-CN (Traditional and Simplified Chinese)
  - EUC-JP, SHIFT_JIS, CP932, ISO-2022-JP (Japanese)
- - EUC-KR, ISO-2022-KR (Korean)
+ - EUC-KR, ISO-2022-KR, Johab (Korean)
  - KOI8-R, MacCyrillic, IBM855, IBM866, ISO-8859-5, windows-1251 (Cyrillic)
  - ISO-8859-5, windows-1251 (Bulgarian)
- - ISO-8859-1, windows-1252 (Western European languages)
+ - ISO-8859-1, windows-1252, MacRoman (Western European languages)
  - ISO-8859-7, windows-1253 (Greek)
  - ISO-8859-8, windows-1255 (Visual and Logical Hebrew)
  - TIS-620 (Thai)
@@ -32,7 +32,7 @@
    Our ISO-8859-2 and windows-1250 (Hungarian) probers have been temporarily
    disabled until we can retrain the models.
 
-Requires Python 2.7 or 3.5+.
+Requires Python 3.7+.
 
 Installation
 ------------
@@ -59,10 +59,7 @@
 About
 -----
 
-This is a continuation of Mark Pilgrim's excellent chardet. Previously, two
-versions needed to be maintained: one that supported python 2.x and one that
-supported python 3.x.  We've recently merged with `Ian Cordasco <https://github.com/sigmavirus24>`_'s
-`charade <https://github.com/sigmavirus24/charade>`_ fork, so now we have one
-coherent version that works for Python 2.7+ and 3.4+.
+This is a continuation of Mark Pilgrim's excellent original chardet port from C, and `Ian Cordasco <https://github.com/sigmavirus24>`_'s
+`charade <https://github.com/sigmavirus24/charade>`_ Python 3-compatible fork.
 
 :maintainer: Dan Blanchard

_service:tar_scm:chardet-4.0.0.tar.gz/chardet.egg-info/PKG-INFO -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet.egg-info/PKG-INFO Changed

_service:tar_scm:chardet-4.0.0.tar.gz/chardet.egg-info/SOURCES.txt -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet.egg-info/SOURCES.txt Changed

@@ -2,8 +2,8 @@
 MANIFEST.in
 NOTES.rst
 README.rst
+pyproject.toml
 setup.cfg
-setup.py
 test.py
 chardet/__init__.py
 chardet/big5freq.py
@@ -12,7 +12,7 @@
 chardet/charsetgroupprober.py
 chardet/charsetprober.py
 chardet/codingstatemachine.py
-chardet/compat.py
+chardet/codingstatemachinedict.py
 chardet/cp949prober.py
 chardet/enums.py
 chardet/escprober.py
@@ -26,6 +26,8 @@
 chardet/gb2312prober.py
 chardet/hebrewprober.py
 chardet/jisfreq.py
+chardet/johabfreq.py
+chardet/johabprober.py
 chardet/jpcntx.py
 chardet/langbulgarianmodel.py
 chardet/langgreekmodel.py
@@ -35,13 +37,17 @@
 chardet/langthaimodel.py
 chardet/langturkishmodel.py
 chardet/latin1prober.py
+chardet/macromanprober.py
 chardet/mbcharsetprober.py
 chardet/mbcsgroupprober.py
 chardet/mbcssm.py
+chardet/py.typed
+chardet/resultdict.py
 chardet/sbcharsetprober.py
 chardet/sbcsgroupprober.py
 chardet/sjisprober.py
 chardet/universaldetector.py
+chardet/utf1632prober.py
 chardet/utf8prober.py
 chardet/version.py
 chardet.egg-info/PKG-INFO
@@ -214,6 +220,9 @@
 tests/IBM866/music.peeps.ru.xml
 tests/IBM866/newsru.com.xml
 tests/IBM866/susu.ac.ru.xml
+tests/Johab/hlpro-readme.txt
+tests/Johab/iyagi-readme.txt
+tests/Johab/mdir-doc.txt
 tests/KOI8-R/_chromium_KOI8-R_with_no_encoding_specified.html
 tests/KOI8-R/_ude_1.txt
 tests/KOI8-R/aif.ru.health.xml
@@ -251,6 +260,7 @@
 tests/MacCyrillic/music.peeps.ru.xml
 tests/MacCyrillic/newsru.com.xml
 tests/MacCyrillic/susu.ac.ru.xml
+tests/MacRoman/ioreg_output.txt
 tests/SHIFT_JIS/10e.org.xml
 tests/SHIFT_JIS/1affliate.com.xml
 tests/SHIFT_JIS/_chromium_Shift-JIS_with_no_encoding_specified.html
@@ -289,8 +299,16 @@
 tests/TIS-620/trickspot.boxchart.com.xml
 tests/UTF-16/bom-utf-16-be.srt
 tests/UTF-16/bom-utf-16-le.srt
+tests/UTF-16BE/nobom-utf16be.txt
+tests/UTF-16BE/plane1-utf-16be.html
+tests/UTF-16LE/nobom-utf16le.txt
+tests/UTF-16LE/plane1-utf-16le.html
 tests/UTF-32/bom-utf-32-be.srt
 tests/UTF-32/bom-utf-32-le.srt
+tests/UTF-32BE/nobom-utf32be.txt
+tests/UTF-32BE/plane1-utf-32be.html
+tests/UTF-32LE/nobom-utf32le.txt
+tests/UTF-32LE/plane1-utf-32le.html
 tests/ascii/_chromium_iso-8859-1_with_no_encoding_specified.html
 tests/ascii/_mozilla_bug638318_text.html
 tests/ascii/howto.diveintomark.org.xml
@@ -303,6 +321,12 @@
 tests/iso-8859-1/_ude_4.txt
 tests/iso-8859-1/_ude_5.txt
 tests/iso-8859-1/_ude_6.txt
+tests/iso-8859-2-croatian/_ude_1.txt
+tests/iso-8859-2-czech/_ude_1.txt
+tests/iso-8859-2-czech/_ude_2.txt
+tests/iso-8859-2-hungarian/_ude_1.txt
+tests/iso-8859-2-hungarian/_ude_2.txt
+tests/iso-8859-2-hungarian/_ude_3.txt
 tests/iso-8859-2-hungarian/auto-apro.hu.xml
 tests/iso-8859-2-hungarian/cigartower.hu.xml
 tests/iso-8859-2-hungarian/escience.hu.xml
@@ -314,6 +338,10 @@
 tests/iso-8859-2-hungarian/shamalt.uw.hu.mv.xml
 tests/iso-8859-2-hungarian/shamalt.uw.hu.xml
 tests/iso-8859-2-hungarian/ugyanmar.blogspot.com.xml
+tests/iso-8859-2-polish/_ude_1.txt
+tests/iso-8859-2-slovak/_ude_1.txt
+tests/iso-8859-2-slovak/_ude_2.txt
+tests/iso-8859-2-slovene/_ude_1.txt
 tests/iso-8859-5-bulgarian/aero-bg.com.xml
 tests/iso-8859-5-bulgarian/bbc.co.uk.popshow.xml
 tests/iso-8859-5-bulgarian/bpm.cult.bg.2.xml
@@ -327,26 +355,29 @@
 tests/iso-8859-5-bulgarian/ecloga.cult.bg.xml
 tests/iso-8859-5-bulgarian/ide.li.xml
 tests/iso-8859-5-bulgarian/linux-bg.org.xml
-tests/iso-8859-5-cyrillic/_chromium_ISO-8859-5_with_no_encoding_specified.html
-tests/iso-8859-5-cyrillic/aif.ru.health.xml
-tests/iso-8859-5-cyrillic/aug32.hole.ru.xml
-tests/iso-8859-5-cyrillic/aviaport.ru.xml
-tests/iso-8859-5-cyrillic/blog.mlmaster.com.xml
-tests/iso-8859-5-cyrillic/forum.template-toolkit.ru.1.xml
-tests/iso-8859-5-cyrillic/forum.template-toolkit.ru.4.xml
-tests/iso-8859-5-cyrillic/forum.template-toolkit.ru.6.xml
-tests/iso-8859-5-cyrillic/forum.template-toolkit.ru.8.xml
-tests/iso-8859-5-cyrillic/forum.template-toolkit.ru.9.xml
-tests/iso-8859-5-cyrillic/greek.ru.xml
-tests/iso-8859-5-cyrillic/intertat.ru.xml
-tests/iso-8859-5-cyrillic/janulalife.blogspot.com.xml
-tests/iso-8859-5-cyrillic/kapranoff.ru.xml
-tests/iso-8859-5-cyrillic/money.rin.ru.xml
-tests/iso-8859-5-cyrillic/music.peeps.ru.xml
-tests/iso-8859-5-cyrillic/newsru.com.xml
-tests/iso-8859-5-cyrillic/susu.ac.ru.xml
+tests/iso-8859-5-russian/_chromium_ISO-8859-5_with_no_encoding_specified.html
+tests/iso-8859-5-russian/aif.ru.health.xml
+tests/iso-8859-5-russian/aug32.hole.ru.xml
+tests/iso-8859-5-russian/aviaport.ru.xml
+tests/iso-8859-5-russian/blog.mlmaster.com.xml
+tests/iso-8859-5-russian/forum.template-toolkit.ru.1.xml
+tests/iso-8859-5-russian/forum.template-toolkit.ru.4.xml
+tests/iso-8859-5-russian/forum.template-toolkit.ru.6.xml
+tests/iso-8859-5-russian/forum.template-toolkit.ru.8.xml
+tests/iso-8859-5-russian/forum.template-toolkit.ru.9.xml
+tests/iso-8859-5-russian/greek.ru.xml
+tests/iso-8859-5-russian/intertat.ru.xml
+tests/iso-8859-5-russian/janulalife.blogspot.com.xml
+tests/iso-8859-5-russian/kapranoff.ru.xml
+tests/iso-8859-5-russian/money.rin.ru.xml
+tests/iso-8859-5-russian/music.peeps.ru.xml
+tests/iso-8859-5-russian/newsru.com.xml
+tests/iso-8859-5-russian/susu.ac.ru.xml
 tests/iso-8859-6-arabic/_chromium_ISO-8859-6_with_no_encoding_specified.html
 tests/iso-8859-7-greek/_chromium_ISO-8859-7_with_no_encoding_specified.html
+tests/iso-8859-7-greek/_ude_1.txt
+tests/iso-8859-7-greek/_ude_2.txt
+tests/iso-8859-7-greek/_ude_3.txt
 tests/iso-8859-7-greek/_ude_greek.txt
 tests/iso-8859-7-greek/disabled.gr.xml
 tests/iso-8859-7-greek/hotstation.gr.xml
@@ -357,6 +388,8 @@
 tests/iso-8859-7-greek/naftemporiki.gr.mrt.xml
 tests/iso-8859-7-greek/naftemporiki.gr.spo.xml
 tests/iso-8859-7-greek/naftemporiki.gr.wld.xml
+tests/iso-8859-9-turkish/_ude_1.txt
+tests/iso-8859-9-turkish/_ude_2.txt
 tests/iso-8859-9-turkish/divxplanet.com.xml
 tests/iso-8859-9-turkish/subtitle.srt
 tests/iso-8859-9-turkish/wikitop_tr_ISO-8859-9.txt
@@ -381,12 +414,24 @@
 tests/utf-8/weblabor.hu.xml
 tests/utf-8-sig/_ude_4.txt
 tests/utf-8-sig/bom-utf-8.srt
+tests/windows-1250-croatian/_ude_1.txt
+tests/windows-1250-czech/_ude_1.txt
+tests/windows-1250-czech/_ude_2.txt
+tests/windows-1250-hungarian/_ude_1.txt
+tests/windows-1250-hungarian/_ude_2.txt
+tests/windows-1250-hungarian/_ude_3.txt
 tests/windows-1250-hungarian/bbc.co.uk.hu.forum.xml
 tests/windows-1250-hungarian/bbc.co.uk.hu.learningenglish.xml
 tests/windows-1250-hungarian/bbc.co.uk.hu.pressreview.xml
 tests/windows-1250-hungarian/bbc.co.uk.hu.xml
 tests/windows-1250-hungarian/objektivhir.hu.xml
 tests/windows-1250-hungarian/torokorszag.blogspot.com.xml
+tests/windows-1250-polish/_ude_1.txt
+tests/windows-1250-romanian/_ude_1.txt
+tests/windows-1250-slovak/_ude_1.txt
+tests/windows-1250-slovak/_ude_2.txt
+tests/windows-1250-slovak/_ude_3.txt
+tests/windows-1250-slovene/_ude_1.txt
 tests/windows-1251-bulgarian/bbc.co.uk.popshow.xml
 tests/windows-1251-bulgarian/bpm.cult.bg.2.xml
 tests/windows-1251-bulgarian/bpm.cult.bg.3.xml
@@ -403,28 +448,31 @@
 tests/windows-1251-bulgarian/informator.org.xml
 tests/windows-1251-bulgarian/linux-bg.org.xml
 tests/windows-1251-bulgarian/rinennor.org.xml
-tests/windows-1251-cyrillic/_chromium_windows-1251_with_no_encoding_specified.html
-tests/windows-1251-cyrillic/_ude_1.txt
-tests/windows-1251-cyrillic/aif.ru.health.xml
-tests/windows-1251-cyrillic/anthropology.ru.xml
-tests/windows-1251-cyrillic/aug32.hole.ru.xml
-tests/windows-1251-cyrillic/aviaport.ru.xml
-tests/windows-1251-cyrillic/blog.mlmaster.com.xml
-tests/windows-1251-cyrillic/forum.template-toolkit.ru.1.xml
-tests/windows-1251-cyrillic/forum.template-toolkit.ru.4.xml

_service:tar_scm:chardet-4.0.0.tar.gz/chardet.egg-info/entry_points.txt -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet.egg-info/entry_points.txt Changed

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/__init__.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/__init__.py Changed

@@ -15,69 +15,101 @@
 # 02110-1301  USA
 ######################### END LICENSE BLOCK #########################
 
+from typing import List, Union
 
-from .universaldetector import UniversalDetector
+from .charsetgroupprober import CharSetGroupProber
+from .charsetprober import CharSetProber
 from .enums import InputState
-from .version import __version__, VERSION
-
+from .resultdict import ResultDict
+from .universaldetector import UniversalDetector
+from .version import VERSION, __version__
 
-__all__ = 'UniversalDetector', 'detect', 'detect_all', '__version__', 'VERSION'
+__all__ = "UniversalDetector", "detect", "detect_all", "__version__", "VERSION"
 
 
-def detect(byte_str):
+def detect(
+    byte_str: Unionbytes, bytearray, should_rename_legacy: bool = False
+) -> ResultDict:
     """
     Detect the encoding of the given byte string.
 
     :param byte_str:     The byte sequence to examine.
     :type byte_str:      ``bytes`` or ``bytearray``
+    :param should_rename_legacy:  Should we rename legacy encodings
+                                  to their more modern equivalents?
+    :type should_rename_legacy:   ``bool``
     """
     if not isinstance(byte_str, bytearray):
         if not isinstance(byte_str, bytes):
-            raise TypeError('Expected object of type bytes or bytearray, got: '
-                            '{}'.format(type(byte_str)))
-        else:
-            byte_str = bytearray(byte_str)
-    detector = UniversalDetector()
+            raise TypeError(
+                f"Expected object of type bytes or bytearray, got: {type(byte_str)}"
+            )
+        byte_str = bytearray(byte_str)
+    detector = UniversalDetector(should_rename_legacy=should_rename_legacy)
     detector.feed(byte_str)
     return detector.close()
 
 
-def detect_all(byte_str):
+def detect_all(
+    byte_str: Unionbytes, bytearray,
+    ignore_threshold: bool = False,
+    should_rename_legacy: bool = False,
+) -> ListResultDict:
     """
     Detect all the possible encodings of the given byte string.
 
-    :param byte_str:     The byte sequence to examine.
-    :type byte_str:      ``bytes`` or ``bytearray``
+    :param byte_str:          The byte sequence to examine.
+    :type byte_str:           ``bytes`` or ``bytearray``
+    :param ignore_threshold:  Include encodings that are below
+                              ``UniversalDetector.MINIMUM_THRESHOLD``
+                              in results.
+    :type ignore_threshold:   ``bool``
+    :param should_rename_legacy:  Should we rename legacy encodings
+                                  to their more modern equivalents?
+    :type should_rename_legacy:   ``bool``
     """
     if not isinstance(byte_str, bytearray):
         if not isinstance(byte_str, bytes):
-            raise TypeError('Expected object of type bytes or bytearray, got: '
-                            '{}'.format(type(byte_str)))
-        else:
-            byte_str = bytearray(byte_str)
+            raise TypeError(
+                f"Expected object of type bytes or bytearray, got: {type(byte_str)}"
+            )
+        byte_str = bytearray(byte_str)
 
-    detector = UniversalDetector()
+    detector = UniversalDetector(should_rename_legacy=should_rename_legacy)
     detector.feed(byte_str)
     detector.close()
 
-    if detector._input_state == InputState.HIGH_BYTE:
-        results = 
-        for prober in detector._charset_probers:
-            if prober.get_confidence() > detector.MINIMUM_THRESHOLD:
-                charset_name = prober.charset_name
-                lower_charset_name = prober.charset_name.lower()
+    if detector.input_state == InputState.HIGH_BYTE:
+        results: ListResultDict = 
+        probers: ListCharSetProber = 
+        for prober in detector.charset_probers:
+            if isinstance(prober, CharSetGroupProber):
+                probers.extend(p for p in prober.probers)
+            else:
+                probers.append(prober)
+        for prober in probers:
+            if ignore_threshold or prober.get_confidence() > detector.MINIMUM_THRESHOLD:
+                charset_name = prober.charset_name or ""
+                lower_charset_name = charset_name.lower()
                 # Use Windows encoding name instead of ISO-8859 if we saw any
                 # extra Windows-specific bytes
-                if lower_charset_name.startswith('iso-8859'):
-                    if detector._has_win_bytes:
-                        charset_name = detector.ISO_WIN_MAP.get(lower_charset_name,
-                                                            charset_name)
-                results.append({
-                    'encoding': charset_name,
-                    'confidence': prober.get_confidence(),
-                    'language': prober.language,
-                })
+                if lower_charset_name.startswith("iso-8859") and detector.has_win_bytes:
+                    charset_name = detector.ISO_WIN_MAP.get(
+                        lower_charset_name, charset_name
+                    )
+                # Rename legacy encodings with superset encodings if asked
+                if should_rename_legacy:
+                    charset_name = detector.LEGACY_MAP.get(
+                        charset_name.lower(), charset_name
+                    )
+                results.append(
+                    {
+                        "encoding": charset_name,
+                        "confidence": prober.get_confidence(),
+                        "language": prober.language,
+                    }
+                )
         if len(results) > 0:
-            return sorted(results, key=lambda result: -result'confidence')
+            return sorted(results, key=lambda result: -result"confidence")
 
     return detector.result

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/big5freq.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/big5freq.py Changed

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/big5prober.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/big5prober.py Changed

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/chardistribution.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/chardistribution.py Changed

@@ -25,40 +25,58 @@
 # 02110-1301  USA
 ######################### END LICENSE BLOCK #########################
 
-from .euctwfreq import (EUCTW_CHAR_TO_FREQ_ORDER, EUCTW_TABLE_SIZE,
-                        EUCTW_TYPICAL_DISTRIBUTION_RATIO)
-from .euckrfreq import (EUCKR_CHAR_TO_FREQ_ORDER, EUCKR_TABLE_SIZE,
-                        EUCKR_TYPICAL_DISTRIBUTION_RATIO)
-from .gb2312freq import (GB2312_CHAR_TO_FREQ_ORDER, GB2312_TABLE_SIZE,
-                         GB2312_TYPICAL_DISTRIBUTION_RATIO)
-from .big5freq import (BIG5_CHAR_TO_FREQ_ORDER, BIG5_TABLE_SIZE,
-                       BIG5_TYPICAL_DISTRIBUTION_RATIO)
-from .jisfreq import (JIS_CHAR_TO_FREQ_ORDER, JIS_TABLE_SIZE,
-                      JIS_TYPICAL_DISTRIBUTION_RATIO)
-
-
-class CharDistributionAnalysis(object):
+from typing import Tuple, Union
+
+from .big5freq import (
+    BIG5_CHAR_TO_FREQ_ORDER,
+    BIG5_TABLE_SIZE,
+    BIG5_TYPICAL_DISTRIBUTION_RATIO,
+)
+from .euckrfreq import (
+    EUCKR_CHAR_TO_FREQ_ORDER,
+    EUCKR_TABLE_SIZE,
+    EUCKR_TYPICAL_DISTRIBUTION_RATIO,
+)
+from .euctwfreq import (
+    EUCTW_CHAR_TO_FREQ_ORDER,
+    EUCTW_TABLE_SIZE,
+    EUCTW_TYPICAL_DISTRIBUTION_RATIO,
+)
+from .gb2312freq import (
+    GB2312_CHAR_TO_FREQ_ORDER,
+    GB2312_TABLE_SIZE,
+    GB2312_TYPICAL_DISTRIBUTION_RATIO,
+)
+from .jisfreq import (
+    JIS_CHAR_TO_FREQ_ORDER,
+    JIS_TABLE_SIZE,
+    JIS_TYPICAL_DISTRIBUTION_RATIO,
+)
+from .johabfreq import JOHAB_TO_EUCKR_ORDER_TABLE
+
+
+class CharDistributionAnalysis:
     ENOUGH_DATA_THRESHOLD = 1024
     SURE_YES = 0.99
     SURE_NO = 0.01
     MINIMUM_DATA_THRESHOLD = 3
 
-    def __init__(self):
+    def __init__(self) -> None:
         # Mapping table to get frequency order from char order (get from
         # GetOrder())
-        self._char_to_freq_order = None
-        self._table_size = None  # Size of above table
+        self._char_to_freq_order: Tupleint, ... = tuple()
+        self._table_size = 0  # Size of above table
         # This is a constant value which varies from language to language,
         # used in calculating confidence.  See
         # http://www.mozilla.org/projects/intl/UniversalCharsetDetection.html
         # for further detail.
-        self.typical_distribution_ratio = None
-        self._done = None
-        self._total_chars = None
-        self._freq_chars = None
+        self.typical_distribution_ratio = 0.0
+        self._done = False
+        self._total_chars = 0
+        self._freq_chars = 0
         self.reset()
 
-    def reset(self):
+    def reset(self) -> None:
         """reset analyser, clear any state"""
         # If this flag is set to True, detection is done and conclusion has
         # been made
@@ -67,7 +85,7 @@
         # The number of characters whose frequency order is less than 512
         self._freq_chars = 0
 
-    def feed(self, char, char_len):
+    def feed(self, char: Unionbytes, bytearray, char_len: int) -> None:
         """feed a character with known length"""
         if char_len == 2:
             # we only care about 2-bytes character in our distribution analysis
@@ -81,7 +99,7 @@
                 if 512 > self._char_to_freq_orderorder:
                     self._freq_chars += 1
 
-    def get_confidence(self):
+    def get_confidence(self) -> float:
         """return confidence based on existing data"""
         # if we didn't receive any character in our consideration range,
         # return negative answer
@@ -89,20 +107,21 @@
             return self.SURE_NO
 
         if self._total_chars != self._freq_chars:
-            r = (self._freq_chars / ((self._total_chars - self._freq_chars)
-                 * self.typical_distribution_ratio))
+            r = self._freq_chars / (
+                (self._total_chars - self._freq_chars) * self.typical_distribution_ratio
+            )
             if r < self.SURE_YES:
                 return r
 
         # normalize confidence (we don't want to be 100% sure)
         return self.SURE_YES
 
-    def got_enough_data(self):
+    def got_enough_data(self) -> bool:
         # It is not necessary to receive all data to draw conclusion.
         # For charset detection, certain amount of data is enough
         return self._total_chars > self.ENOUGH_DATA_THRESHOLD
 
-    def get_order(self, byte_str):
+    def get_order(self, _: Unionbytes, bytearray) -> int:
         # We do not handle characters based on the original encoding string,
         # but convert this encoding string to a number, here called order.
         # This allows multiple encodings of a language to share one frequency
@@ -111,13 +130,13 @@
 
 
 class EUCTWDistributionAnalysis(CharDistributionAnalysis):
-    def __init__(self):
-        super(EUCTWDistributionAnalysis, self).__init__()
+    def __init__(self) -> None:
+        super().__init__()
         self._char_to_freq_order = EUCTW_CHAR_TO_FREQ_ORDER
         self._table_size = EUCTW_TABLE_SIZE
         self.typical_distribution_ratio = EUCTW_TYPICAL_DISTRIBUTION_RATIO
 
-    def get_order(self, byte_str):
+    def get_order(self, byte_str: Unionbytes, bytearray) -> int:
         # for euc-TW encoding, we are interested
         #   first  byte range: 0xc4 -- 0xfe
         #   second byte range: 0xa1 -- 0xfe
@@ -125,18 +144,17 @@
         first_char = byte_str0
         if first_char >= 0xC4:
             return 94 * (first_char - 0xC4) + byte_str1 - 0xA1
-        else:
-            return -1
+        return -1
 
 
 class EUCKRDistributionAnalysis(CharDistributionAnalysis):
-    def __init__(self):
-        super(EUCKRDistributionAnalysis, self).__init__()
+    def __init__(self) -> None:
+        super().__init__()
         self._char_to_freq_order = EUCKR_CHAR_TO_FREQ_ORDER
         self._table_size = EUCKR_TABLE_SIZE
         self.typical_distribution_ratio = EUCKR_TYPICAL_DISTRIBUTION_RATIO
 
-    def get_order(self, byte_str):
+    def get_order(self, byte_str: Unionbytes, bytearray) -> int:
         # for euc-KR encoding, we are interested
         #   first  byte range: 0xb0 -- 0xfe
         #   second byte range: 0xa1 -- 0xfe
@@ -144,18 +162,32 @@
         first_char = byte_str0
         if first_char >= 0xB0:
             return 94 * (first_char - 0xB0) + byte_str1 - 0xA1
-        else:
-            return -1
+        return -1
+
+
+class JOHABDistributionAnalysis(CharDistributionAnalysis):
+    def __init__(self) -> None:
+        super().__init__()
+        self._char_to_freq_order = EUCKR_CHAR_TO_FREQ_ORDER
+        self._table_size = EUCKR_TABLE_SIZE
+        self.typical_distribution_ratio = EUCKR_TYPICAL_DISTRIBUTION_RATIO
+
+    def get_order(self, byte_str: Unionbytes, bytearray) -> int:
+        first_char = byte_str0
+        if 0x88 <= first_char < 0xD4:
+            code = first_char * 256 + byte_str1
+            return JOHAB_TO_EUCKR_ORDER_TABLE.get(code, -1)
+        return -1
 
 
 class GB2312DistributionAnalysis(CharDistributionAnalysis):
-    def __init__(self):
-        super(GB2312DistributionAnalysis, self).__init__()
+    def __init__(self) -> None:
+        super().__init__()
         self._char_to_freq_order = GB2312_CHAR_TO_FREQ_ORDER
         self._table_size = GB2312_TABLE_SIZE
         self.typical_distribution_ratio = GB2312_TYPICAL_DISTRIBUTION_RATIO
 
-    def get_order(self, byte_str):
+    def get_order(self, byte_str: Unionbytes, bytearray) -> int:
         # for GB2312 encoding, we are interested

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/charsetgroupprober.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/charsetgroupprober.py Changed

@@ -25,29 +25,30 @@
 # 02110-1301  USA
 ######################### END LICENSE BLOCK #########################
 
-from .enums import ProbingState
+from typing import List, Optional, Union
+
 from .charsetprober import CharSetProber
+from .enums import LanguageFilter, ProbingState
 
 
 class CharSetGroupProber(CharSetProber):
-    def __init__(self, lang_filter=None):
-        super(CharSetGroupProber, self).__init__(lang_filter=lang_filter)
+    def __init__(self, lang_filter: LanguageFilter = LanguageFilter.NONE) -> None:
+        super().__init__(lang_filter=lang_filter)
         self._active_num = 0
-        self.probers = 
-        self._best_guess_prober = None
+        self.probers: ListCharSetProber = 
+        self._best_guess_prober: OptionalCharSetProber = None
 
-    def reset(self):
-        super(CharSetGroupProber, self).reset()
+    def reset(self) -> None:
+        super().reset()
         self._active_num = 0
         for prober in self.probers:
-            if prober:
-                prober.reset()
-                prober.active = True
-                self._active_num += 1
+            prober.reset()
+            prober.active = True
+            self._active_num += 1
         self._best_guess_prober = None
 
     @property
-    def charset_name(self):
+    def charset_name(self) -> Optionalstr:
         if not self._best_guess_prober:
             self.get_confidence()
             if not self._best_guess_prober:
@@ -55,17 +56,15 @@
         return self._best_guess_prober.charset_name
 
     @property
-    def language(self):
+    def language(self) -> Optionalstr:
         if not self._best_guess_prober:
             self.get_confidence()
             if not self._best_guess_prober:
                 return None
         return self._best_guess_prober.language
 
-    def feed(self, byte_str):
+    def feed(self, byte_str: Unionbytes, bytearray) -> ProbingState:
         for prober in self.probers:
-            if not prober:
-                continue
             if not prober.active:
                 continue
             state = prober.feed(byte_str)
@@ -75,7 +74,7 @@
                 self._best_guess_prober = prober
                 self._state = ProbingState.FOUND_IT
                 return self.state
-            elif state == ProbingState.NOT_ME:
+            if state == ProbingState.NOT_ME:
                 prober.active = False
                 self._active_num -= 1
                 if self._active_num <= 0:
@@ -83,22 +82,22 @@
                     return self.state
         return self.state
 
-    def get_confidence(self):
+    def get_confidence(self) -> float:
         state = self.state
         if state == ProbingState.FOUND_IT:
             return 0.99
-        elif state == ProbingState.NOT_ME:
+        if state == ProbingState.NOT_ME:
             return 0.01
         best_conf = 0.0
         self._best_guess_prober = None
         for prober in self.probers:
-            if not prober:
-                continue
             if not prober.active:
-                self.logger.debug('%s not active', prober.charset_name)
+                self.logger.debug("%s not active", prober.charset_name)
                 continue
             conf = prober.get_confidence()
-            self.logger.debug('%s %s confidence = %s', prober.charset_name, prober.language, conf)
+            self.logger.debug(
+                "%s %s confidence = %s", prober.charset_name, prober.language, conf
+            )
             if best_conf < conf:
                 best_conf = conf
                 self._best_guess_prober = prober

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/charsetprober.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/charsetprober.py Changed

@@ -28,54 +28,62 @@
 
 import logging
 import re
+from typing import Optional, Union
 
-from .enums import ProbingState
+from .enums import LanguageFilter, ProbingState
 
+INTERNATIONAL_WORDS_PATTERN = re.compile(
+    b"a-zA-Z*\x80-\xFF+a-zA-Z*^a-zA-Z\x80-\xFF?"
+)
 
-class CharSetProber(object):
+
+class CharSetProber:
 
     SHORTCUT_THRESHOLD = 0.95
 
-    def __init__(self, lang_filter=None):
-        self._state = None
+    def __init__(self, lang_filter: LanguageFilter = LanguageFilter.NONE) -> None:
+        self._state = ProbingState.DETECTING
+        self.active = True
         self.lang_filter = lang_filter
         self.logger = logging.getLogger(__name__)
 
-    def reset(self):
+    def reset(self) -> None:
         self._state = ProbingState.DETECTING
 
     @property
-    def charset_name(self):
+    def charset_name(self) -> Optionalstr:
         return None
 
-    def feed(self, buf):
-        pass
+    @property
+    def language(self) -> Optionalstr:
+        raise NotImplementedError
+
+    def feed(self, byte_str: Unionbytes, bytearray) -> ProbingState:
+        raise NotImplementedError
 
     @property
-    def state(self):
+    def state(self) -> ProbingState:
         return self._state
 
-    def get_confidence(self):
+    def get_confidence(self) -> float:
         return 0.0
 
     @staticmethod
-    def filter_high_byte_only(buf):
-        buf = re.sub(b'(\x00-\x7F)+', b' ', buf)
+    def filter_high_byte_only(buf: Unionbytes, bytearray) -> bytes:
+        buf = re.sub(b"(\x00-\x7F)+", b" ", buf)
         return buf
 
     @staticmethod
-    def filter_international_words(buf):
+    def filter_international_words(buf: Unionbytes, bytearray) -> bytearray:
         """
         We define three types of bytes:
         alphabet: english alphabets a-zA-Z
         international: international characters \x80-\xFF
         marker: everything else ^a-zA-Z\x80-\xFF
-
         The input buffer can be thought to contain a series of words delimited
         by markers. This function works to filter all words that contain at
         least one international character. All contiguous sequences of markers
         are replaced by a single space ascii character.
-
         This filter applies to all scripts which do not use English characters.
         """
         filtered = bytearray()
@@ -83,8 +91,7 @@
         # This regex expression filters out only words that have at-least one
         # international character. The word may include one marker character at
         # the end.
-        words = re.findall(b'a-zA-Z*\x80-\xFF+a-zA-Z*^a-zA-Z\x80-\xFF?',
-                           buf)
+        words = INTERNATIONAL_WORDS_PATTERN.findall(buf)
 
         for word in words:
             filtered.extend(word:-1)
@@ -94,20 +101,17 @@
             # similarly across all languages and may thus have similar
             # frequencies).
             last_char = word-1:
-            if not last_char.isalpha() and last_char < b'\x80':
-                last_char = b' '
+            if not last_char.isalpha() and last_char < b"\x80":
+                last_char = b" "
             filtered.extend(last_char)
 
         return filtered
 
     @staticmethod
-    def filter_with_english_letters(buf):
+    def remove_xml_tags(buf: Unionbytes, bytearray) -> bytes:
         """
         Returns a copy of ``buf`` that retains only the sequences of English
         alphabet and high byte characters that are not between <> characters.
-        Also retains English alphabet and high byte characters immediately
-        before occurrences of >.
-
         This filter can be applied to all scripts which contain both English
         characters and extended ASCII characters, but is currently only used by
         ``Latin1Prober``.
@@ -115,26 +119,24 @@
         filtered = bytearray()
         in_tag = False
         prev = 0
+        buf = memoryview(buf).cast("c")
 
-        for curr in range(len(buf)):
-            # Slice here to get bytes instead of an int with Python 3
-            buf_char = bufcurr:curr + 1
-            # Check if we're coming out of or entering an HTML tag
-            if buf_char == b'>':
-                in_tag = False
-            elif buf_char == b'<':
-                in_tag = True
+        for curr, buf_char in enumerate(buf):
+            # Check if we're coming out of or entering an XML tag
 
-            # If current character is not extended-ASCII and not alphabetic...
-            if buf_char < b'\x80' and not buf_char.isalpha():
-                # ...and we're not in a tag
+            # https://github.com/python/typeshed/issues/8182
+            if buf_char == b">":  # type: ignorecomparison-overlap
+                prev = curr + 1
+                in_tag = False
+            # https://github.com/python/typeshed/issues/8182
+            elif buf_char == b"<":  # type: ignorecomparison-overlap
                 if curr > prev and not in_tag:
                     # Keep everything after last non-extended-ASCII,
                     # non-alphabetic character
                     filtered.extend(bufprev:curr)
                     # Output a space to delimit stretch we kept
-                    filtered.extend(b' ')
-                prev = curr + 1
+                    filtered.extend(b" ")
+                in_tag = True
 
         # If we're not in a tag...
         if not in_tag:

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/cli/__init__.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/cli/__init__.py Changed

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/cli/chardetect.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/cli/chardetect.py Changed

@@ -12,17 +12,21 @@
 
 """
 
-from __future__ import absolute_import, print_function, unicode_literals
 
 import argparse
 import sys
+from typing import Iterable, List, Optional
 
-from chardet import __version__
-from chardet.compat import PY2
-from chardet.universaldetector import UniversalDetector
+from .. import __version__
+from ..universaldetector import UniversalDetector
 
 
-def description_of(lines, name='stdin'):
+def description_of(
+    lines: Iterablebytes,
+    name: str = "stdin",
+    minimal: bool = False,
+    should_rename_legacy: bool = False,
+) -> Optionalstr:
     """
     Return a string describing the probable encoding of a file or
     list of strings.
@@ -31,8 +35,11 @@
     :type lines: Iterable of bytes
     :param name: Name of file or collection of lines
     :type name: str
+    :param should_rename_legacy:  Should we rename legacy encodings to
+                                  their more modern equivalents?
+    :type should_rename_legacy:   ``bool``
     """
-    u = UniversalDetector()
+    u = UniversalDetector(should_rename_legacy=should_rename_legacy)
     for line in lines:
         line = bytearray(line)
         u.feed(line)
@@ -41,16 +48,14 @@
             break
     u.close()
     result = u.result
-    if PY2:
-        name = name.decode(sys.getfilesystemencoding(), 'ignore')
-    if result'encoding':
-        return '{}: {} with confidence {}'.format(name, result'encoding',
-                                                     result'confidence')
-    else:
-        return '{}: no result'.format(name)
+    if minimal:
+        return result"encoding"
+    if result"encoding":
+        return f'{name}: {result"encoding"} with confidence {result"confidence"}'
+    return f"{name}: no result"
 
 
-def main(argv=None):
+def main(argv: OptionalListstr = None) -> None:
     """
     Handles command line arguments and gets things started.
 
@@ -60,25 +65,48 @@
     """
     # Get command line arguments
     parser = argparse.ArgumentParser(
-        description="Takes one or more file paths and reports their detected \
-                     encodings")
-    parser.add_argument('input',
-                        help='File whose encoding we would like to determine. \
-                              (default: stdin)',
-                        type=argparse.FileType('rb'), nargs='*',
-                        default=sys.stdin if PY2 else sys.stdin.buffer)
-    parser.add_argument('--version', action='version',
-                        version='%(prog)s {}'.format(__version__))
+        description=(
+            "Takes one or more file paths and reports their detected encodings"
+        )
+    )
+    parser.add_argument(
+        "input",
+        help="File whose encoding we would like to determine. (default: stdin)",
+        type=argparse.FileType("rb"),
+        nargs="*",
+        default=sys.stdin.buffer,
+    )
+    parser.add_argument(
+        "--minimal",
+        help="Print only the encoding to standard output",
+        action="store_true",
+    )
+    parser.add_argument(
+        "-l",
+        "--legacy",
+        help="Rename legacy encodings to more modern ones.",
+        action="store_true",
+    )
+    parser.add_argument(
+        "--version", action="version", version=f"%(prog)s {__version__}"
+    )
     args = parser.parse_args(argv)
 
     for f in args.input:
         if f.isatty():
-            print("You are running chardetect interactively. Press " +
-                  "CTRL-D twice at the start of a blank line to signal the " +
-                  "end of your input. If you want help, run chardetect " +
-                  "--help\n", file=sys.stderr)
-        print(description_of(f, f.name))
-
-
-if __name__ == '__main__':
+            print(
+                "You are running chardetect interactively. Press "
+                "CTRL-D twice at the start of a blank line to signal the "
+                "end of your input. If you want help, run chardetect "
+                "--help\n",
+                file=sys.stderr,
+            )
+        print(
+            description_of(
+                f, f.name, minimal=args.minimal, should_rename_legacy=args.legacy
+            )
+        )
+
+
+if __name__ == "__main__":
     main()

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/codingstatemachine.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/codingstatemachine.py Changed

@@ -27,10 +27,11 @@
 
 import logging
 
+from .codingstatemachinedict import CodingStateMachineDict
 from .enums import MachineState
 
 
-class CodingStateMachine(object):
+class CodingStateMachine:
     """
     A state machine to verify a byte sequence for a particular encoding. For
     each byte the detector receives, it will feed that byte to every active
@@ -52,37 +53,38 @@
                  negative answer for this encoding. Detector will exclude this
                  encoding from consideration from here on.
     """
-    def __init__(self, sm):
+
+    def __init__(self, sm: CodingStateMachineDict) -> None:
         self._model = sm
         self._curr_byte_pos = 0
         self._curr_char_len = 0
-        self._curr_state = None
+        self._curr_state = MachineState.START
+        self.active = True
         self.logger = logging.getLogger(__name__)
         self.reset()
 
-    def reset(self):
+    def reset(self) -> None:
         self._curr_state = MachineState.START
 
-    def next_state(self, c):
+    def next_state(self, c: int) -> int:
         # for each byte we get its class
         # if it is first byte, we also get byte length
-        byte_class = self._model'class_table'c
+        byte_class = self._model"class_table"c
         if self._curr_state == MachineState.START:
             self._curr_byte_pos = 0
-            self._curr_char_len = self._model'char_len_table'byte_class
+            self._curr_char_len = self._model"char_len_table"byte_class
         # from byte's class and state_table, we get its next state
-        curr_state = (self._curr_state * self._model'class_factor'
-                      + byte_class)
-        self._curr_state = self._model'state_table'curr_state
+        curr_state = self._curr_state * self._model"class_factor" + byte_class
+        self._curr_state = self._model"state_table"curr_state
         self._curr_byte_pos += 1
         return self._curr_state
 
-    def get_current_charlen(self):
+    def get_current_charlen(self) -> int:
         return self._curr_char_len
 
-    def get_coding_state_machine(self):
-        return self._model'name'
+    def get_coding_state_machine(self) -> str:
+        return self._model"name"
 
     @property
-    def language(self):
-        return self._model'language'
+    def language(self) -> str:
+        return self._model"language"

_service:tar_scm:chardet-5.1.0.tar.gz/chardet/codingstatemachinedict.py Added

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/cp949prober.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/cp949prober.py Changed

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/enums.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/enums.py Changed

@@ -4,21 +4,26 @@
 :author: Dan Blanchard (dan.blanchard@gmail.com)
 """
 
+from enum import Enum, Flag
 
-class InputState(object):
+
+class InputState:
     """
     This enum represents the different states a universal detector can be in.
     """
+
     PURE_ASCII = 0
     ESC_ASCII = 1
     HIGH_BYTE = 2
 
 
-class LanguageFilter(object):
+class LanguageFilter(Flag):
     """
     This enum represents the different language filters we can apply to a
     ``UniversalDetector``.
     """
+
+    NONE = 0x00
     CHINESE_SIMPLIFIED = 0x01
     CHINESE_TRADITIONAL = 0x02
     JAPANESE = 0x04
@@ -29,46 +34,50 @@
     CJK = CHINESE | JAPANESE | KOREAN
 
 
-class ProbingState(object):
+class ProbingState(Enum):
     """
     This enum represents the different states a prober can be in.
     """
+
     DETECTING = 0
     FOUND_IT = 1
     NOT_ME = 2
 
 
-class MachineState(object):
+class MachineState:
     """
     This enum represents the different states a state machine can be in.
     """
+
     START = 0
     ERROR = 1
     ITS_ME = 2
 
 
-class SequenceLikelihood(object):
+class SequenceLikelihood:
     """
     This enum represents the likelihood of a character following the previous one.
     """
+
     NEGATIVE = 0
     UNLIKELY = 1
     LIKELY = 2
     POSITIVE = 3
 
     @classmethod
-    def get_num_categories(cls):
+    def get_num_categories(cls) -> int:
         """:returns: The number of likelihood categories in the enum."""
         return 4
 
 
-class CharacterCategory(object):
+class CharacterCategory:
     """
     This enum represents the different categories language models for
     ``SingleByteCharsetProber`` put characters into.
 
     Anything less than CONTROL is considered a letter.
     """
+
     UNDEFINED = 255
     LINE_BREAK = 254
     SYMBOL = 253

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/escprober.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/escprober.py Changed

@@ -25,11 +25,17 @@
 # 02110-1301  USA
 ######################### END LICENSE BLOCK #########################
 
+from typing import Optional, Union
+
 from .charsetprober import CharSetProber
 from .codingstatemachine import CodingStateMachine
-from .enums import LanguageFilter, ProbingState, MachineState
-from .escsm import (HZ_SM_MODEL, ISO2022CN_SM_MODEL, ISO2022JP_SM_MODEL,
-                    ISO2022KR_SM_MODEL)
+from .enums import LanguageFilter, MachineState, ProbingState
+from .escsm import (
+    HZ_SM_MODEL,
+    ISO2022CN_SM_MODEL,
+    ISO2022JP_SM_MODEL,
+    ISO2022KR_SM_MODEL,
+)
 
 
 class EscCharSetProber(CharSetProber):
@@ -39,8 +45,8 @@
     identify these encodings.
     """
 
-    def __init__(self, lang_filter=None):
-        super(EscCharSetProber, self).__init__(lang_filter=lang_filter)
+    def __init__(self, lang_filter: LanguageFilter = LanguageFilter.NONE) -> None:
+        super().__init__(lang_filter=lang_filter)
         self.coding_sm = 
         if self.lang_filter & LanguageFilter.CHINESE_SIMPLIFIED:
             self.coding_sm.append(CodingStateMachine(HZ_SM_MODEL))
@@ -49,17 +55,15 @@
             self.coding_sm.append(CodingStateMachine(ISO2022JP_SM_MODEL))
         if self.lang_filter & LanguageFilter.KOREAN:
             self.coding_sm.append(CodingStateMachine(ISO2022KR_SM_MODEL))
-        self.active_sm_count = None
-        self._detected_charset = None
-        self._detected_language = None
-        self._state = None
+        self.active_sm_count = 0
+        self._detected_charset: Optionalstr = None
+        self._detected_language: Optionalstr = None
+        self._state = ProbingState.DETECTING
         self.reset()
 
-    def reset(self):
-        super(EscCharSetProber, self).reset()
+    def reset(self) -> None:
+        super().reset()
         for coding_sm in self.coding_sm:
-            if not coding_sm:
-                continue
             coding_sm.active = True
             coding_sm.reset()
         self.active_sm_count = len(self.coding_sm)
@@ -67,23 +71,20 @@
         self._detected_language = None
 
     @property
-    def charset_name(self):
+    def charset_name(self) -> Optionalstr:
         return self._detected_charset
 
     @property
-    def language(self):
+    def language(self) -> Optionalstr:
         return self._detected_language
 
-    def get_confidence(self):
-        if self._detected_charset:
-            return 0.99
-        else:
-            return 0.00
+    def get_confidence(self) -> float:
+        return 0.99 if self._detected_charset else 0.00
 
-    def feed(self, byte_str):
+    def feed(self, byte_str: Unionbytes, bytearray) -> ProbingState:
         for c in byte_str:
             for coding_sm in self.coding_sm:
-                if not coding_sm or not coding_sm.active:
+                if not coding_sm.active:
                     continue
                 coding_state = coding_sm.next_state(c)
                 if coding_state == MachineState.ERROR:

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/escsm.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/escsm.py Changed

@@ -12,7 +12,7 @@
 # This library is free software; you can redistribute it and/or
 # modify it under the terms of the GNU Lesser General Public
 # License as published by the Free Software Foundation; either
-# version 2.1 of the License, or (at your option) any later version.
+# version 2.1 of the License,  or (at your option) any later version.
 #
 # This library is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
@@ -20,227 +20,242 @@
 # Lesser General Public License for more details.
 #
 # You should have received a copy of the GNU Lesser General Public
-# License along with this library; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
+# License along with this library; if not,  write to the Free Software
+# Foundation,  Inc.,  51 Franklin St,  Fifth Floor,  Boston,  MA
 # 02110-1301  USA
 ######################### END LICENSE BLOCK #########################
 
+from .codingstatemachinedict import CodingStateMachineDict
 from .enums import MachineState
 
+# fmt: off
 HZ_CLS = (
-1,0,0,0,0,0,0,0,  # 00 - 07
-0,0,0,0,0,0,0,0,  # 08 - 0f
-0,0,0,0,0,0,0,0,  # 10 - 17
-0,0,0,1,0,0,0,0,  # 18 - 1f
-0,0,0,0,0,0,0,0,  # 20 - 27
-0,0,0,0,0,0,0,0,  # 28 - 2f
-0,0,0,0,0,0,0,0,  # 30 - 37
-0,0,0,0,0,0,0,0,  # 38 - 3f
-0,0,0,0,0,0,0,0,  # 40 - 47
-0,0,0,0,0,0,0,0,  # 48 - 4f
-0,0,0,0,0,0,0,0,  # 50 - 57
-0,0,0,0,0,0,0,0,  # 58 - 5f
-0,0,0,0,0,0,0,0,  # 60 - 67
-0,0,0,0,0,0,0,0,  # 68 - 6f
-0,0,0,0,0,0,0,0,  # 70 - 77
-0,0,0,4,0,5,2,0,  # 78 - 7f
-1,1,1,1,1,1,1,1,  # 80 - 87
-1,1,1,1,1,1,1,1,  # 88 - 8f
-1,1,1,1,1,1,1,1,  # 90 - 97
-1,1,1,1,1,1,1,1,  # 98 - 9f
-1,1,1,1,1,1,1,1,  # a0 - a7
-1,1,1,1,1,1,1,1,  # a8 - af
-1,1,1,1,1,1,1,1,  # b0 - b7
-1,1,1,1,1,1,1,1,  # b8 - bf
-1,1,1,1,1,1,1,1,  # c0 - c7
-1,1,1,1,1,1,1,1,  # c8 - cf
-1,1,1,1,1,1,1,1,  # d0 - d7
-1,1,1,1,1,1,1,1,  # d8 - df
-1,1,1,1,1,1,1,1,  # e0 - e7
-1,1,1,1,1,1,1,1,  # e8 - ef
-1,1,1,1,1,1,1,1,  # f0 - f7
-1,1,1,1,1,1,1,1,  # f8 - ff
+    1, 0, 0, 0, 0, 0, 0, 0,  # 00 - 07
+    0, 0, 0, 0, 0, 0, 0, 0,  # 08 - 0f
+    0, 0, 0, 0, 0, 0, 0, 0,  # 10 - 17
+    0, 0, 0, 1, 0, 0, 0, 0,  # 18 - 1f
+    0, 0, 0, 0, 0, 0, 0, 0,  # 20 - 27
+    0, 0, 0, 0, 0, 0, 0, 0,  # 28 - 2f
+    0, 0, 0, 0, 0, 0, 0, 0,  # 30 - 37
+    0, 0, 0, 0, 0, 0, 0, 0,  # 38 - 3f
+    0, 0, 0, 0, 0, 0, 0, 0,  # 40 - 47
+    0, 0, 0, 0, 0, 0, 0, 0,  # 48 - 4f
+    0, 0, 0, 0, 0, 0, 0, 0,  # 50 - 57
+    0, 0, 0, 0, 0, 0, 0, 0,  # 58 - 5f
+    0, 0, 0, 0, 0, 0, 0, 0,  # 60 - 67
+    0, 0, 0, 0, 0, 0, 0, 0,  # 68 - 6f
+    0, 0, 0, 0, 0, 0, 0, 0,  # 70 - 77
+    0, 0, 0, 4, 0, 5, 2, 0,  # 78 - 7f
+    1, 1, 1, 1, 1, 1, 1, 1,  # 80 - 87
+    1, 1, 1, 1, 1, 1, 1, 1,  # 88 - 8f
+    1, 1, 1, 1, 1, 1, 1, 1,  # 90 - 97
+    1, 1, 1, 1, 1, 1, 1, 1,  # 98 - 9f
+    1, 1, 1, 1, 1, 1, 1, 1,  # a0 - a7
+    1, 1, 1, 1, 1, 1, 1, 1,  # a8 - af
+    1, 1, 1, 1, 1, 1, 1, 1,  # b0 - b7
+    1, 1, 1, 1, 1, 1, 1, 1,  # b8 - bf
+    1, 1, 1, 1, 1, 1, 1, 1,  # c0 - c7
+    1, 1, 1, 1, 1, 1, 1, 1,  # c8 - cf
+    1, 1, 1, 1, 1, 1, 1, 1,  # d0 - d7
+    1, 1, 1, 1, 1, 1, 1, 1,  # d8 - df
+    1, 1, 1, 1, 1, 1, 1, 1,  # e0 - e7
+    1, 1, 1, 1, 1, 1, 1, 1,  # e8 - ef
+    1, 1, 1, 1, 1, 1, 1, 1,  # f0 - f7
+    1, 1, 1, 1, 1, 1, 1, 1,  # f8 - ff
 )
 
 HZ_ST = (
-MachineState.START,MachineState.ERROR,     3,MachineState.START,MachineState.START,MachineState.START,MachineState.ERROR,MachineState.ERROR,# 00-07
-MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,# 08-0f
-MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ERROR,MachineState.ERROR,MachineState.START,MachineState.START,     4,MachineState.ERROR,# 10-17
-     5,MachineState.ERROR,     6,MachineState.ERROR,     5,     5,     4,MachineState.ERROR,# 18-1f
-     4,MachineState.ERROR,     4,     4,     4,MachineState.ERROR,     4,MachineState.ERROR,# 20-27
-     4,MachineState.ITS_ME,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,# 28-2f
+MachineState.START, MachineState.ERROR,      3, MachineState.START, MachineState.START, MachineState.START, MachineState.ERROR, MachineState.ERROR, # 00-07
+MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ERROR, MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ITS_ME, # 08-0f
+MachineState.ITS_ME, MachineState.ITS_ME, MachineState.ERROR, MachineState.ERROR, MachineState.START, MachineState.START,      4, MachineState.ERROR, # 10-17
+     5, MachineState.ERROR,      6, MachineState.ERROR,      5,      5,      4, MachineState.ERROR, # 18-1f
+     4, MachineState.ERROR,      4,      4,      4, MachineState.ERROR,      4, MachineState.ERROR, # 20-27
+     4, MachineState.ITS_ME, MachineState.START, MachineState.START, MachineState.START, MachineState.START, MachineState.START, MachineState.START, # 28-2f
 )
+# fmt: on
 
 HZ_CHAR_LEN_TABLE = (0, 0, 0, 0, 0, 0)
 
-HZ_SM_MODEL = {'class_table': HZ_CLS,
-               'class_factor': 6,
-               'state_table': HZ_ST,
-               'char_len_table': HZ_CHAR_LEN_TABLE,
-               'name': "HZ-GB-2312",
-               'language': 'Chinese'}
+HZ_SM_MODEL: CodingStateMachineDict = {
+    "class_table": HZ_CLS,
+    "class_factor": 6,
+    "state_table": HZ_ST,
+    "char_len_table": HZ_CHAR_LEN_TABLE,
+    "name": "HZ-GB-2312",
+    "language": "Chinese",
+}
 
+# fmt: off
 ISO2022CN_CLS = (
-2,0,0,0,0,0,0,0,  # 00 - 07
-0,0,0,0,0,0,0,0,  # 08 - 0f
-0,0,0,0,0,0,0,0,  # 10 - 17
-0,0,0,1,0,0,0,0,  # 18 - 1f
-0,0,0,0,0,0,0,0,  # 20 - 27
-0,3,0,0,0,0,0,0,  # 28 - 2f
-0,0,0,0,0,0,0,0,  # 30 - 37
-0,0,0,0,0,0,0,0,  # 38 - 3f
-0,0,0,4,0,0,0,0,  # 40 - 47
-0,0,0,0,0,0,0,0,  # 48 - 4f
-0,0,0,0,0,0,0,0,  # 50 - 57
-0,0,0,0,0,0,0,0,  # 58 - 5f
-0,0,0,0,0,0,0,0,  # 60 - 67
-0,0,0,0,0,0,0,0,  # 68 - 6f
-0,0,0,0,0,0,0,0,  # 70 - 77
-0,0,0,0,0,0,0,0,  # 78 - 7f
-2,2,2,2,2,2,2,2,  # 80 - 87
-2,2,2,2,2,2,2,2,  # 88 - 8f
-2,2,2,2,2,2,2,2,  # 90 - 97
-2,2,2,2,2,2,2,2,  # 98 - 9f
-2,2,2,2,2,2,2,2,  # a0 - a7
-2,2,2,2,2,2,2,2,  # a8 - af
-2,2,2,2,2,2,2,2,  # b0 - b7
-2,2,2,2,2,2,2,2,  # b8 - bf
-2,2,2,2,2,2,2,2,  # c0 - c7
-2,2,2,2,2,2,2,2,  # c8 - cf
-2,2,2,2,2,2,2,2,  # d0 - d7
-2,2,2,2,2,2,2,2,  # d8 - df
-2,2,2,2,2,2,2,2,  # e0 - e7
-2,2,2,2,2,2,2,2,  # e8 - ef
-2,2,2,2,2,2,2,2,  # f0 - f7
-2,2,2,2,2,2,2,2,  # f8 - ff
+    2, 0, 0, 0, 0, 0, 0, 0,  # 00 - 07
+    0, 0, 0, 0, 0, 0, 0, 0,  # 08 - 0f
+    0, 0, 0, 0, 0, 0, 0, 0,  # 10 - 17
+    0, 0, 0, 1, 0, 0, 0, 0,  # 18 - 1f
+    0, 0, 0, 0, 0, 0, 0, 0,  # 20 - 27
+    0, 3, 0, 0, 0, 0, 0, 0,  # 28 - 2f
+    0, 0, 0, 0, 0, 0, 0, 0,  # 30 - 37
+    0, 0, 0, 0, 0, 0, 0, 0,  # 38 - 3f
+    0, 0, 0, 4, 0, 0, 0, 0,  # 40 - 47
+    0, 0, 0, 0, 0, 0, 0, 0,  # 48 - 4f
+    0, 0, 0, 0, 0, 0, 0, 0,  # 50 - 57
+    0, 0, 0, 0, 0, 0, 0, 0,  # 58 - 5f
+    0, 0, 0, 0, 0, 0, 0, 0,  # 60 - 67
+    0, 0, 0, 0, 0, 0, 0, 0,  # 68 - 6f
+    0, 0, 0, 0, 0, 0, 0, 0,  # 70 - 77
+    0, 0, 0, 0, 0, 0, 0, 0,  # 78 - 7f
+    2, 2, 2, 2, 2, 2, 2, 2,  # 80 - 87
+    2, 2, 2, 2, 2, 2, 2, 2,  # 88 - 8f
+    2, 2, 2, 2, 2, 2, 2, 2,  # 90 - 97
+    2, 2, 2, 2, 2, 2, 2, 2,  # 98 - 9f
+    2, 2, 2, 2, 2, 2, 2, 2,  # a0 - a7
+    2, 2, 2, 2, 2, 2, 2, 2,  # a8 - af
+    2, 2, 2, 2, 2, 2, 2, 2,  # b0 - b7
+    2, 2, 2, 2, 2, 2, 2, 2,  # b8 - bf
+    2, 2, 2, 2, 2, 2, 2, 2,  # c0 - c7
+    2, 2, 2, 2, 2, 2, 2, 2,  # c8 - cf
+    2, 2, 2, 2, 2, 2, 2, 2,  # d0 - d7
+    2, 2, 2, 2, 2, 2, 2, 2,  # d8 - df
+    2, 2, 2, 2, 2, 2, 2, 2,  # e0 - e7
+    2, 2, 2, 2, 2, 2, 2, 2,  # e8 - ef
+    2, 2, 2, 2, 2, 2, 2, 2,  # f0 - f7
+    2, 2, 2, 2, 2, 2, 2, 2,  # f8 - ff
 )
 
 ISO2022CN_ST = (
-MachineState.START,     3,MachineState.ERROR,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,# 00-07
-MachineState.START,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,# 08-0f
-MachineState.ERROR,MachineState.ERROR,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,# 10-17
-MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,     4,MachineState.ERROR,# 18-1f
-MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ITS_ME,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,# 20-27
-     5,     6,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,# 28-2f
-MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ITS_ME,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,MachineState.ERROR,# 30-37

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/eucjpprober.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/eucjpprober.py Changed

@@ -25,68 +25,78 @@
 # 02110-1301  USA
 ######################### END LICENSE BLOCK #########################
 
-from .enums import ProbingState, MachineState
-from .mbcharsetprober import MultiByteCharSetProber
-from .codingstatemachine import CodingStateMachine
+from typing import Union
+
 from .chardistribution import EUCJPDistributionAnalysis
+from .codingstatemachine import CodingStateMachine
+from .enums import MachineState, ProbingState
 from .jpcntx import EUCJPContextAnalysis
+from .mbcharsetprober import MultiByteCharSetProber
 from .mbcssm import EUCJP_SM_MODEL
 
 
 class EUCJPProber(MultiByteCharSetProber):
-    def __init__(self):
-        super(EUCJPProber, self).__init__()
+    def __init__(self) -> None:
+        super().__init__()
         self.coding_sm = CodingStateMachine(EUCJP_SM_MODEL)
         self.distribution_analyzer = EUCJPDistributionAnalysis()
         self.context_analyzer = EUCJPContextAnalysis()
         self.reset()
 
-    def reset(self):
-        super(EUCJPProber, self).reset()
+    def reset(self) -> None:
+        super().reset()
         self.context_analyzer.reset()
 
     @property
-    def charset_name(self):
+    def charset_name(self) -> str:
         return "EUC-JP"
 
     @property
-    def language(self):
+    def language(self) -> str:
         return "Japanese"
 
-    def feed(self, byte_str):
-        for i in range(len(byte_str)):
-            # PY3K: byte_str is a byte array, so byte_stri is an int, not a byte
-            coding_state = self.coding_sm.next_state(byte_stri)
+    def feed(self, byte_str: Unionbytes, bytearray) -> ProbingState:
+        assert self.coding_sm is not None
+        assert self.distribution_analyzer is not None
+
+        for i, byte in enumerate(byte_str):
+            # PY3K: byte_str is a byte array, so byte is an int, not a byte
+            coding_state = self.coding_sm.next_state(byte)
             if coding_state == MachineState.ERROR:
-                self.logger.debug('%s %s prober hit error at byte %s',
-                                  self.charset_name, self.language, i)
+                self.logger.debug(
+                    "%s %s prober hit error at byte %s",
+                    self.charset_name,
+                    self.language,
+                    i,
+                )
                 self._state = ProbingState.NOT_ME
                 break
-            elif coding_state == MachineState.ITS_ME:
+            if coding_state == MachineState.ITS_ME:
                 self._state = ProbingState.FOUND_IT
                 break
-            elif coding_state == MachineState.START:
+            if coding_state == MachineState.START:
                 char_len = self.coding_sm.get_current_charlen()
                 if i == 0:
-                    self._last_char1 = byte_str0
+                    self._last_char1 = byte
                     self.context_analyzer.feed(self._last_char, char_len)
                     self.distribution_analyzer.feed(self._last_char, char_len)
                 else:
-                    self.context_analyzer.feed(byte_stri - 1:i + 1,
-                                                char_len)
-                    self.distribution_analyzer.feed(byte_stri - 1:i + 1,
-                                                     char_len)
+                    self.context_analyzer.feed(byte_stri - 1 : i + 1, char_len)
+                    self.distribution_analyzer.feed(byte_stri - 1 : i + 1, char_len)
 
         self._last_char0 = byte_str-1
 
         if self.state == ProbingState.DETECTING:
-            if (self.context_analyzer.got_enough_data() and
-               (self.get_confidence() > self.SHORTCUT_THRESHOLD)):
+            if self.context_analyzer.got_enough_data() and (
+                self.get_confidence() > self.SHORTCUT_THRESHOLD
+            ):
                 self._state = ProbingState.FOUND_IT
 
         return self.state
 
-    def get_confidence(self):
+    def get_confidence(self) -> float:
+        assert self.distribution_analyzer is not None
+
         context_conf = self.context_analyzer.get_confidence()
         distrib_conf = self.distribution_analyzer.get_confidence()
         return max(context_conf, distrib_conf)

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/euckrfreq.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/euckrfreq.py Changed

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/euckrprober.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/euckrprober.py Changed

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/euctwfreq.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/euctwfreq.py Changed

@@ -43,345 +43,346 @@
 
 EUCTW_TYPICAL_DISTRIBUTION_RATIO = 0.75
 
-# Char to FreqOrder table ,
+# Char to FreqOrder table
 EUCTW_TABLE_SIZE = 5376
 
+# fmt: off
 EUCTW_CHAR_TO_FREQ_ORDER = (
-   1,1800,1506, 255,1431, 198,   9,  82,   6,7310, 177, 202,3615,1256,2808, 110,  # 2742
-3735,  33,3241, 261,  76,  44,2113,  16,2931,2184,1176, 659,3868,  26,3404,2643,  # 2758
-1198,3869,3313,4060, 410,2211, 302, 590, 361,1963,   8, 204,  58,4296,7311,1931,  # 2774
-  63,7312,7313, 317,1614,  75, 222, 159,4061,2412,1480,7314,3500,3068, 224,2809,  # 2790
-3616,   3,  10,3870,1471,  29,2774,1135,2852,1939, 873, 130,3242,1123, 312,7315,  # 2806
-4297,2051, 507, 252, 682,7316, 142,1914, 124, 206,2932,  34,3501,3173,  64, 604,  # 2822
-7317,2494,1976,1977, 155,1990, 645, 641,1606,7318,3405, 337,  72, 406,7319,  80,  # 2838
- 630, 238,3174,1509, 263, 939,1092,2644, 756,1440,1094,3406, 449,  69,2969, 591,  # 2854
- 179,2095, 471, 115,2034,1843,  60,  50,2970, 134, 806,1868, 734,2035,3407, 180,  # 2870
- 995,1607, 156, 537,2893, 688,7320, 319,1305, 779,2144, 514,2374, 298,4298, 359,  # 2886
-2495,  90,2707,1338, 663,  11, 906,1099,2545,  20,2436, 182, 532,1716,7321, 732,  # 2902
-1376,4062,1311,1420,3175,  25,2312,1056, 113, 399, 382,1949, 242,3408,2467, 529,  # 2918
-3243, 475,1447,3617,7322, 117,  21, 656, 810,1297,2295,2329,3502,7323, 126,4063,  # 2934
- 706, 456, 150, 613,4299,  71,1118,2036,4064, 145,3069,  85, 835, 486,2114,1246,  # 2950
-1426, 428, 727,1285,1015, 800, 106, 623, 303,1281,7324,2127,2354, 347,3736, 221,  # 2966
-3503,3110,7325,1955,1153,4065,  83, 296,1199,3070, 192, 624,  93,7326, 822,1897,  # 2982
-2810,3111, 795,2064, 991,1554,1542,1592,  27,  43,2853, 859, 139,1456, 860,4300,  # 2998
- 437, 712,3871, 164,2392,3112, 695, 211,3017,2096, 195,3872,1608,3504,3505,3618,  # 3014
-3873, 234, 811,2971,2097,3874,2229,1441,3506,1615,2375, 668,2076,1638, 305, 228,  # 3030
-1664,4301, 467, 415,7327, 262,2098,1593, 239, 108, 300, 200,1033, 512,1247,2077,  # 3046
-7328,7329,2173,3176,3619,2673, 593, 845,1062,3244,  88,1723,2037,3875,1950, 212,  # 3062
- 266, 152, 149, 468,1898,4066,4302,  77, 187,7330,3018,  37,   5,2972,7331,3876,  # 3078
-7332,7333,  39,2517,4303,2894,3177,2078,  55, 148,  74,4304, 545, 483,1474,1029,  # 3094
-1665, 217,1869,1531,3113,1104,2645,4067,  24, 172,3507, 900,3877,3508,3509,4305,  # 3110
-  32,1408,2811,1312, 329, 487,2355,2247,2708, 784,2674,   4,3019,3314,1427,1788,  # 3126
- 188, 109, 499,7334,3620,1717,1789, 888,1217,3020,4306,7335,3510,7336,3315,1520,  # 3142
-3621,3878, 196,1034, 775,7337,7338, 929,1815, 249, 439,  38,7339,1063,7340, 794,  # 3158
-3879,1435,2296,  46, 178,3245,2065,7341,2376,7342, 214,1709,4307, 804,  35, 707,  # 3174
- 324,3622,1601,2546, 140, 459,4068,7343,7344,1365, 839, 272, 978,2257,2572,3409,  # 3190
-2128,1363,3623,1423, 697, 100,3071,  48,  70,1231, 495,3114,2193,7345,1294,7346,  # 3206
-2079, 462, 586,1042,3246, 853, 256, 988, 185,2377,3410,1698, 434,1084,7347,3411,  # 3222
- 314,2615,2775,4308,2330,2331, 569,2280, 637,1816,2518, 757,1162,1878,1616,3412,  # 3238
- 287,1577,2115, 768,4309,1671,2854,3511,2519,1321,3737, 909,2413,7348,4069, 933,  # 3254
-3738,7349,2052,2356,1222,4310, 765,2414,1322, 786,4311,7350,1919,1462,1677,2895,  # 3270
-1699,7351,4312,1424,2437,3115,3624,2590,3316,1774,1940,3413,3880,4070, 309,1369,  # 3286
-1130,2812, 364,2230,1653,1299,3881,3512,3882,3883,2646, 525,1085,3021, 902,2000,  # 3302
-1475, 964,4313, 421,1844,1415,1057,2281, 940,1364,3116, 376,4314,4315,1381,   7,  # 3318
-2520, 983,2378, 336,1710,2675,1845, 321,3414, 559,1131,3022,2742,1808,1132,1313,  # 3334
- 265,1481,1857,7352, 352,1203,2813,3247, 167,1089, 420,2814, 776, 792,1724,3513,  # 3350
-4071,2438,3248,7353,4072,7354, 446, 229, 333,2743, 901,3739,1200,1557,4316,2647,  # 3366
-1920, 395,2744,2676,3740,4073,1835, 125, 916,3178,2616,4317,7355,7356,3741,7357,  # 3382
-7358,7359,4318,3117,3625,1133,2547,1757,3415,1510,2313,1409,3514,7360,2145, 438,  # 3398
-2591,2896,2379,3317,1068, 958,3023, 461, 311,2855,2677,4074,1915,3179,4075,1978,  # 3414
- 383, 750,2745,2617,4076, 274, 539, 385,1278,1442,7361,1154,1964, 384, 561, 210,  # 3430
-  98,1295,2548,3515,7362,1711,2415,1482,3416,3884,2897,1257, 129,7363,3742, 642,  # 3446
- 523,2776,2777,2648,7364, 141,2231,1333,  68, 176, 441, 876, 907,4077, 603,2592,  # 3462
- 710, 171,3417, 404, 549,  18,3118,2393,1410,3626,1666,7365,3516,4319,2898,4320,  # 3478
-7366,2973, 368,7367, 146, 366,  99, 871,3627,1543, 748, 807,1586,1185,  22,2258,  # 3494
- 379,3743,3180,7368,3181, 505,1941,2618,1991,1382,2314,7369, 380,2357, 218, 702,  # 3510
-1817,1248,3418,3024,3517,3318,3249,7370,2974,3628, 930,3250,3744,7371,  59,7372,  # 3526
- 585, 601,4078, 497,3419,1112,1314,4321,1801,7373,1223,1472,2174,7374, 749,1836,  # 3542
- 690,1899,3745,1772,3885,1476, 429,1043,1790,2232,2116, 917,4079, 447,1086,1629,  # 3558
-7375, 556,7376,7377,2020,1654, 844,1090, 105, 550, 966,1758,2815,1008,1782, 686,  # 3574
-1095,7378,2282, 793,1602,7379,3518,2593,4322,4080,2933,2297,4323,3746, 980,2496,  # 3590
- 544, 353, 527,4324, 908,2678,2899,7380, 381,2619,1942,1348,7381,1341,1252, 560,  # 3606
-3072,7382,3420,2856,7383,2053, 973, 886,2080, 143,4325,7384,7385, 157,3886, 496,  # 3622
-4081,  57, 840, 540,2038,4326,4327,3421,2117,1445, 970,2259,1748,1965,2081,4082,  # 3638
-3119,1234,1775,3251,2816,3629, 773,1206,2129,1066,2039,1326,3887,1738,1725,4083,  # 3654
- 279,3120,  51,1544,2594, 423,1578,2130,2066, 173,4328,1879,7386,7387,1583, 264,  # 3670
- 610,3630,4329,2439, 280, 154,7388,7389,7390,1739, 338,1282,3073, 693,2857,1411,  # 3686
-1074,3747,2440,7391,4330,7392,7393,1240, 952,2394,7394,2900,1538,2679, 685,1483,  # 3702
-4084,2468,1436, 953,4085,2054,4331, 671,2395,  79,4086,2441,3252, 608, 567,2680,  # 3718
-3422,4087,4088,1691, 393,1261,1791,2396,7395,4332,7396,7397,7398,7399,1383,1672,  # 3734
-3748,3182,1464, 522,1119, 661,1150, 216, 675,4333,3888,1432,3519, 609,4334,2681,  # 3750
-2397,7400,7401,7402,4089,3025,   0,7403,2469, 315, 231,2442, 301,3319,4335,2380,  # 3766
-7404, 233,4090,3631,1818,4336,4337,7405,  96,1776,1315,2082,7406, 257,7407,1809,  # 3782
-3632,2709,1139,1819,4091,2021,1124,2163,2778,1777,2649,7408,3074, 363,1655,3183,  # 3798
-7409,2975,7410,7411,7412,3889,1567,3890, 718, 103,3184, 849,1443, 341,3320,2934,  # 3814
-1484,7413,1712, 127,  67, 339,4092,2398, 679,1412, 821,7414,7415, 834, 738, 351,  # 3830
-2976,2146, 846, 235,1497,1880, 418,1992,3749,2710, 186,1100,2147,2746,3520,1545,  # 3846
-1355,2935,2858,1377, 583,3891,4093,2573,2977,7416,1298,3633,1078,2549,3634,2358,  # 3862
-  78,3750,3751, 267,1289,2099,2001,1594,4094, 348, 369,1274,2194,2175,1837,4338,  # 3878
-1820,2817,3635,2747,2283,2002,4339,2936,2748, 144,3321, 882,4340,3892,2749,3423,  # 3894
-4341,2901,7417,4095,1726, 320,7418,3893,3026, 788,2978,7419,2818,1773,1327,2859,  # 3910
-3894,2819,7420,1306,4342,2003,1700,3752,3521,2359,2650, 787,2022, 506, 824,3636,  # 3926
- 534, 323,4343,1044,3322,2023,1900, 946,3424,7421,1778,1500,1678,7422,1881,4344,  # 3942
- 165, 243,4345,3637,2521, 123, 683,4096, 764,4346,  36,3895,1792, 589,2902, 816,  # 3958
- 626,1667,3027,2233,1639,1555,1622,3753,3896,7423,3897,2860,1370,1228,1932, 891,  # 3974
-2083,2903, 304,4097,7424, 292,2979,2711,3522, 691,2100,4098,1115,4347, 118, 662,  # 3990
-7425, 611,1156, 854,2381,1316,2861,   2, 386, 515,2904,7426,7427,3253, 868,2234,  # 4006
-1486, 855,2651, 785,2212,3028,7428,1040,3185,3523,7429,3121, 448,7430,1525,7431,  # 4022
-2164,4348,7432,3754,7433,4099,2820,3524,3122, 503, 818,3898,3123,1568, 814, 676,  # 4038
-1444, 306,1749,7434,3755,1416,1030, 197,1428, 805,2821,1501,4349,7435,7436,7437,  # 4054
-1993,7438,4350,7439,7440,2195,  13,2779,3638,2980,3124,1229,1916,7441,3756,2131,  # 4070
-7442,4100,4351,2399,3525,7443,2213,1511,1727,1120,7444,7445, 646,3757,2443, 307,  # 4086
-7446,7447,1595,3186,7448,7449,7450,3639,1113,1356,3899,1465,2522,2523,7451, 519,  # 4102
-7452, 128,2132,  92,2284,1979,7453,3900,1512, 342,3125,2196,7454,2780,2214,1980,  # 4118
-3323,7455, 290,1656,1317, 789, 827,2360,7456,3758,4352, 562, 581,3901,7457, 401,  # 4134
-4353,2248,  94,4354,1399,2781,7458,1463,2024,4355,3187,1943,7459, 828,1105,4101,  # 4150
-1262,1394,7460,4102, 605,4356,7461,1783,2862,7462,2822, 819,2101, 578,2197,2937,  # 4166
-7463,1502, 436,3254,4103,3255,2823,3902,2905,3425,3426,7464,2712,2315,7465,7466,  # 4182
-2332,2067,  23,4357, 193, 826,3759,2102, 699,1630,4104,3075, 390,1793,1064,3526,  # 4198
-7467,1579,3076,3077,1400,7468,4105,1838,1640,2863,7469,4358,4359, 137,4106, 598,  # 4214
-3078,1966, 780, 104, 974,2938,7470, 278, 899, 253, 402, 572, 504, 493,1339,7471,  # 4230
-3903,1275,4360,2574,2550,7472,3640,3029,3079,2249, 565,1334,2713, 863,  41,7473,  # 4246
-7474,4361,7475,1657,2333,  19, 463,2750,4107, 606,7476,2981,3256,1087,2084,1323,  # 4262
-2652,2982,7477,1631,1623,1750,4108,2682,7478,2864, 791,2714,2653,2334, 232,2416,  # 4278
-7479,2983,1498,7480,2654,2620, 755,1366,3641,3257,3126,2025,1609, 119,1917,3427,  # 4294
- 862,1026,4109,7481,3904,3760,4362,3905,4363,2260,1951,2470,7482,1125, 817,4110,  # 4310
-4111,3906,1513,1766,2040,1487,4112,3030,3258,2824,3761,3127,7483,7484,1507,7485,  # 4326
-2683, 733,  40,1632,1106,2865, 345,4113, 841,2524, 230,4364,2984,1846,3259,3428,  # 4342
-7486,1263, 986,3429,7487, 735, 879, 254,1137, 857, 622,1300,1180,1388,1562,3907,  # 4358
-3908,2939, 967,2751,2655,1349, 592,2133,1692,3324,2985,1994,4114,1679,3909,1901,  # 4374
-2185,7488, 739,3642,2715,1296,1290,7489,4115,2198,2199,1921,1563,2595,2551,1870,  # 4390
-2752,2986,7490, 435,7491, 343,1108, 596,  17,1751,4365,2235,3430,3643,7492,4366,  # 4406
- 294,3527,2940,1693, 477, 979, 281,2041,3528, 643,2042,3644,2621,2782,2261,1031,  # 4422
-2335,2134,2298,3529,4367, 367,1249,2552,7493,3530,7494,4368,1283,3325,2004, 240,  # 4438
-1762,3326,4369,4370, 836,1069,3128, 474,7495,2148,2525, 268,3531,7496,3188,1521,  # 4454
-1284,7497,1658,1546,4116,7498,3532,3533,7499,4117,3327,2684,1685,4118, 961,1673,  # 4470
-2622, 190,2005,2200,3762,4371,4372,7500, 570,2497,3645,1490,7501,4373,2623,3260,  # 4486
-1956,4374, 584,1514, 396,1045,1944,7502,4375,1967,2444,7503,7504,4376,3910, 619,  # 4502
-7505,3129,3261, 215,2006,2783,2553,3189,4377,3190,4378, 763,4119,3763,4379,7506,  # 4518
-7507,1957,1767,2941,3328,3646,1174, 452,1477,4380,3329,3130,7508,2825,1253,2382,  # 4534
-2186,1091,2285,4120, 492,7509, 638,1169,1824,2135,1752,3911, 648, 926,1021,1324,  # 4550
-4381, 520,4382, 997, 847,1007, 892,4383,3764,2262,1871,3647,7510,2400,1784,4384,  # 4566
-1952,2942,3080,3191,1728,4121,2043,3648,4385,2007,1701,3131,1551,  30,2263,4122,  # 4582
-7511,2026,4386,3534,7512, 501,7513,4123, 594,3431,2165,1821,3535,3432,3536,3192,  # 4598
- 829,2826,4124,7514,1680,3132,1225,4125,7515,3262,4387,4126,3133,2336,7516,4388,  # 4614
-4127,7517,3912,3913,7518,1847,2383,2596,3330,7519,4389, 374,3914, 652,4128,4129,  # 4630
- 375,1140, 798,7520,7521,7522,2361,4390,2264, 546,1659, 138,3031,2445,4391,7523,  # 4646
-2250, 612,1848, 910, 796,3765,1740,1371, 825,3766,3767,7524,2906,2554,7525, 692,  # 4662
- 444,3032,2624, 801,4392,4130,7526,1491, 244,1053,3033,4131,4132, 340,7527,3915,  # 4678
-1041,2987, 293,1168,  87,1357,7528,1539, 959,7529,2236, 721, 694,4133,3768, 219,  # 4694
-1478, 644,1417,3331,2656,1413,1401,1335,1389,3916,7530,7531,2988,2362,3134,1825,  # 4710
- 730,1515, 184,2827,  66,4393,7532,1660,2943, 246,3332, 378,1457, 226,3433, 975,  # 4726
-3917,2944,1264,3537, 674, 696,7533, 163,7534,1141,2417,2166, 713,3538,3333,4394,  # 4742
-3918,7535,7536,1186,  15,7537,1079,1070,7538,1522,3193,3539, 276,1050,2716, 758,  # 4758
-1126, 653,2945,3263,7539,2337, 889,3540,3919,3081,2989, 903,1250,4395,3920,3434,  # 4774
-3541,1342,1681,1718, 766,3264, 286,  89,2946,3649,7540,1713,7541,2597,3334,2990,  # 4790
-7542,2947,2215,3194,2866,7543,4396,2498,2526, 181, 387,1075,3921, 731,2187,3335,  # 4806
-7544,3265, 310, 313,3435,2299, 770,4134,  54,3034, 189,4397,3082,3769,3922,7545,  # 4822
-1230,1617,1849, 355,3542,4135,4398,3336, 111,4136,3650,1350,3135,3436,3035,4137,  # 4838
-2149,3266,3543,7546,2784,3923,3924,2991, 722,2008,7547,1071, 247,1207,2338,2471,  # 4854
-1378,4399,2009, 864,1437,1214,4400, 373,3770,1142,2216, 667,4401, 442,2753,2555,  # 4870
-3771,3925,1968,4138,3267,1839, 837, 170,1107, 934,1336,1882,7548,7549,2118,4139,  # 4886
-2828, 743,1569,7550,4402,4140, 582,2384,1418,3437,7551,1802,7552, 357,1395,1729,  # 4902
-3651,3268,2418,1564,2237,7553,3083,3772,1633,4403,1114,2085,4141,1532,7554, 482,  # 4918
-2446,4404,7555,7556,1492, 833,1466,7557,2717,3544,1641,2829,7558,1526,1272,3652,  # 4934
-4142,1686,1794, 416,2556,1902,1953,1803,7559,3773,2785,3774,1159,2316,7560,2867,  # 4950
-4405,1610,1584,3036,2419,2754, 443,3269,1163,3136,7561,7562,3926,7563,4143,2499,  # 4966
-3037,4406,3927,3137,2103,1647,3545,2010,1872,4144,7564,4145, 431,3438,7565, 250,  # 4982
-  97,  81,4146,7566,1648,1850,1558, 160, 848,7567, 866, 740,1694,7568,2201,2830,  # 4998
-3195,4147,4407,3653,1687, 950,2472, 426, 469,3196,3654,3655,3928,7569,7570,1188,  # 5014
- 424,1995, 861,3546,4148,3775,2202,2685, 168,1235,3547,4149,7571,2086,1674,4408,  # 5030
-3337,3270, 220,2557,1009,7572,3776, 670,2992, 332,1208, 717,7573,7574,3548,2447,  # 5046
-3929,3338,7575, 513,7576,1209,2868,3339,3138,4409,1080,7577,7578,7579,7580,2527,  # 5062
-3656,3549, 815,1587,3930,3931,7581,3550,3439,3777,1254,4410,1328,3038,1390,3932,  # 5078
-1741,3933,3778,3934,7582, 236,3779,2448,3271,7583,7584,3657,3780,1273,3781,4411,  # 5094
-7585, 308,7586,4412, 245,4413,1851,2473,1307,2575, 430, 715,2136,2449,7587, 270,  # 5110
- 199,2869,3935,7588,3551,2718,1753, 761,1754, 725,1661,1840,4414,3440,3658,7589,  # 5126
-7590, 587,  14,3272, 227,2598, 326, 480,2265, 943,2755,3552, 291, 650,1883,7591,  # 5142
-1702,1226, 102,1547,  62,3441, 904,4415,3442,1164,4150,7592,7593,1224,1548,2756,  # 5158
- 391, 498,1493,7594,1386,1419,7595,2055,1177,4416, 813, 880,1081,2363, 566,1145,  # 5174
-4417,2286,1001,1035,2558,2599,2238, 394,1286,7596,7597,2068,7598,  86,1494,1730,  # 5190
-3936, 491,1588, 745, 897,2948, 843,3340,3937,2757,2870,3273,1768, 998,2217,2069,  # 5206
- 397,1826,1195,1969,3659,2993,3341, 284,7599,3782,2500,2137,2119,1903,7600,3938,  # 5222
-2150,3939,4151,1036,3443,1904, 114,2559,4152, 209,1527,7601,7602,2949,2831,2625,  # 5238
-2385,2719,3139, 812,2560,7603,3274,7604,1559, 737,1884,3660,1210, 885,  28,2686,  # 5254
-3553,3783,7605,4153,1004,1779,4418,7606, 346,1981,2218,2687,4419,3784,1742, 797,  # 5270
-1642,3940,1933,1072,1384,2151, 896,3941,3275,3661,3197,2871,3554,7607,2561,1958,  # 5286
-4420,2450,1785,7608,7609,7610,3942,4154,1005,1308,3662,4155,2720,4421,4422,1528,  # 5302
-2600, 161,1178,4156,1982, 987,4423,1101,4157, 631,3943,1157,3198,2420,1343,1241,  # 5318
-1016,2239,2562, 372, 877,2339,2501,1160, 555,1934, 911,3944,7611, 466,1170, 169,  # 5334
-1051,2907,2688,3663,2474,2994,1182,2011,2563,1251,2626,7612, 992,2340,3444,1540,  # 5350
-2721,1201,2070,2401,1996,2475,7613,4424, 528,1922,2188,1503,1873,1570,2364,3342,  # 5366
-3276,7614, 557,1073,7615,1827,3445,2087,2266,3140,3039,3084, 767,3085,2786,4425,  # 5382
-1006,4158,4426,2341,1267,2176,3664,3199, 778,3945,3200,2722,1597,2657,7616,4427,  # 5398
-7617,3446,7618,7619,7620,3277,2689,1433,3278, 131,  95,1504,3946, 723,4159,3141,  # 5414
-1841,3555,2758,2189,3947,2027,2104,3665,7621,2995,3948,1218,7622,3343,3201,3949,  # 5430
-4160,2576, 248,1634,3785, 912,7623,2832,3666,3040,3786, 654,  53,7624,2996,7625,  # 5446
-1688,4428, 777,3447,1032,3950,1425,7626, 191, 820,2120,2833, 971,4429, 931,3202,  # 5462
- 135, 664, 783,3787,1997, 772,2908,1935,3951,3788,4430,2909,3203, 282,2723, 640,  # 5478
-1372,3448,1127, 922, 325,3344,7627,7628, 711,2044,7629,7630,3952,2219,2787,1936,  # 5494
-3953,3345,2220,2251,3789,2300,7631,4431,3790,1258,3279,3954,3204,2138,2950,3955,  # 5510
-3956,7632,2221, 258,3205,4432, 101,1227,7633,3280,1755,7634,1391,3281,7635,2910,  # 5526
-2056, 893,7636,7637,7638,1402,4161,2342,7639,7640,3206,3556,7641,7642, 878,1325,  # 5542
-1780,2788,4433, 259,1385,2577, 744,1183,2267,4434,7643,3957,2502,7644, 684,1024,  # 5558
-4162,7645, 472,3557,3449,1165,3282,3958,3959, 322,2152, 881, 455,1695,1152,1340,  # 5574
- 660, 554,2153,4435,1058,4436,4163, 830,1065,3346,3960,4437,1923,7646,1703,1918,  # 5590
-7647, 932,2268, 122,7648,4438, 947, 677,7649,3791,2627, 297,1905,1924,2269,4439,  # 5606
-2317,3283,7650,7651,4164,7652,4165,  84,4166, 112, 989,7653, 547,1059,3961, 701,  # 5622
-3558,1019,7654,4167,7655,3450, 942, 639, 457,2301,2451, 993,2951, 407, 851, 494,  # 5638
-4440,3347, 927,7656,1237,7657,2421,3348, 573,4168, 680, 921,2911,1279,1874, 285,  # 5654
- 790,1448,1983, 719,2167,7658,7659,4441,3962,3963,1649,7660,1541, 563,7661,1077,  # 5670
-7662,3349,3041,3451, 511,2997,3964,3965,3667,3966,1268,2564,3350,3207,4442,4443,  # 5686
-7663, 535,1048,1276,1189,2912,2028,3142,1438,1373,2834,2952,1134,2012,7664,4169,  # 5702
-1238,2578,3086,1259,7665, 700,7666,2953,3143,3668,4170,7667,4171,1146,1875,1906,  # 5718
-4444,2601,3967, 781,2422, 132,1589, 203, 147, 273,2789,2402, 898,1786,2154,3968,  # 5734
-3969,7668,3792,2790,7669,7670,4445,4446,7671,3208,7672,1635,3793, 965,7673,1804,  # 5750
-2690,1516,3559,1121,1082,1329,3284,3970,1449,3794,  65,1128,2835,2913,2759,1590,  # 5766

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/euctwprober.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/euctwprober.py Changed

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/gb2312freq.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/gb2312freq.py Changed

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/gb2312prober.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/gb2312prober.py Changed

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/hebrewprober.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/hebrewprober.py Changed

@@ -25,8 +25,11 @@
 # 02110-1301  USA
 ######################### END LICENSE BLOCK #########################
 
+from typing import Optional, Union
+
 from .charsetprober import CharSetProber
 from .enums import ProbingState
+from .sbcharsetprober import SingleByteCharSetProber
 
 # This prober doesn't actually recognize a language or a charset.
 # It is a helper prober for the use of the Hebrew model probers
@@ -125,18 +128,20 @@
 # model probers scores. The answer is returned in the form of the name of the
 # charset identified, either "windows-1255" or "ISO-8859-8".
 
+
 class HebrewProber(CharSetProber):
+    SPACE = 0x20
     # windows-1255 / ISO-8859-8 code points of interest
-    FINAL_KAF = 0xea
-    NORMAL_KAF = 0xeb
-    FINAL_MEM = 0xed
-    NORMAL_MEM = 0xee
-    FINAL_NUN = 0xef
-    NORMAL_NUN = 0xf0
-    FINAL_PE = 0xf3
-    NORMAL_PE = 0xf4
-    FINAL_TSADI = 0xf5
-    NORMAL_TSADI = 0xf6
+    FINAL_KAF = 0xEA
+    NORMAL_KAF = 0xEB
+    FINAL_MEM = 0xED
+    NORMAL_MEM = 0xEE
+    FINAL_NUN = 0xEF
+    NORMAL_NUN = 0xF0
+    FINAL_PE = 0xF3
+    NORMAL_PE = 0xF4
+    FINAL_TSADI = 0xF5
+    NORMAL_TSADI = 0xF6
 
     # Minimum Visual vs Logical final letter score difference.
     # If the difference is below this, don't rely solely on the final letter score
@@ -151,35 +156,44 @@
     VISUAL_HEBREW_NAME = "ISO-8859-8"
     LOGICAL_HEBREW_NAME = "windows-1255"
 
-    def __init__(self):
-        super(HebrewProber, self).__init__()
-        self._final_char_logical_score = None
-        self._final_char_visual_score = None
-        self._prev = None
-        self._before_prev = None
-        self._logical_prober = None
-        self._visual_prober = None
+    def __init__(self) -> None:
+        super().__init__()
+        self._final_char_logical_score = 0
+        self._final_char_visual_score = 0
+        self._prev = self.SPACE
+        self._before_prev = self.SPACE
+        self._logical_prober: OptionalSingleByteCharSetProber = None
+        self._visual_prober: OptionalSingleByteCharSetProber = None
         self.reset()
 
-    def reset(self):
+    def reset(self) -> None:
         self._final_char_logical_score = 0
         self._final_char_visual_score = 0
         # The two last characters seen in the previous buffer,
         # mPrev and mBeforePrev are initialized to space in order to simulate
         # a word delimiter at the beginning of the data
-        self._prev = ' '
-        self._before_prev = ' '
+        self._prev = self.SPACE
+        self._before_prev = self.SPACE
         # These probers are owned by the group prober.
 
-    def set_model_probers(self, logicalProber, visualProber):
-        self._logical_prober = logicalProber
-        self._visual_prober = visualProber
+    def set_model_probers(
+        self,
+        logical_prober: SingleByteCharSetProber,
+        visual_prober: SingleByteCharSetProber,
+    ) -> None:
+        self._logical_prober = logical_prober
+        self._visual_prober = visual_prober
 
-    def is_final(self, c):
-        return c in self.FINAL_KAF, self.FINAL_MEM, self.FINAL_NUN,
-                     self.FINAL_PE, self.FINAL_TSADI
+    def is_final(self, c: int) -> bool:
+        return c in 
+            self.FINAL_KAF,
+            self.FINAL_MEM,
+            self.FINAL_NUN,
+            self.FINAL_PE,
+            self.FINAL_TSADI,
+        
 
-    def is_non_final(self, c):
+    def is_non_final(self, c: int) -> bool:
         # The normal Tsadi is not a good Non-Final letter due to words like
         # 'lechotet' (to chat) containing an apostrophe after the tsadi. This
         # apostrophe is converted to a space in FilterWithoutEnglishLetters
@@ -190,10 +204,9 @@
         # for example legally end with a Non-Final Pe or Kaf. However, the
         # benefit of these letters as Non-Final letters outweighs the damage
         # since these words are quite rare.
-        return c in self.NORMAL_KAF, self.NORMAL_MEM,
-                     self.NORMAL_NUN, self.NORMAL_PE
+        return c in self.NORMAL_KAF, self.NORMAL_MEM, self.NORMAL_NUN, self.NORMAL_PE
 
-    def feed(self, byte_str):
+    def feed(self, byte_str: Unionbytes, bytearray) -> ProbingState:
         # Final letter analysis for logical-visual decision.
         # Look for evidence that the received buffer is either logical Hebrew
         # or visual Hebrew.
@@ -227,9 +240,9 @@
         byte_str = self.filter_high_byte_only(byte_str)
 
         for cur in byte_str:
-            if cur == ' ':
+            if cur == self.SPACE:
                 # We stand on a space - a word just ended
-                if self._before_prev != ' ':
+                if self._before_prev != self.SPACE:
                     # next-to-last char was not a space so self._prev is not a
                     # 1 letter word
                     if self.is_final(self._prev):
@@ -241,8 +254,11 @@
                         self._final_char_visual_score += 1
             else:
                 # Not standing on a space
-                if ((self._before_prev == ' ') and
-                        (self.is_final(self._prev)) and (cur != ' ')):
+                if (
+                    (self._before_prev == self.SPACE)
+                    and (self.is_final(self._prev))
+                    and (cur != self.SPACE)
+                ):
                     # case (3) -2:space-1:final lettercur:not space
                     self._final_char_visual_score += 1
             self._before_prev = self._prev
@@ -253,7 +269,10 @@
         return ProbingState.DETECTING
 
     @property
-    def charset_name(self):
+    def charset_name(self) -> str:
+        assert self._logical_prober is not None
+        assert self._visual_prober is not None
+
         # Make the decision: is it Logical or Visual?
         # If the final letter score distance is dominant enough, rely on it.
         finalsub = self._final_char_logical_score - self._final_char_visual_score
@@ -263,8 +282,9 @@
             return self.VISUAL_HEBREW_NAME
 
         # It's not dominant enough, try to rely on the model scores instead.
-        modelsub = (self._logical_prober.get_confidence()
-                    - self._visual_prober.get_confidence())
+        modelsub = (
+            self._logical_prober.get_confidence() - self._visual_prober.get_confidence()
+        )
         if modelsub > self.MIN_MODEL_DISTANCE:
             return self.LOGICAL_HEBREW_NAME
         if modelsub < -self.MIN_MODEL_DISTANCE:
@@ -280,13 +300,17 @@
         return self.LOGICAL_HEBREW_NAME
 
     @property
-    def language(self):
-        return 'Hebrew'
+    def language(self) -> str:
+        return "Hebrew"
 
     @property
-    def state(self):
+    def state(self) -> ProbingState:
+        assert self._logical_prober is not None
+        assert self._visual_prober is not None
+
         # Remain active as long as any of the model probers are active.
-        if (self._logical_prober.state == ProbingState.NOT_ME) and \
-           (self._visual_prober.state == ProbingState.NOT_ME):
+        if (self._logical_prober.state == ProbingState.NOT_ME) and (
+            self._visual_prober.state == ProbingState.NOT_ME
+        ):
             return ProbingState.NOT_ME
         return ProbingState.DETECTING

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/jisfreq.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/jisfreq.py Changed

_service:tar_scm:chardet-5.1.0.tar.gz/chardet/johabfreq.py Added

@@ -0,0 +1,2382 @@
+######################## BEGIN LICENSE BLOCK ########################
+# The Original Code is Mozilla Communicator client code.
+#
+# The Initial Developer of the Original Code is
+# Netscape Communications Corporation.
+# Portions created by the Initial Developer are Copyright (C) 1998
+# the Initial Developer. All Rights Reserved.
+#
+# Contributor(s):
+#   Mark Pilgrim - port to Python
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
+# 02110-1301  USA
+######################### END LICENSE BLOCK #########################
+
+# The frequency data itself is the same as euc-kr.
+# This is just a mapping table to euc-kr.
+
+JOHAB_TO_EUCKR_ORDER_TABLE = {
+    0x8861: 0,
+    0x8862: 1,
+    0x8865: 2,
+    0x8868: 3,
+    0x8869: 4,
+    0x886A: 5,
+    0x886B: 6,
+    0x8871: 7,
+    0x8873: 8,
+    0x8874: 9,
+    0x8875: 10,
+    0x8876: 11,
+    0x8877: 12,
+    0x8878: 13,
+    0x8879: 14,
+    0x887B: 15,
+    0x887C: 16,
+    0x887D: 17,
+    0x8881: 18,
+    0x8882: 19,
+    0x8885: 20,
+    0x8889: 21,
+    0x8891: 22,
+    0x8893: 23,
+    0x8895: 24,
+    0x8896: 25,
+    0x8897: 26,
+    0x88A1: 27,
+    0x88A2: 28,
+    0x88A5: 29,
+    0x88A9: 30,
+    0x88B5: 31,
+    0x88B7: 32,
+    0x88C1: 33,
+    0x88C5: 34,
+    0x88C9: 35,
+    0x88E1: 36,
+    0x88E2: 37,
+    0x88E5: 38,
+    0x88E8: 39,
+    0x88E9: 40,
+    0x88EB: 41,
+    0x88F1: 42,
+    0x88F3: 43,
+    0x88F5: 44,
+    0x88F6: 45,
+    0x88F7: 46,
+    0x88F8: 47,
+    0x88FB: 48,
+    0x88FC: 49,
+    0x88FD: 50,
+    0x8941: 51,
+    0x8945: 52,
+    0x8949: 53,
+    0x8951: 54,
+    0x8953: 55,
+    0x8955: 56,
+    0x8956: 57,
+    0x8957: 58,
+    0x8961: 59,
+    0x8962: 60,
+    0x8963: 61,
+    0x8965: 62,
+    0x8968: 63,
+    0x8969: 64,
+    0x8971: 65,
+    0x8973: 66,
+    0x8975: 67,
+    0x8976: 68,
+    0x8977: 69,
+    0x897B: 70,
+    0x8981: 71,
+    0x8985: 72,
+    0x8989: 73,
+    0x8993: 74,
+    0x8995: 75,
+    0x89A1: 76,
+    0x89A2: 77,
+    0x89A5: 78,
+    0x89A8: 79,
+    0x89A9: 80,
+    0x89AB: 81,
+    0x89AD: 82,
+    0x89B0: 83,
+    0x89B1: 84,
+    0x89B3: 85,
+    0x89B5: 86,
+    0x89B7: 87,
+    0x89B8: 88,
+    0x89C1: 89,
+    0x89C2: 90,
+    0x89C5: 91,
+    0x89C9: 92,
+    0x89CB: 93,
+    0x89D1: 94,
+    0x89D3: 95,
+    0x89D5: 96,
+    0x89D7: 97,
+    0x89E1: 98,
+    0x89E5: 99,
+    0x89E9: 100,
+    0x89F3: 101,
+    0x89F6: 102,
+    0x89F7: 103,
+    0x8A41: 104,
+    0x8A42: 105,
+    0x8A45: 106,
+    0x8A49: 107,
+    0x8A51: 108,
+    0x8A53: 109,
+    0x8A55: 110,
+    0x8A57: 111,
+    0x8A61: 112,
+    0x8A65: 113,
+    0x8A69: 114,
+    0x8A73: 115,
+    0x8A75: 116,
+    0x8A81: 117,
+    0x8A82: 118,
+    0x8A85: 119,
+    0x8A88: 120,
+    0x8A89: 121,
+    0x8A8A: 122,
+    0x8A8B: 123,
+    0x8A90: 124,
+    0x8A91: 125,
+    0x8A93: 126,
+    0x8A95: 127,
+    0x8A97: 128,
+    0x8A98: 129,
+    0x8AA1: 130,
+    0x8AA2: 131,
+    0x8AA5: 132,
+    0x8AA9: 133,
+    0x8AB6: 134,
+    0x8AB7: 135,
+    0x8AC1: 136,
+    0x8AD5: 137,
+    0x8AE1: 138,
+    0x8AE2: 139,
+    0x8AE5: 140,
+    0x8AE9: 141,
+    0x8AF1: 142,
+    0x8AF3: 143,
+    0x8AF5: 144,
+    0x8B41: 145,
+    0x8B45: 146,
+    0x8B49: 147,
+    0x8B61: 148,
+    0x8B62: 149,
+    0x8B65: 150,
+    0x8B68: 151,
+    0x8B69: 152,
+    0x8B6A: 153,
+    0x8B71: 154,
+    0x8B73: 155,
+    0x8B75: 156,
+    0x8B77: 157,
+    0x8B81: 158,
+    0x8BA1: 159,
+    0x8BA2: 160,
+    0x8BA5: 161,
+    0x8BA8: 162,
+    0x8BA9: 163,
+    0x8BAB: 164,
+    0x8BB1: 165,
+    0x8BB3: 166,
+    0x8BB5: 167,

_service:tar_scm:chardet-5.1.0.tar.gz/chardet/johabprober.py Added

@@ -0,0 +1,47 @@
+######################## BEGIN LICENSE BLOCK ########################
+# The Original Code is mozilla.org code.
+#
+# The Initial Developer of the Original Code is
+# Netscape Communications Corporation.
+# Portions created by the Initial Developer are Copyright (C) 1998
+# the Initial Developer. All Rights Reserved.
+#
+# Contributor(s):
+#   Mark Pilgrim - port to Python
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
+# 02110-1301  USA
+######################### END LICENSE BLOCK #########################
+
+from .chardistribution import JOHABDistributionAnalysis
+from .codingstatemachine import CodingStateMachine
+from .mbcharsetprober import MultiByteCharSetProber
+from .mbcssm import JOHAB_SM_MODEL
+
+
+class JOHABProber(MultiByteCharSetProber):
+    def __init__(self) -> None:
+        super().__init__()
+        self.coding_sm = CodingStateMachine(JOHAB_SM_MODEL)
+        self.distribution_analyzer = JOHABDistributionAnalysis()
+        self.reset()
+
+    @property
+    def charset_name(self) -> str:
+        return "Johab"
+
+    @property
+    def language(self) -> str:
+        return "Korean"

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/jpcntx.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/jpcntx.py Changed

@@ -25,110 +25,114 @@
 # 02110-1301  USA
 ######################### END LICENSE BLOCK #########################
 
+from typing import List, Tuple, Union
 
 # This is hiragana 2-char sequence table, the number in each cell represents its frequency category
-jp2CharContext = (
-(0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,1,0,0,0,0,0,0,0,0,0,0,1),
-(2,4,0,4,0,3,0,4,0,3,4,4,4,2,4,3,3,4,3,2,3,3,4,2,3,3,3,2,4,1,4,3,3,1,5,4,3,4,3,4,3,5,3,0,3,5,4,2,0,3,1,0,3,3,0,3,3,0,1,1,0,4,3,0,3,3,0,4,0,2,0,3,5,5,5,5,4,0,4,1,0,3,4),
-(0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2),
-(0,4,0,5,0,5,0,4,0,4,5,4,4,3,5,3,5,1,5,3,4,3,4,4,3,4,3,3,4,3,5,4,4,3,5,5,3,5,5,5,3,5,5,3,4,5,5,3,1,3,2,0,3,4,0,4,2,0,4,2,1,5,3,2,3,5,0,4,0,2,0,5,4,4,5,4,5,0,4,0,0,4,4),
-(0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0),
-(0,3,0,4,0,3,0,3,0,4,5,4,3,3,3,3,4,3,5,4,4,3,5,4,4,3,4,3,4,4,4,4,5,3,4,4,3,4,5,5,4,5,5,1,4,5,4,3,0,3,3,1,3,3,0,4,4,0,3,3,1,5,3,3,3,5,0,4,0,3,0,4,4,3,4,3,3,0,4,1,1,3,4),
-(0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0),
-(0,4,0,3,0,3,0,4,0,3,4,4,3,2,2,1,2,1,3,1,3,3,3,3,3,4,3,1,3,3,5,3,3,0,4,3,0,5,4,3,3,5,4,4,3,4,4,5,0,1,2,0,1,2,0,2,2,0,1,0,0,5,2,2,1,4,0,3,0,1,0,4,4,3,5,4,3,0,2,1,0,4,3),
-(0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0),
-(0,3,0,5,0,4,0,2,1,4,4,2,4,1,4,2,4,2,4,3,3,3,4,3,3,3,3,1,4,2,3,3,3,1,4,4,1,1,1,4,3,3,2,0,2,4,3,2,0,3,3,0,3,1,1,0,0,0,3,3,0,4,2,2,3,4,0,4,0,3,0,4,4,5,3,4,4,0,3,0,0,1,4),
-(1,4,0,4,0,4,0,4,0,3,5,4,4,3,4,3,5,4,3,3,4,3,5,4,4,4,4,3,4,2,4,3,3,1,5,4,3,2,4,5,4,5,5,4,4,5,4,4,0,3,2,2,3,3,0,4,3,1,3,2,1,4,3,3,4,5,0,3,0,2,0,4,5,5,4,5,4,0,4,0,0,5,4),
-(0,5,0,5,0,4,0,3,0,4,4,3,4,3,3,3,4,0,4,4,4,3,4,3,4,3,3,1,4,2,4,3,4,0,5,4,1,4,5,4,4,5,3,2,4,3,4,3,2,4,1,3,3,3,2,3,2,0,4,3,3,4,3,3,3,4,0,4,0,3,0,4,5,4,4,4,3,0,4,1,0,1,3),
-(0,3,1,4,0,3,0,2,0,3,4,4,3,1,4,2,3,3,4,3,4,3,4,3,4,4,3,2,3,1,5,4,4,1,4,4,3,5,4,4,3,5,5,4,3,4,4,3,1,2,3,1,2,2,0,3,2,0,3,1,0,5,3,3,3,4,3,3,3,3,4,4,4,4,5,4,2,0,3,3,2,4,3),
-(0,2,0,3,0,1,0,1,0,0,3,2,0,0,2,0,1,0,2,1,3,3,3,1,2,3,1,0,1,0,4,2,1,1,3,3,0,4,3,3,1,4,3,3,0,3,3,2,0,0,0,0,1,0,0,2,0,0,0,0,0,4,1,0,2,3,2,2,2,1,3,3,3,4,4,3,2,0,3,1,0,3,3),
-(0,4,0,4,0,3,0,3,0,4,4,4,3,3,3,3,3,3,4,3,4,2,4,3,4,3,3,2,4,3,4,5,4,1,4,5,3,5,4,5,3,5,4,0,3,5,5,3,1,3,3,2,2,3,0,3,4,1,3,3,2,4,3,3,3,4,0,4,0,3,0,4,5,4,4,5,3,0,4,1,0,3,4),
-(0,2,0,3,0,3,0,0,0,2,2,2,1,0,1,0,0,0,3,0,3,0,3,0,1,3,1,0,3,1,3,3,3,1,3,3,3,0,1,3,1,3,4,0,0,3,1,1,0,3,2,0,0,0,0,1,3,0,1,0,0,3,3,2,0,3,0,0,0,0,0,3,4,3,4,3,3,0,3,0,0,2,3),
-(2,3,0,3,0,2,0,1,0,3,3,4,3,1,3,1,1,1,3,1,4,3,4,3,3,3,0,0,3,1,5,4,3,1,4,3,2,5,5,4,4,4,4,3,3,4,4,4,0,2,1,1,3,2,0,1,2,0,0,1,0,4,1,3,3,3,0,3,0,1,0,4,4,4,5,5,3,0,2,0,0,4,4),
-(0,2,0,1,0,3,1,3,0,2,3,3,3,0,3,1,0,0,3,0,3,2,3,1,3,2,1,1,0,0,4,2,1,0,2,3,1,4,3,2,0,4,4,3,1,3,1,3,0,1,0,0,1,0,0,0,1,0,0,0,0,4,1,1,1,2,0,3,0,0,0,3,4,2,4,3,2,0,1,0,0,3,3),
-(0,1,0,4,0,5,0,4,0,2,4,4,2,3,3,2,3,3,5,3,3,3,4,3,4,2,3,0,4,3,3,3,4,1,4,3,2,1,5,5,3,4,5,1,3,5,4,2,0,3,3,0,1,3,0,4,2,0,1,3,1,4,3,3,3,3,0,3,0,1,0,3,4,4,4,5,5,0,3,0,1,4,5),
-(0,2,0,3,0,3,0,0,0,2,3,1,3,0,4,0,1,1,3,0,3,4,3,2,3,1,0,3,3,2,3,1,3,0,2,3,0,2,1,4,1,2,2,0,0,3,3,0,0,2,0,0,0,1,0,0,0,0,2,2,0,3,2,1,3,3,0,2,0,2,0,0,3,3,1,2,4,0,3,0,2,2,3),
-(2,4,0,5,0,4,0,4,0,2,4,4,4,3,4,3,3,3,1,2,4,3,4,3,4,4,5,0,3,3,3,3,2,0,4,3,1,4,3,4,1,4,4,3,3,4,4,3,1,2,3,0,4,2,0,4,1,0,3,3,0,4,3,3,3,4,0,4,0,2,0,3,5,3,4,5,2,0,3,0,0,4,5),
-(0,3,0,4,0,1,0,1,0,1,3,2,2,1,3,0,3,0,2,0,2,0,3,0,2,0,0,0,1,0,1,1,0,0,3,1,0,0,0,4,0,3,1,0,2,1,3,0,0,0,0,0,0,3,0,0,0,0,0,0,0,4,2,2,3,1,0,3,0,0,0,1,4,4,4,3,0,0,4,0,0,1,4),
-(1,4,1,5,0,3,0,3,0,4,5,4,4,3,5,3,3,4,4,3,4,1,3,3,3,3,2,1,4,1,5,4,3,1,4,4,3,5,4,4,3,5,4,3,3,4,4,4,0,3,3,1,2,3,0,3,1,0,3,3,0,5,4,4,4,4,4,4,3,3,5,4,4,3,3,5,4,0,3,2,0,4,4),
-(0,2,0,3,0,1,0,0,0,1,3,3,3,2,4,1,3,0,3,1,3,0,2,2,1,1,0,0,2,0,4,3,1,0,4,3,0,4,4,4,1,4,3,1,1,3,3,1,0,2,0,0,1,3,0,0,0,0,2,0,0,4,3,2,4,3,5,4,3,3,3,4,3,3,4,3,3,0,2,1,0,3,3),
-(0,2,0,4,0,3,0,2,0,2,5,5,3,4,4,4,4,1,4,3,3,0,4,3,4,3,1,3,3,2,4,3,0,3,4,3,0,3,4,4,2,4,4,0,4,5,3,3,2,2,1,1,1,2,0,1,5,0,3,3,2,4,3,3,3,4,0,3,0,2,0,4,4,3,5,5,0,0,3,0,2,3,3),
-(0,3,0,4,0,3,0,1,0,3,4,3,3,1,3,3,3,0,3,1,3,0,4,3,3,1,1,0,3,0,3,3,0,0,4,4,0,1,5,4,3,3,5,0,3,3,4,3,0,2,0,1,1,1,0,1,3,0,1,2,1,3,3,2,3,3,0,3,0,1,0,1,3,3,4,4,1,0,1,2,2,1,3),
-(0,1,0,4,0,4,0,3,0,1,3,3,3,2,3,1,1,0,3,0,3,3,4,3,2,4,2,0,1,0,4,3,2,0,4,3,0,5,3,3,2,4,4,4,3,3,3,4,0,1,3,0,0,1,0,0,1,0,0,0,0,4,2,3,3,3,0,3,0,0,0,4,4,4,5,3,2,0,3,3,0,3,5),
-(0,2,0,3,0,0,0,3,0,1,3,0,2,0,0,0,1,0,3,1,1,3,3,0,0,3,0,0,3,0,2,3,1,0,3,1,0,3,3,2,0,4,2,2,0,2,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,2,1,2,0,1,0,1,0,0,0,1,3,1,2,0,0,0,1,0,0,1,4),
-(0,3,0,3,0,5,0,1,0,2,4,3,1,3,3,2,1,1,5,2,1,0,5,1,2,0,0,0,3,3,2,2,3,2,4,3,0,0,3,3,1,3,3,0,2,5,3,4,0,3,3,0,1,2,0,2,2,0,3,2,0,2,2,3,3,3,0,2,0,1,0,3,4,4,2,5,4,0,3,0,0,3,5),
-(0,3,0,3,0,3,0,1,0,3,3,3,3,0,3,0,2,0,2,1,1,0,2,0,1,0,0,0,2,1,0,0,1,0,3,2,0,0,3,3,1,2,3,1,0,3,3,0,0,1,0,0,0,0,0,2,0,0,0,0,0,2,3,1,2,3,0,3,0,1,0,3,2,1,0,4,3,0,1,1,0,3,3),
-(0,4,0,5,0,3,0,3,0,4,5,5,4,3,5,3,4,3,5,3,3,2,5,3,4,4,4,3,4,3,4,5,5,3,4,4,3,4,4,5,4,4,4,3,4,5,5,4,2,3,4,2,3,4,0,3,3,1,4,3,2,4,3,3,5,5,0,3,0,3,0,5,5,5,5,4,4,0,4,0,1,4,4),
-(0,4,0,4,0,3,0,3,0,3,5,4,4,2,3,2,5,1,3,2,5,1,4,2,3,2,3,3,4,3,3,3,3,2,5,4,1,3,3,5,3,4,4,0,4,4,3,1,1,3,1,0,2,3,0,2,3,0,3,0,0,4,3,1,3,4,0,3,0,2,0,4,4,4,3,4,5,0,4,0,0,3,4),
-(0,3,0,3,0,3,1,2,0,3,4,4,3,3,3,0,2,2,4,3,3,1,3,3,3,1,1,0,3,1,4,3,2,3,4,4,2,4,4,4,3,4,4,3,2,4,4,3,1,3,3,1,3,3,0,4,1,0,2,2,1,4,3,2,3,3,5,4,3,3,5,4,4,3,3,0,4,0,3,2,2,4,4),
-(0,2,0,1,0,0,0,0,0,1,2,1,3,0,0,0,0,0,2,0,1,2,1,0,0,1,0,0,0,0,3,0,0,1,0,1,1,3,1,0,0,0,1,1,0,1,1,0,0,0,0,0,2,0,0,0,0,0,0,0,0,1,1,2,2,0,3,4,0,0,0,1,1,0,0,1,0,0,0,0,0,1,1),
-(0,1,0,0,0,1,0,0,0,0,4,0,4,1,4,0,3,0,4,0,3,0,4,0,3,0,3,0,4,1,5,1,4,0,0,3,0,5,0,5,2,0,1,0,0,0,2,1,4,0,1,3,0,0,3,0,0,3,1,1,4,1,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0),
-(1,4,0,5,0,3,0,2,0,3,5,4,4,3,4,3,5,3,4,3,3,0,4,3,3,3,3,3,3,2,4,4,3,1,3,4,4,5,4,4,3,4,4,1,3,5,4,3,3,3,1,2,2,3,3,1,3,1,3,3,3,5,3,3,4,5,0,3,0,3,0,3,4,3,4,4,3,0,3,0,2,4,3),
-(0,1,0,4,0,0,0,0,0,1,4,0,4,1,4,2,4,0,3,0,1,0,1,0,0,0,0,0,2,0,3,1,1,1,0,3,0,0,0,1,2,1,0,0,1,1,1,1,0,1,0,0,0,1,0,0,3,0,0,0,0,3,2,0,2,2,0,1,0,0,0,2,3,2,3,3,0,0,0,0,2,1,0),
-(0,5,1,5,0,3,0,3,0,5,4,4,5,1,5,3,3,0,4,3,4,3,5,3,4,3,3,2,4,3,4,3,3,0,3,3,1,4,4,3,4,4,4,3,4,5,5,3,2,3,1,1,3,3,1,3,1,1,3,3,2,4,5,3,3,5,0,4,0,3,0,4,4,3,5,3,3,0,3,4,0,4,3),
-(0,5,0,5,0,3,0,2,0,4,4,3,5,2,4,3,3,3,4,4,4,3,5,3,5,3,3,1,4,0,4,3,3,0,3,3,0,4,4,4,4,5,4,3,3,5,5,3,2,3,1,2,3,2,0,1,0,0,3,2,2,4,4,3,1,5,0,4,0,3,0,4,3,1,3,2,1,0,3,3,0,3,3),
-(0,4,0,5,0,5,0,4,0,4,5,5,5,3,4,3,3,2,5,4,4,3,5,3,5,3,4,0,4,3,4,4,3,2,4,4,3,4,5,4,4,5,5,0,3,5,5,4,1,3,3,2,3,3,1,3,1,0,4,3,1,4,4,3,4,5,0,4,0,2,0,4,3,4,4,3,3,0,4,0,0,5,5),
-(0,4,0,4,0,5,0,1,1,3,3,4,4,3,4,1,3,0,5,1,3,0,3,1,3,1,1,0,3,0,3,3,4,0,4,3,0,4,4,4,3,4,4,0,3,5,4,1,0,3,0,0,2,3,0,3,1,0,3,1,0,3,2,1,3,5,0,3,0,1,0,3,2,3,3,4,4,0,2,2,0,4,4),
-(2,4,0,5,0,4,0,3,0,4,5,5,4,3,5,3,5,3,5,3,5,2,5,3,4,3,3,4,3,4,5,3,2,1,5,4,3,2,3,4,5,3,4,1,2,5,4,3,0,3,3,0,3,2,0,2,3,0,4,1,0,3,4,3,3,5,0,3,0,1,0,4,5,5,5,4,3,0,4,2,0,3,5),
-(0,5,0,4,0,4,0,2,0,5,4,3,4,3,4,3,3,3,4,3,4,2,5,3,5,3,4,1,4,3,4,4,4,0,3,5,0,4,4,4,4,5,3,1,3,4,5,3,3,3,3,3,3,3,0,2,2,0,3,3,2,4,3,3,3,5,3,4,1,3,3,5,3,2,0,0,0,0,4,3,1,3,3),
-(0,1,0,3,0,3,0,1,0,1,3,3,3,2,3,3,3,0,3,0,0,0,3,1,3,0,0,0,2,2,2,3,0,0,3,2,0,1,2,4,1,3,3,0,0,3,3,3,0,1,0,0,2,1,0,0,3,0,3,1,0,3,0,0,1,3,0,2,0,1,0,3,3,1,3,3,0,0,1,1,0,3,3),
-(0,2,0,3,0,2,1,4,0,2,2,3,1,1,3,1,1,0,2,0,3,1,2,3,1,3,0,0,1,0,4,3,2,3,3,3,1,4,2,3,3,3,3,1,0,3,1,4,0,1,1,0,1,2,0,1,1,0,1,1,0,3,1,3,2,2,0,1,0,0,0,2,3,3,3,1,0,0,0,0,0,2,3),
-(0,5,0,4,0,5,0,2,0,4,5,5,3,3,4,3,3,1,5,4,4,2,4,4,4,3,4,2,4,3,5,5,4,3,3,4,3,3,5,5,4,5,5,1,3,4,5,3,1,4,3,1,3,3,0,3,3,1,4,3,1,4,5,3,3,5,0,4,0,3,0,5,3,3,1,4,3,0,4,0,1,5,3),
-(0,5,0,5,0,4,0,2,0,4,4,3,4,3,3,3,3,3,5,4,4,4,4,4,4,5,3,3,5,2,4,4,4,3,4,4,3,3,4,4,5,5,3,3,4,3,4,3,3,4,3,3,3,3,1,2,2,1,4,3,3,5,4,4,3,4,0,4,0,3,0,4,4,4,4,4,1,0,4,2,0,2,4),
-(0,4,0,4,0,3,0,1,0,3,5,2,3,0,3,0,2,1,4,2,3,3,4,1,4,3,3,2,4,1,3,3,3,0,3,3,0,0,3,3,3,5,3,3,3,3,3,2,0,2,0,0,2,0,0,2,0,0,1,0,0,3,1,2,2,3,0,3,0,2,0,4,4,3,3,4,1,0,3,0,0,2,4),
-(0,0,0,4,0,0,0,0,0,0,1,0,1,0,2,0,0,0,0,0,1,0,2,0,1,0,0,0,0,0,3,1,3,0,3,2,0,0,0,1,0,3,2,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,4,0,2,0,0,0,0,0,0,2),
-(0,2,1,3,0,2,0,2,0,3,3,3,3,1,3,1,3,3,3,3,3,3,4,2,2,1,2,1,4,0,4,3,1,3,3,3,2,4,3,5,4,3,3,3,3,3,3,3,0,1,3,0,2,0,0,1,0,0,1,0,0,4,2,0,2,3,0,3,3,0,3,3,4,2,3,1,4,0,1,2,0,2,3),
-(0,3,0,3,0,1,0,3,0,2,3,3,3,0,3,1,2,0,3,3,2,3,3,2,3,2,3,1,3,0,4,3,2,0,3,3,1,4,3,3,2,3,4,3,1,3,3,1,1,0,1,1,0,1,0,1,0,1,0,0,0,4,1,1,0,3,0,3,1,0,2,3,3,3,3,3,1,0,0,2,0,3,3),
-(0,0,0,0,0,0,0,0,0,0,3,0,2,0,3,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,3,0,3,0,3,1,0,1,0,1,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,2,0,2,3,0,0,0,0,0,0,0,0,3),
-(0,2,0,3,1,3,0,3,0,2,3,3,3,1,3,1,3,1,3,1,3,3,3,1,3,0,2,3,1,1,4,3,3,2,3,3,1,2,2,4,1,3,3,0,1,4,2,3,0,1,3,0,3,0,0,1,3,0,2,0,0,3,3,2,1,3,0,3,0,2,0,3,4,4,4,3,1,0,3,0,0,3,3),
-(0,2,0,1,0,2,0,0,0,1,3,2,2,1,3,0,1,1,3,0,3,2,3,1,2,0,2,0,1,1,3,3,3,0,3,3,1,1,2,3,2,3,3,1,2,3,2,0,0,1,0,0,0,0,0,0,3,0,1,0,0,2,1,2,1,3,0,3,0,0,0,3,4,4,4,3,2,0,2,0,0,2,4),
-(0,0,0,1,0,1,0,0,0,0,1,0,0,0,1,0,0,0,0,0,0,0,1,1,1,0,0,0,0,0,0,0,0,0,2,2,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,1,3,1,0,0,0,0,0,0,0,3),
-(0,3,0,3,0,2,0,3,0,3,3,3,2,3,2,2,2,0,3,1,3,3,3,2,3,3,0,0,3,0,3,2,2,0,2,3,1,4,3,4,3,3,2,3,1,5,4,4,0,3,1,2,1,3,0,3,1,1,2,0,2,3,1,3,1,3,0,3,0,1,0,3,3,4,4,2,1,0,2,1,0,2,4),
-(0,1,0,3,0,1,0,2,0,1,4,2,5,1,4,0,2,0,2,1,3,1,4,0,2,1,0,0,2,1,4,1,1,0,3,3,0,5,1,3,2,3,3,1,0,3,2,3,0,1,0,0,0,0,0,0,1,0,0,0,0,4,0,1,0,3,0,2,0,1,0,3,3,3,4,3,3,0,0,0,0,2,3),
-(0,0,0,1,0,0,0,0,0,0,2,0,1,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,1,0,0,1,0,0,0,0,0,3),
-(0,1,0,3,0,4,0,3,0,2,4,3,1,0,3,2,2,1,3,1,2,2,3,1,1,1,2,1,3,0,1,2,0,1,3,2,1,3,0,5,5,1,0,0,1,3,2,1,0,3,0,0,1,0,0,0,0,0,3,4,0,1,1,1,3,2,0,2,0,1,0,2,3,3,1,2,3,0,1,0,1,0,4),
-(0,0,0,1,0,3,0,3,0,2,2,1,0,0,4,0,3,0,3,1,3,0,3,0,3,0,1,0,3,0,3,1,3,0,3,3,0,0,1,2,1,1,1,0,1,2,0,0,0,1,0,0,1,0,0,0,0,0,0,0,0,2,2,1,2,0,0,2,0,0,0,0,2,3,3,3,3,0,0,0,0,1,4),
-(0,0,0,3,0,3,0,0,0,0,3,1,1,0,3,0,1,0,2,0,1,0,0,0,0,0,0,0,1,0,3,0,2,0,2,3,0,0,2,2,3,1,2,0,0,1,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,2,0,0,0,0,2,3),
-(2,4,0,5,0,5,0,4,0,3,4,3,3,3,4,3,3,3,4,3,4,4,5,4,5,5,5,2,3,0,5,5,4,1,5,4,3,1,5,4,3,4,4,3,3,4,3,3,0,3,2,0,2,3,0,3,0,0,3,3,0,5,3,2,3,3,0,3,0,3,0,3,4,5,4,5,3,0,4,3,0,3,4),
-(0,3,0,3,0,3,0,3,0,3,3,4,3,2,3,2,3,0,4,3,3,3,3,3,3,3,3,0,3,2,4,3,3,1,3,4,3,4,4,4,3,4,4,3,2,4,4,1,0,2,0,0,1,1,0,2,0,0,3,1,0,5,3,2,1,3,0,3,0,1,2,4,3,2,4,3,3,0,3,2,0,4,4),
-(0,3,0,3,0,1,0,0,0,1,4,3,3,2,3,1,3,1,4,2,3,2,4,2,3,4,3,0,2,2,3,3,3,0,3,3,3,0,3,4,1,3,3,0,3,4,3,3,0,1,1,0,1,0,0,0,4,0,3,0,0,3,1,2,1,3,0,4,0,1,0,4,3,3,4,3,3,0,2,0,0,3,3),
-(0,3,0,4,0,1,0,3,0,3,4,3,3,0,3,3,3,1,3,1,3,3,4,3,3,3,0,0,3,1,5,3,3,1,3,3,2,5,4,3,3,4,5,3,2,5,3,4,0,1,0,0,0,0,0,2,0,0,1,1,0,4,2,2,1,3,0,3,0,2,0,4,4,3,5,3,2,0,1,1,0,3,4),
-(0,5,0,4,0,5,0,2,0,4,4,3,3,2,3,3,3,1,4,3,4,1,5,3,4,3,4,0,4,2,4,3,4,1,5,4,0,4,4,4,4,5,4,1,3,5,4,2,1,4,1,1,3,2,0,3,1,0,3,2,1,4,3,3,3,4,0,4,0,3,0,4,4,4,3,3,3,0,4,2,0,3,4),
-(1,4,0,4,0,3,0,1,0,3,3,3,1,1,3,3,2,2,3,3,1,0,3,2,2,1,2,0,3,1,2,1,2,0,3,2,0,2,2,3,3,4,3,0,3,3,1,2,0,1,1,3,1,2,0,0,3,0,1,1,0,3,2,2,3,3,0,3,0,0,0,2,3,3,4,3,3,0,1,0,0,1,4),
-(0,4,0,4,0,4,0,0,0,3,4,4,3,1,4,2,3,2,3,3,3,1,4,3,4,0,3,0,4,2,3,3,2,2,5,4,2,1,3,4,3,4,3,1,3,3,4,2,0,2,1,0,3,3,0,0,2,0,3,1,0,4,4,3,4,3,0,4,0,1,0,2,4,4,4,4,4,0,3,2,0,3,3),
-(0,0,0,1,0,4,0,0,0,0,0,0,1,1,1,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,0,3,2,0,0,1,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,2),
-(0,2,0,3,0,4,0,4,0,1,3,3,3,0,4,0,2,1,2,1,1,1,2,0,3,1,1,0,1,0,3,1,0,0,3,3,2,0,1,1,0,0,0,0,0,1,0,2,0,2,2,0,3,1,0,0,1,0,1,1,0,1,2,0,3,0,0,0,0,1,0,0,3,3,4,3,1,0,1,0,3,0,2),
-(0,0,0,3,0,5,0,0,0,0,1,0,2,0,3,1,0,1,3,0,0,0,2,0,0,0,1,0,0,0,1,1,0,0,4,0,0,0,2,3,0,1,4,1,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,0,0,0,0,1,0,0,0,0,0,0,0,2,0,0,3,0,0,0,0,0,3),
-(0,2,0,5,0,5,0,1,0,2,4,3,3,2,5,1,3,2,3,3,3,0,4,1,2,0,3,0,4,0,2,2,1,1,5,3,0,0,1,4,2,3,2,0,3,3,3,2,0,2,4,1,1,2,0,1,1,0,3,1,0,1,3,1,2,3,0,2,0,0,0,1,3,5,4,4,4,0,3,0,0,1,3),
-(0,4,0,5,0,4,0,4,0,4,5,4,3,3,4,3,3,3,4,3,4,4,5,3,4,5,4,2,4,2,3,4,3,1,4,4,1,3,5,4,4,5,5,4,4,5,5,5,2,3,3,1,4,3,1,3,3,0,3,3,1,4,3,4,4,4,0,3,0,4,0,3,3,4,4,5,0,0,4,3,0,4,5),
-(0,4,0,4,0,3,0,3,0,3,4,4,4,3,3,2,4,3,4,3,4,3,5,3,4,3,2,1,4,2,4,4,3,1,3,4,2,4,5,5,3,4,5,4,1,5,4,3,0,3,2,2,3,2,1,3,1,0,3,3,3,5,3,3,3,5,4,4,2,3,3,4,3,3,3,2,1,0,3,2,1,4,3),
-(0,4,0,5,0,4,0,3,0,3,5,5,3,2,4,3,4,0,5,4,4,1,4,4,4,3,3,3,4,3,5,5,2,3,3,4,1,2,5,5,3,5,5,2,3,5,5,4,0,3,2,0,3,3,1,1,5,1,4,1,0,4,3,2,3,5,0,4,0,3,0,5,4,3,4,3,0,0,4,1,0,4,4),
-(1,3,0,4,0,2,0,2,0,2,5,5,3,3,3,3,3,0,4,2,3,4,4,4,3,4,0,0,3,4,5,4,3,3,3,3,2,5,5,4,5,5,5,4,3,5,5,5,1,3,1,0,1,0,0,3,2,0,4,2,0,5,2,3,2,4,1,3,0,3,0,4,5,4,5,4,3,0,4,2,0,5,4),
-(0,3,0,4,0,5,0,3,0,3,4,4,3,2,3,2,3,3,3,3,3,2,4,3,3,2,2,0,3,3,3,3,3,1,3,3,3,0,4,4,3,4,4,1,1,4,4,2,0,3,1,0,1,1,0,4,1,0,2,3,1,3,3,1,3,4,0,3,0,1,0,3,1,3,0,0,1,0,2,0,0,4,4),
-(0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0),
-(0,3,0,3,0,2,0,3,0,1,5,4,3,3,3,1,4,2,1,2,3,4,4,2,4,4,5,0,3,1,4,3,4,0,4,3,3,3,2,3,2,5,3,4,3,2,2,3,0,0,3,0,2,1,0,1,2,0,0,0,0,2,1,1,3,1,0,2,0,4,0,3,4,4,4,5,2,0,2,0,0,1,3),
-(0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,1,1,1,0,0,1,1,0,0,0,4,2,1,1,0,1,0,3,2,0,0,3,1,1,1,2,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,1,0,0,0,2,0,0,0,1,4,0,4,2,1,0,0,0,0,0,1),
-(0,0,0,0,0,0,0,0,0,1,0,1,0,0,0,0,1,0,0,0,0,0,0,1,0,1,0,0,0,0,3,1,0,0,0,2,0,2,1,0,0,1,2,1,0,1,1,0,0,3,0,0,0,0,0,0,0,0,0,0,0,1,3,1,0,0,0,0,0,1,0,0,2,1,0,0,0,0,0,0,0,0,2),
-(0,4,0,4,0,4,0,3,0,4,4,3,4,2,4,3,2,0,4,4,4,3,5,3,5,3,3,2,4,2,4,3,4,3,1,4,0,2,3,4,4,4,3,3,3,4,4,4,3,4,1,3,4,3,2,1,2,1,3,3,3,4,4,3,3,5,0,4,0,3,0,4,3,3,3,2,1,0,3,0,0,3,3),
-(0,4,0,3,0,3,0,3,0,3,5,5,3,3,3,3,4,3,4,3,3,3,4,4,4,3,3,3,3,4,3,5,3,3,1,3,2,4,5,5,5,5,4,3,4,5,5,3,2,2,3,3,3,3,2,3,3,1,2,3,2,4,3,3,3,4,0,4,0,2,0,4,3,2,2,1,2,0,3,0,0,4,1),
+# fmt: off
+jp2_char_context = (
+    (0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1),
+    (2, 4, 0, 4, 0, 3, 0, 4, 0, 3, 4, 4, 4, 2, 4, 3, 3, 4, 3, 2, 3, 3, 4, 2, 3, 3, 3, 2, 4, 1, 4, 3, 3, 1, 5, 4, 3, 4, 3, 4, 3, 5, 3, 0, 3, 5, 4, 2, 0, 3, 1, 0, 3, 3, 0, 3, 3, 0, 1, 1, 0, 4, 3, 0, 3, 3, 0, 4, 0, 2, 0, 3, 5, 5, 5, 5, 4, 0, 4, 1, 0, 3, 4),
+    (0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2),
+    (0, 4, 0, 5, 0, 5, 0, 4, 0, 4, 5, 4, 4, 3, 5, 3, 5, 1, 5, 3, 4, 3, 4, 4, 3, 4, 3, 3, 4, 3, 5, 4, 4, 3, 5, 5, 3, 5, 5, 5, 3, 5, 5, 3, 4, 5, 5, 3, 1, 3, 2, 0, 3, 4, 0, 4, 2, 0, 4, 2, 1, 5, 3, 2, 3, 5, 0, 4, 0, 2, 0, 5, 4, 4, 5, 4, 5, 0, 4, 0, 0, 4, 4),
+    (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0),
+    (0, 3, 0, 4, 0, 3, 0, 3, 0, 4, 5, 4, 3, 3, 3, 3, 4, 3, 5, 4, 4, 3, 5, 4, 4, 3, 4, 3, 4, 4, 4, 4, 5, 3, 4, 4, 3, 4, 5, 5, 4, 5, 5, 1, 4, 5, 4, 3, 0, 3, 3, 1, 3, 3, 0, 4, 4, 0, 3, 3, 1, 5, 3, 3, 3, 5, 0, 4, 0, 3, 0, 4, 4, 3, 4, 3, 3, 0, 4, 1, 1, 3, 4),
+    (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0),
+    (0, 4, 0, 3, 0, 3, 0, 4, 0, 3, 4, 4, 3, 2, 2, 1, 2, 1, 3, 1, 3, 3, 3, 3, 3, 4, 3, 1, 3, 3, 5, 3, 3, 0, 4, 3, 0, 5, 4, 3, 3, 5, 4, 4, 3, 4, 4, 5, 0, 1, 2, 0, 1, 2, 0, 2, 2, 0, 1, 0, 0, 5, 2, 2, 1, 4, 0, 3, 0, 1, 0, 4, 4, 3, 5, 4, 3, 0, 2, 1, 0, 4, 3),
+    (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0),
+    (0, 3, 0, 5, 0, 4, 0, 2, 1, 4, 4, 2, 4, 1, 4, 2, 4, 2, 4, 3, 3, 3, 4, 3, 3, 3, 3, 1, 4, 2, 3, 3, 3, 1, 4, 4, 1, 1, 1, 4, 3, 3, 2, 0, 2, 4, 3, 2, 0, 3, 3, 0, 3, 1, 1, 0, 0, 0, 3, 3, 0, 4, 2, 2, 3, 4, 0, 4, 0, 3, 0, 4, 4, 5, 3, 4, 4, 0, 3, 0, 0, 1, 4),
+    (1, 4, 0, 4, 0, 4, 0, 4, 0, 3, 5, 4, 4, 3, 4, 3, 5, 4, 3, 3, 4, 3, 5, 4, 4, 4, 4, 3, 4, 2, 4, 3, 3, 1, 5, 4, 3, 2, 4, 5, 4, 5, 5, 4, 4, 5, 4, 4, 0, 3, 2, 2, 3, 3, 0, 4, 3, 1, 3, 2, 1, 4, 3, 3, 4, 5, 0, 3, 0, 2, 0, 4, 5, 5, 4, 5, 4, 0, 4, 0, 0, 5, 4),
+    (0, 5, 0, 5, 0, 4, 0, 3, 0, 4, 4, 3, 4, 3, 3, 3, 4, 0, 4, 4, 4, 3, 4, 3, 4, 3, 3, 1, 4, 2, 4, 3, 4, 0, 5, 4, 1, 4, 5, 4, 4, 5, 3, 2, 4, 3, 4, 3, 2, 4, 1, 3, 3, 3, 2, 3, 2, 0, 4, 3, 3, 4, 3, 3, 3, 4, 0, 4, 0, 3, 0, 4, 5, 4, 4, 4, 3, 0, 4, 1, 0, 1, 3),
+    (0, 3, 1, 4, 0, 3, 0, 2, 0, 3, 4, 4, 3, 1, 4, 2, 3, 3, 4, 3, 4, 3, 4, 3, 4, 4, 3, 2, 3, 1, 5, 4, 4, 1, 4, 4, 3, 5, 4, 4, 3, 5, 5, 4, 3, 4, 4, 3, 1, 2, 3, 1, 2, 2, 0, 3, 2, 0, 3, 1, 0, 5, 3, 3, 3, 4, 3, 3, 3, 3, 4, 4, 4, 4, 5, 4, 2, 0, 3, 3, 2, 4, 3),
+    (0, 2, 0, 3, 0, 1, 0, 1, 0, 0, 3, 2, 0, 0, 2, 0, 1, 0, 2, 1, 3, 3, 3, 1, 2, 3, 1, 0, 1, 0, 4, 2, 1, 1, 3, 3, 0, 4, 3, 3, 1, 4, 3, 3, 0, 3, 3, 2, 0, 0, 0, 0, 1, 0, 0, 2, 0, 0, 0, 0, 0, 4, 1, 0, 2, 3, 2, 2, 2, 1, 3, 3, 3, 4, 4, 3, 2, 0, 3, 1, 0, 3, 3),
+    (0, 4, 0, 4, 0, 3, 0, 3, 0, 4, 4, 4, 3, 3, 3, 3, 3, 3, 4, 3, 4, 2, 4, 3, 4, 3, 3, 2, 4, 3, 4, 5, 4, 1, 4, 5, 3, 5, 4, 5, 3, 5, 4, 0, 3, 5, 5, 3, 1, 3, 3, 2, 2, 3, 0, 3, 4, 1, 3, 3, 2, 4, 3, 3, 3, 4, 0, 4, 0, 3, 0, 4, 5, 4, 4, 5, 3, 0, 4, 1, 0, 3, 4),
+    (0, 2, 0, 3, 0, 3, 0, 0, 0, 2, 2, 2, 1, 0, 1, 0, 0, 0, 3, 0, 3, 0, 3, 0, 1, 3, 1, 0, 3, 1, 3, 3, 3, 1, 3, 3, 3, 0, 1, 3, 1, 3, 4, 0, 0, 3, 1, 1, 0, 3, 2, 0, 0, 0, 0, 1, 3, 0, 1, 0, 0, 3, 3, 2, 0, 3, 0, 0, 0, 0, 0, 3, 4, 3, 4, 3, 3, 0, 3, 0, 0, 2, 3),
+    (2, 3, 0, 3, 0, 2, 0, 1, 0, 3, 3, 4, 3, 1, 3, 1, 1, 1, 3, 1, 4, 3, 4, 3, 3, 3, 0, 0, 3, 1, 5, 4, 3, 1, 4, 3, 2, 5, 5, 4, 4, 4, 4, 3, 3, 4, 4, 4, 0, 2, 1, 1, 3, 2, 0, 1, 2, 0, 0, 1, 0, 4, 1, 3, 3, 3, 0, 3, 0, 1, 0, 4, 4, 4, 5, 5, 3, 0, 2, 0, 0, 4, 4),
+    (0, 2, 0, 1, 0, 3, 1, 3, 0, 2, 3, 3, 3, 0, 3, 1, 0, 0, 3, 0, 3, 2, 3, 1, 3, 2, 1, 1, 0, 0, 4, 2, 1, 0, 2, 3, 1, 4, 3, 2, 0, 4, 4, 3, 1, 3, 1, 3, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 4, 1, 1, 1, 2, 0, 3, 0, 0, 0, 3, 4, 2, 4, 3, 2, 0, 1, 0, 0, 3, 3),
+    (0, 1, 0, 4, 0, 5, 0, 4, 0, 2, 4, 4, 2, 3, 3, 2, 3, 3, 5, 3, 3, 3, 4, 3, 4, 2, 3, 0, 4, 3, 3, 3, 4, 1, 4, 3, 2, 1, 5, 5, 3, 4, 5, 1, 3, 5, 4, 2, 0, 3, 3, 0, 1, 3, 0, 4, 2, 0, 1, 3, 1, 4, 3, 3, 3, 3, 0, 3, 0, 1, 0, 3, 4, 4, 4, 5, 5, 0, 3, 0, 1, 4, 5),
+    (0, 2, 0, 3, 0, 3, 0, 0, 0, 2, 3, 1, 3, 0, 4, 0, 1, 1, 3, 0, 3, 4, 3, 2, 3, 1, 0, 3, 3, 2, 3, 1, 3, 0, 2, 3, 0, 2, 1, 4, 1, 2, 2, 0, 0, 3, 3, 0, 0, 2, 0, 0, 0, 1, 0, 0, 0, 0, 2, 2, 0, 3, 2, 1, 3, 3, 0, 2, 0, 2, 0, 0, 3, 3, 1, 2, 4, 0, 3, 0, 2, 2, 3),
+    (2, 4, 0, 5, 0, 4, 0, 4, 0, 2, 4, 4, 4, 3, 4, 3, 3, 3, 1, 2, 4, 3, 4, 3, 4, 4, 5, 0, 3, 3, 3, 3, 2, 0, 4, 3, 1, 4, 3, 4, 1, 4, 4, 3, 3, 4, 4, 3, 1, 2, 3, 0, 4, 2, 0, 4, 1, 0, 3, 3, 0, 4, 3, 3, 3, 4, 0, 4, 0, 2, 0, 3, 5, 3, 4, 5, 2, 0, 3, 0, 0, 4, 5),
+    (0, 3, 0, 4, 0, 1, 0, 1, 0, 1, 3, 2, 2, 1, 3, 0, 3, 0, 2, 0, 2, 0, 3, 0, 2, 0, 0, 0, 1, 0, 1, 1, 0, 0, 3, 1, 0, 0, 0, 4, 0, 3, 1, 0, 2, 1, 3, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 4, 2, 2, 3, 1, 0, 3, 0, 0, 0, 1, 4, 4, 4, 3, 0, 0, 4, 0, 0, 1, 4),
+    (1, 4, 1, 5, 0, 3, 0, 3, 0, 4, 5, 4, 4, 3, 5, 3, 3, 4, 4, 3, 4, 1, 3, 3, 3, 3, 2, 1, 4, 1, 5, 4, 3, 1, 4, 4, 3, 5, 4, 4, 3, 5, 4, 3, 3, 4, 4, 4, 0, 3, 3, 1, 2, 3, 0, 3, 1, 0, 3, 3, 0, 5, 4, 4, 4, 4, 4, 4, 3, 3, 5, 4, 4, 3, 3, 5, 4, 0, 3, 2, 0, 4, 4),
+    (0, 2, 0, 3, 0, 1, 0, 0, 0, 1, 3, 3, 3, 2, 4, 1, 3, 0, 3, 1, 3, 0, 2, 2, 1, 1, 0, 0, 2, 0, 4, 3, 1, 0, 4, 3, 0, 4, 4, 4, 1, 4, 3, 1, 1, 3, 3, 1, 0, 2, 0, 0, 1, 3, 0, 0, 0, 0, 2, 0, 0, 4, 3, 2, 4, 3, 5, 4, 3, 3, 3, 4, 3, 3, 4, 3, 3, 0, 2, 1, 0, 3, 3),
+    (0, 2, 0, 4, 0, 3, 0, 2, 0, 2, 5, 5, 3, 4, 4, 4, 4, 1, 4, 3, 3, 0, 4, 3, 4, 3, 1, 3, 3, 2, 4, 3, 0, 3, 4, 3, 0, 3, 4, 4, 2, 4, 4, 0, 4, 5, 3, 3, 2, 2, 1, 1, 1, 2, 0, 1, 5, 0, 3, 3, 2, 4, 3, 3, 3, 4, 0, 3, 0, 2, 0, 4, 4, 3, 5, 5, 0, 0, 3, 0, 2, 3, 3),
+    (0, 3, 0, 4, 0, 3, 0, 1, 0, 3, 4, 3, 3, 1, 3, 3, 3, 0, 3, 1, 3, 0, 4, 3, 3, 1, 1, 0, 3, 0, 3, 3, 0, 0, 4, 4, 0, 1, 5, 4, 3, 3, 5, 0, 3, 3, 4, 3, 0, 2, 0, 1, 1, 1, 0, 1, 3, 0, 1, 2, 1, 3, 3, 2, 3, 3, 0, 3, 0, 1, 0, 1, 3, 3, 4, 4, 1, 0, 1, 2, 2, 1, 3),
+    (0, 1, 0, 4, 0, 4, 0, 3, 0, 1, 3, 3, 3, 2, 3, 1, 1, 0, 3, 0, 3, 3, 4, 3, 2, 4, 2, 0, 1, 0, 4, 3, 2, 0, 4, 3, 0, 5, 3, 3, 2, 4, 4, 4, 3, 3, 3, 4, 0, 1, 3, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 4, 2, 3, 3, 3, 0, 3, 0, 0, 0, 4, 4, 4, 5, 3, 2, 0, 3, 3, 0, 3, 5),
+    (0, 2, 0, 3, 0, 0, 0, 3, 0, 1, 3, 0, 2, 0, 0, 0, 1, 0, 3, 1, 1, 3, 3, 0, 0, 3, 0, 0, 3, 0, 2, 3, 1, 0, 3, 1, 0, 3, 3, 2, 0, 4, 2, 2, 0, 2, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 1, 2, 0, 1, 0, 1, 0, 0, 0, 1, 3, 1, 2, 0, 0, 0, 1, 0, 0, 1, 4),
+    (0, 3, 0, 3, 0, 5, 0, 1, 0, 2, 4, 3, 1, 3, 3, 2, 1, 1, 5, 2, 1, 0, 5, 1, 2, 0, 0, 0, 3, 3, 2, 2, 3, 2, 4, 3, 0, 0, 3, 3, 1, 3, 3, 0, 2, 5, 3, 4, 0, 3, 3, 0, 1, 2, 0, 2, 2, 0, 3, 2, 0, 2, 2, 3, 3, 3, 0, 2, 0, 1, 0, 3, 4, 4, 2, 5, 4, 0, 3, 0, 0, 3, 5),
+    (0, 3, 0, 3, 0, 3, 0, 1, 0, 3, 3, 3, 3, 0, 3, 0, 2, 0, 2, 1, 1, 0, 2, 0, 1, 0, 0, 0, 2, 1, 0, 0, 1, 0, 3, 2, 0, 0, 3, 3, 1, 2, 3, 1, 0, 3, 3, 0, 0, 1, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 2, 3, 1, 2, 3, 0, 3, 0, 1, 0, 3, 2, 1, 0, 4, 3, 0, 1, 1, 0, 3, 3),
+    (0, 4, 0, 5, 0, 3, 0, 3, 0, 4, 5, 5, 4, 3, 5, 3, 4, 3, 5, 3, 3, 2, 5, 3, 4, 4, 4, 3, 4, 3, 4, 5, 5, 3, 4, 4, 3, 4, 4, 5, 4, 4, 4, 3, 4, 5, 5, 4, 2, 3, 4, 2, 3, 4, 0, 3, 3, 1, 4, 3, 2, 4, 3, 3, 5, 5, 0, 3, 0, 3, 0, 5, 5, 5, 5, 4, 4, 0, 4, 0, 1, 4, 4),
+    (0, 4, 0, 4, 0, 3, 0, 3, 0, 3, 5, 4, 4, 2, 3, 2, 5, 1, 3, 2, 5, 1, 4, 2, 3, 2, 3, 3, 4, 3, 3, 3, 3, 2, 5, 4, 1, 3, 3, 5, 3, 4, 4, 0, 4, 4, 3, 1, 1, 3, 1, 0, 2, 3, 0, 2, 3, 0, 3, 0, 0, 4, 3, 1, 3, 4, 0, 3, 0, 2, 0, 4, 4, 4, 3, 4, 5, 0, 4, 0, 0, 3, 4),
+    (0, 3, 0, 3, 0, 3, 1, 2, 0, 3, 4, 4, 3, 3, 3, 0, 2, 2, 4, 3, 3, 1, 3, 3, 3, 1, 1, 0, 3, 1, 4, 3, 2, 3, 4, 4, 2, 4, 4, 4, 3, 4, 4, 3, 2, 4, 4, 3, 1, 3, 3, 1, 3, 3, 0, 4, 1, 0, 2, 2, 1, 4, 3, 2, 3, 3, 5, 4, 3, 3, 5, 4, 4, 3, 3, 0, 4, 0, 3, 2, 2, 4, 4),
+    (0, 2, 0, 1, 0, 0, 0, 0, 0, 1, 2, 1, 3, 0, 0, 0, 0, 0, 2, 0, 1, 2, 1, 0, 0, 1, 0, 0, 0, 0, 3, 0, 0, 1, 0, 1, 1, 3, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 2, 2, 0, 3, 4, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1),
+    (0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 4, 0, 4, 1, 4, 0, 3, 0, 4, 0, 3, 0, 4, 0, 3, 0, 3, 0, 4, 1, 5, 1, 4, 0, 0, 3, 0, 5, 0, 5, 2, 0, 1, 0, 0, 0, 2, 1, 4, 0, 1, 3, 0, 0, 3, 0, 0, 3, 1, 1, 4, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0),
+    (1, 4, 0, 5, 0, 3, 0, 2, 0, 3, 5, 4, 4, 3, 4, 3, 5, 3, 4, 3, 3, 0, 4, 3, 3, 3, 3, 3, 3, 2, 4, 4, 3, 1, 3, 4, 4, 5, 4, 4, 3, 4, 4, 1, 3, 5, 4, 3, 3, 3, 1, 2, 2, 3, 3, 1, 3, 1, 3, 3, 3, 5, 3, 3, 4, 5, 0, 3, 0, 3, 0, 3, 4, 3, 4, 4, 3, 0, 3, 0, 2, 4, 3),
+    (0, 1, 0, 4, 0, 0, 0, 0, 0, 1, 4, 0, 4, 1, 4, 2, 4, 0, 3, 0, 1, 0, 1, 0, 0, 0, 0, 0, 2, 0, 3, 1, 1, 1, 0, 3, 0, 0, 0, 1, 2, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 1, 0, 0, 3, 0, 0, 0, 0, 3, 2, 0, 2, 2, 0, 1, 0, 0, 0, 2, 3, 2, 3, 3, 0, 0, 0, 0, 2, 1, 0),
+    (0, 5, 1, 5, 0, 3, 0, 3, 0, 5, 4, 4, 5, 1, 5, 3, 3, 0, 4, 3, 4, 3, 5, 3, 4, 3, 3, 2, 4, 3, 4, 3, 3, 0, 3, 3, 1, 4, 4, 3, 4, 4, 4, 3, 4, 5, 5, 3, 2, 3, 1, 1, 3, 3, 1, 3, 1, 1, 3, 3, 2, 4, 5, 3, 3, 5, 0, 4, 0, 3, 0, 4, 4, 3, 5, 3, 3, 0, 3, 4, 0, 4, 3),
+    (0, 5, 0, 5, 0, 3, 0, 2, 0, 4, 4, 3, 5, 2, 4, 3, 3, 3, 4, 4, 4, 3, 5, 3, 5, 3, 3, 1, 4, 0, 4, 3, 3, 0, 3, 3, 0, 4, 4, 4, 4, 5, 4, 3, 3, 5, 5, 3, 2, 3, 1, 2, 3, 2, 0, 1, 0, 0, 3, 2, 2, 4, 4, 3, 1, 5, 0, 4, 0, 3, 0, 4, 3, 1, 3, 2, 1, 0, 3, 3, 0, 3, 3),
+    (0, 4, 0, 5, 0, 5, 0, 4, 0, 4, 5, 5, 5, 3, 4, 3, 3, 2, 5, 4, 4, 3, 5, 3, 5, 3, 4, 0, 4, 3, 4, 4, 3, 2, 4, 4, 3, 4, 5, 4, 4, 5, 5, 0, 3, 5, 5, 4, 1, 3, 3, 2, 3, 3, 1, 3, 1, 0, 4, 3, 1, 4, 4, 3, 4, 5, 0, 4, 0, 2, 0, 4, 3, 4, 4, 3, 3, 0, 4, 0, 0, 5, 5),
+    (0, 4, 0, 4, 0, 5, 0, 1, 1, 3, 3, 4, 4, 3, 4, 1, 3, 0, 5, 1, 3, 0, 3, 1, 3, 1, 1, 0, 3, 0, 3, 3, 4, 0, 4, 3, 0, 4, 4, 4, 3, 4, 4, 0, 3, 5, 4, 1, 0, 3, 0, 0, 2, 3, 0, 3, 1, 0, 3, 1, 0, 3, 2, 1, 3, 5, 0, 3, 0, 1, 0, 3, 2, 3, 3, 4, 4, 0, 2, 2, 0, 4, 4),
+    (2, 4, 0, 5, 0, 4, 0, 3, 0, 4, 5, 5, 4, 3, 5, 3, 5, 3, 5, 3, 5, 2, 5, 3, 4, 3, 3, 4, 3, 4, 5, 3, 2, 1, 5, 4, 3, 2, 3, 4, 5, 3, 4, 1, 2, 5, 4, 3, 0, 3, 3, 0, 3, 2, 0, 2, 3, 0, 4, 1, 0, 3, 4, 3, 3, 5, 0, 3, 0, 1, 0, 4, 5, 5, 5, 4, 3, 0, 4, 2, 0, 3, 5),
+    (0, 5, 0, 4, 0, 4, 0, 2, 0, 5, 4, 3, 4, 3, 4, 3, 3, 3, 4, 3, 4, 2, 5, 3, 5, 3, 4, 1, 4, 3, 4, 4, 4, 0, 3, 5, 0, 4, 4, 4, 4, 5, 3, 1, 3, 4, 5, 3, 3, 3, 3, 3, 3, 3, 0, 2, 2, 0, 3, 3, 2, 4, 3, 3, 3, 5, 3, 4, 1, 3, 3, 5, 3, 2, 0, 0, 0, 0, 4, 3, 1, 3, 3),
+    (0, 1, 0, 3, 0, 3, 0, 1, 0, 1, 3, 3, 3, 2, 3, 3, 3, 0, 3, 0, 0, 0, 3, 1, 3, 0, 0, 0, 2, 2, 2, 3, 0, 0, 3, 2, 0, 1, 2, 4, 1, 3, 3, 0, 0, 3, 3, 3, 0, 1, 0, 0, 2, 1, 0, 0, 3, 0, 3, 1, 0, 3, 0, 0, 1, 3, 0, 2, 0, 1, 0, 3, 3, 1, 3, 3, 0, 0, 1, 1, 0, 3, 3),
+    (0, 2, 0, 3, 0, 2, 1, 4, 0, 2, 2, 3, 1, 1, 3, 1, 1, 0, 2, 0, 3, 1, 2, 3, 1, 3, 0, 0, 1, 0, 4, 3, 2, 3, 3, 3, 1, 4, 2, 3, 3, 3, 3, 1, 0, 3, 1, 4, 0, 1, 1, 0, 1, 2, 0, 1, 1, 0, 1, 1, 0, 3, 1, 3, 2, 2, 0, 1, 0, 0, 0, 2, 3, 3, 3, 1, 0, 0, 0, 0, 0, 2, 3),
+    (0, 5, 0, 4, 0, 5, 0, 2, 0, 4, 5, 5, 3, 3, 4, 3, 3, 1, 5, 4, 4, 2, 4, 4, 4, 3, 4, 2, 4, 3, 5, 5, 4, 3, 3, 4, 3, 3, 5, 5, 4, 5, 5, 1, 3, 4, 5, 3, 1, 4, 3, 1, 3, 3, 0, 3, 3, 1, 4, 3, 1, 4, 5, 3, 3, 5, 0, 4, 0, 3, 0, 5, 3, 3, 1, 4, 3, 0, 4, 0, 1, 5, 3),
+    (0, 5, 0, 5, 0, 4, 0, 2, 0, 4, 4, 3, 4, 3, 3, 3, 3, 3, 5, 4, 4, 4, 4, 4, 4, 5, 3, 3, 5, 2, 4, 4, 4, 3, 4, 4, 3, 3, 4, 4, 5, 5, 3, 3, 4, 3, 4, 3, 3, 4, 3, 3, 3, 3, 1, 2, 2, 1, 4, 3, 3, 5, 4, 4, 3, 4, 0, 4, 0, 3, 0, 4, 4, 4, 4, 4, 1, 0, 4, 2, 0, 2, 4),
+    (0, 4, 0, 4, 0, 3, 0, 1, 0, 3, 5, 2, 3, 0, 3, 0, 2, 1, 4, 2, 3, 3, 4, 1, 4, 3, 3, 2, 4, 1, 3, 3, 3, 0, 3, 3, 0, 0, 3, 3, 3, 5, 3, 3, 3, 3, 3, 2, 0, 2, 0, 0, 2, 0, 0, 2, 0, 0, 1, 0, 0, 3, 1, 2, 2, 3, 0, 3, 0, 2, 0, 4, 4, 3, 3, 4, 1, 0, 3, 0, 0, 2, 4),
+    (0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 2, 0, 0, 0, 0, 0, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, 3, 1, 3, 0, 3, 2, 0, 0, 0, 1, 0, 3, 2, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 4, 0, 2, 0, 0, 0, 0, 0, 0, 2),
+    (0, 2, 1, 3, 0, 2, 0, 2, 0, 3, 3, 3, 3, 1, 3, 1, 3, 3, 3, 3, 3, 3, 4, 2, 2, 1, 2, 1, 4, 0, 4, 3, 1, 3, 3, 3, 2, 4, 3, 5, 4, 3, 3, 3, 3, 3, 3, 3, 0, 1, 3, 0, 2, 0, 0, 1, 0, 0, 1, 0, 0, 4, 2, 0, 2, 3, 0, 3, 3, 0, 3, 3, 4, 2, 3, 1, 4, 0, 1, 2, 0, 2, 3),
+    (0, 3, 0, 3, 0, 1, 0, 3, 0, 2, 3, 3, 3, 0, 3, 1, 2, 0, 3, 3, 2, 3, 3, 2, 3, 2, 3, 1, 3, 0, 4, 3, 2, 0, 3, 3, 1, 4, 3, 3, 2, 3, 4, 3, 1, 3, 3, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 0, 0, 4, 1, 1, 0, 3, 0, 3, 1, 0, 2, 3, 3, 3, 3, 3, 1, 0, 0, 2, 0, 3, 3),
+    (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 2, 0, 3, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 3, 0, 3, 0, 3, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 2, 0, 2, 3, 0, 0, 0, 0, 0, 0, 0, 0, 3),
+    (0, 2, 0, 3, 1, 3, 0, 3, 0, 2, 3, 3, 3, 1, 3, 1, 3, 1, 3, 1, 3, 3, 3, 1, 3, 0, 2, 3, 1, 1, 4, 3, 3, 2, 3, 3, 1, 2, 2, 4, 1, 3, 3, 0, 1, 4, 2, 3, 0, 1, 3, 0, 3, 0, 0, 1, 3, 0, 2, 0, 0, 3, 3, 2, 1, 3, 0, 3, 0, 2, 0, 3, 4, 4, 4, 3, 1, 0, 3, 0, 0, 3, 3),
+    (0, 2, 0, 1, 0, 2, 0, 0, 0, 1, 3, 2, 2, 1, 3, 0, 1, 1, 3, 0, 3, 2, 3, 1, 2, 0, 2, 0, 1, 1, 3, 3, 3, 0, 3, 3, 1, 1, 2, 3, 2, 3, 3, 1, 2, 3, 2, 0, 0, 1, 0, 0, 0, 0, 0, 0, 3, 0, 1, 0, 0, 2, 1, 2, 1, 3, 0, 3, 0, 0, 0, 3, 4, 4, 4, 3, 2, 0, 2, 0, 0, 2, 4),
+    (0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 3, 1, 0, 0, 0, 0, 0, 0, 0, 3),
+    (0, 3, 0, 3, 0, 2, 0, 3, 0, 3, 3, 3, 2, 3, 2, 2, 2, 0, 3, 1, 3, 3, 3, 2, 3, 3, 0, 0, 3, 0, 3, 2, 2, 0, 2, 3, 1, 4, 3, 4, 3, 3, 2, 3, 1, 5, 4, 4, 0, 3, 1, 2, 1, 3, 0, 3, 1, 1, 2, 0, 2, 3, 1, 3, 1, 3, 0, 3, 0, 1, 0, 3, 3, 4, 4, 2, 1, 0, 2, 1, 0, 2, 4),
+    (0, 1, 0, 3, 0, 1, 0, 2, 0, 1, 4, 2, 5, 1, 4, 0, 2, 0, 2, 1, 3, 1, 4, 0, 2, 1, 0, 0, 2, 1, 4, 1, 1, 0, 3, 3, 0, 5, 1, 3, 2, 3, 3, 1, 0, 3, 2, 3, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 4, 0, 1, 0, 3, 0, 2, 0, 1, 0, 3, 3, 3, 4, 3, 3, 0, 0, 0, 0, 2, 3),
+    (0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 2, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 1, 0, 0, 1, 0, 0, 0, 0, 0, 3),
+    (0, 1, 0, 3, 0, 4, 0, 3, 0, 2, 4, 3, 1, 0, 3, 2, 2, 1, 3, 1, 2, 2, 3, 1, 1, 1, 2, 1, 3, 0, 1, 2, 0, 1, 3, 2, 1, 3, 0, 5, 5, 1, 0, 0, 1, 3, 2, 1, 0, 3, 0, 0, 1, 0, 0, 0, 0, 0, 3, 4, 0, 1, 1, 1, 3, 2, 0, 2, 0, 1, 0, 2, 3, 3, 1, 2, 3, 0, 1, 0, 1, 0, 4),
+    (0, 0, 0, 1, 0, 3, 0, 3, 0, 2, 2, 1, 0, 0, 4, 0, 3, 0, 3, 1, 3, 0, 3, 0, 3, 0, 1, 0, 3, 0, 3, 1, 3, 0, 3, 3, 0, 0, 1, 2, 1, 1, 1, 0, 1, 2, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 1, 2, 0, 0, 2, 0, 0, 0, 0, 2, 3, 3, 3, 3, 0, 0, 0, 0, 1, 4),
+    (0, 0, 0, 3, 0, 3, 0, 0, 0, 0, 3, 1, 1, 0, 3, 0, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 3, 0, 2, 0, 2, 3, 0, 0, 2, 2, 3, 1, 2, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 2, 0, 0, 0, 0, 2, 3),
+    (2, 4, 0, 5, 0, 5, 0, 4, 0, 3, 4, 3, 3, 3, 4, 3, 3, 3, 4, 3, 4, 4, 5, 4, 5, 5, 5, 2, 3, 0, 5, 5, 4, 1, 5, 4, 3, 1, 5, 4, 3, 4, 4, 3, 3, 4, 3, 3, 0, 3, 2, 0, 2, 3, 0, 3, 0, 0, 3, 3, 0, 5, 3, 2, 3, 3, 0, 3, 0, 3, 0, 3, 4, 5, 4, 5, 3, 0, 4, 3, 0, 3, 4),
+    (0, 3, 0, 3, 0, 3, 0, 3, 0, 3, 3, 4, 3, 2, 3, 2, 3, 0, 4, 3, 3, 3, 3, 3, 3, 3, 3, 0, 3, 2, 4, 3, 3, 1, 3, 4, 3, 4, 4, 4, 3, 4, 4, 3, 2, 4, 4, 1, 0, 2, 0, 0, 1, 1, 0, 2, 0, 0, 3, 1, 0, 5, 3, 2, 1, 3, 0, 3, 0, 1, 2, 4, 3, 2, 4, 3, 3, 0, 3, 2, 0, 4, 4),
+    (0, 3, 0, 3, 0, 1, 0, 0, 0, 1, 4, 3, 3, 2, 3, 1, 3, 1, 4, 2, 3, 2, 4, 2, 3, 4, 3, 0, 2, 2, 3, 3, 3, 0, 3, 3, 3, 0, 3, 4, 1, 3, 3, 0, 3, 4, 3, 3, 0, 1, 1, 0, 1, 0, 0, 0, 4, 0, 3, 0, 0, 3, 1, 2, 1, 3, 0, 4, 0, 1, 0, 4, 3, 3, 4, 3, 3, 0, 2, 0, 0, 3, 3),
+    (0, 3, 0, 4, 0, 1, 0, 3, 0, 3, 4, 3, 3, 0, 3, 3, 3, 1, 3, 1, 3, 3, 4, 3, 3, 3, 0, 0, 3, 1, 5, 3, 3, 1, 3, 3, 2, 5, 4, 3, 3, 4, 5, 3, 2, 5, 3, 4, 0, 1, 0, 0, 0, 0, 0, 2, 0, 0, 1, 1, 0, 4, 2, 2, 1, 3, 0, 3, 0, 2, 0, 4, 4, 3, 5, 3, 2, 0, 1, 1, 0, 3, 4),
+    (0, 5, 0, 4, 0, 5, 0, 2, 0, 4, 4, 3, 3, 2, 3, 3, 3, 1, 4, 3, 4, 1, 5, 3, 4, 3, 4, 0, 4, 2, 4, 3, 4, 1, 5, 4, 0, 4, 4, 4, 4, 5, 4, 1, 3, 5, 4, 2, 1, 4, 1, 1, 3, 2, 0, 3, 1, 0, 3, 2, 1, 4, 3, 3, 3, 4, 0, 4, 0, 3, 0, 4, 4, 4, 3, 3, 3, 0, 4, 2, 0, 3, 4),
+    (1, 4, 0, 4, 0, 3, 0, 1, 0, 3, 3, 3, 1, 1, 3, 3, 2, 2, 3, 3, 1, 0, 3, 2, 2, 1, 2, 0, 3, 1, 2, 1, 2, 0, 3, 2, 0, 2, 2, 3, 3, 4, 3, 0, 3, 3, 1, 2, 0, 1, 1, 3, 1, 2, 0, 0, 3, 0, 1, 1, 0, 3, 2, 2, 3, 3, 0, 3, 0, 0, 0, 2, 3, 3, 4, 3, 3, 0, 1, 0, 0, 1, 4),
+    (0, 4, 0, 4, 0, 4, 0, 0, 0, 3, 4, 4, 3, 1, 4, 2, 3, 2, 3, 3, 3, 1, 4, 3, 4, 0, 3, 0, 4, 2, 3, 3, 2, 2, 5, 4, 2, 1, 3, 4, 3, 4, 3, 1, 3, 3, 4, 2, 0, 2, 1, 0, 3, 3, 0, 0, 2, 0, 3, 1, 0, 4, 4, 3, 4, 3, 0, 4, 0, 1, 0, 2, 4, 4, 4, 4, 4, 0, 3, 2, 0, 3, 3),
+    (0, 0, 0, 1, 0, 4, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 3, 2, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 2),
+    (0, 2, 0, 3, 0, 4, 0, 4, 0, 1, 3, 3, 3, 0, 4, 0, 2, 1, 2, 1, 1, 1, 2, 0, 3, 1, 1, 0, 1, 0, 3, 1, 0, 0, 3, 3, 2, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 2, 0, 2, 2, 0, 3, 1, 0, 0, 1, 0, 1, 1, 0, 1, 2, 0, 3, 0, 0, 0, 0, 1, 0, 0, 3, 3, 4, 3, 1, 0, 1, 0, 3, 0, 2),
+    (0, 0, 0, 3, 0, 5, 0, 0, 0, 0, 1, 0, 2, 0, 3, 1, 0, 1, 3, 0, 0, 0, 2, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 4, 0, 0, 0, 2, 3, 0, 1, 4, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 3, 0, 0, 0, 0, 0, 3),
+    (0, 2, 0, 5, 0, 5, 0, 1, 0, 2, 4, 3, 3, 2, 5, 1, 3, 2, 3, 3, 3, 0, 4, 1, 2, 0, 3, 0, 4, 0, 2, 2, 1, 1, 5, 3, 0, 0, 1, 4, 2, 3, 2, 0, 3, 3, 3, 2, 0, 2, 4, 1, 1, 2, 0, 1, 1, 0, 3, 1, 0, 1, 3, 1, 2, 3, 0, 2, 0, 0, 0, 1, 3, 5, 4, 4, 4, 0, 3, 0, 0, 1, 3),
+    (0, 4, 0, 5, 0, 4, 0, 4, 0, 4, 5, 4, 3, 3, 4, 3, 3, 3, 4, 3, 4, 4, 5, 3, 4, 5, 4, 2, 4, 2, 3, 4, 3, 1, 4, 4, 1, 3, 5, 4, 4, 5, 5, 4, 4, 5, 5, 5, 2, 3, 3, 1, 4, 3, 1, 3, 3, 0, 3, 3, 1, 4, 3, 4, 4, 4, 0, 3, 0, 4, 0, 3, 3, 4, 4, 5, 0, 0, 4, 3, 0, 4, 5),
+    (0, 4, 0, 4, 0, 3, 0, 3, 0, 3, 4, 4, 4, 3, 3, 2, 4, 3, 4, 3, 4, 3, 5, 3, 4, 3, 2, 1, 4, 2, 4, 4, 3, 1, 3, 4, 2, 4, 5, 5, 3, 4, 5, 4, 1, 5, 4, 3, 0, 3, 2, 2, 3, 2, 1, 3, 1, 0, 3, 3, 3, 5, 3, 3, 3, 5, 4, 4, 2, 3, 3, 4, 3, 3, 3, 2, 1, 0, 3, 2, 1, 4, 3),
+    (0, 4, 0, 5, 0, 4, 0, 3, 0, 3, 5, 5, 3, 2, 4, 3, 4, 0, 5, 4, 4, 1, 4, 4, 4, 3, 3, 3, 4, 3, 5, 5, 2, 3, 3, 4, 1, 2, 5, 5, 3, 5, 5, 2, 3, 5, 5, 4, 0, 3, 2, 0, 3, 3, 1, 1, 5, 1, 4, 1, 0, 4, 3, 2, 3, 5, 0, 4, 0, 3, 0, 5, 4, 3, 4, 3, 0, 0, 4, 1, 0, 4, 4),
+    (1, 3, 0, 4, 0, 2, 0, 2, 0, 2, 5, 5, 3, 3, 3, 3, 3, 0, 4, 2, 3, 4, 4, 4, 3, 4, 0, 0, 3, 4, 5, 4, 3, 3, 3, 3, 2, 5, 5, 4, 5, 5, 5, 4, 3, 5, 5, 5, 1, 3, 1, 0, 1, 0, 0, 3, 2, 0, 4, 2, 0, 5, 2, 3, 2, 4, 1, 3, 0, 3, 0, 4, 5, 4, 5, 4, 3, 0, 4, 2, 0, 5, 4),
+    (0, 3, 0, 4, 0, 5, 0, 3, 0, 3, 4, 4, 3, 2, 3, 2, 3, 3, 3, 3, 3, 2, 4, 3, 3, 2, 2, 0, 3, 3, 3, 3, 3, 1, 3, 3, 3, 0, 4, 4, 3, 4, 4, 1, 1, 4, 4, 2, 0, 3, 1, 0, 1, 1, 0, 4, 1, 0, 2, 3, 1, 3, 3, 1, 3, 4, 0, 3, 0, 1, 0, 3, 1, 3, 0, 0, 1, 0, 2, 0, 0, 4, 4),
+    (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0),
+    (0, 3, 0, 3, 0, 2, 0, 3, 0, 1, 5, 4, 3, 3, 3, 1, 4, 2, 1, 2, 3, 4, 4, 2, 4, 4, 5, 0, 3, 1, 4, 3, 4, 0, 4, 3, 3, 3, 2, 3, 2, 5, 3, 4, 3, 2, 2, 3, 0, 0, 3, 0, 2, 1, 0, 1, 2, 0, 0, 0, 0, 2, 1, 1, 3, 1, 0, 2, 0, 4, 0, 3, 4, 4, 4, 5, 2, 0, 2, 0, 0, 1, 3),
+    (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 4, 2, 1, 1, 0, 1, 0, 3, 2, 0, 0, 3, 1, 1, 1, 2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 1, 0, 0, 0, 2, 0, 0, 0, 1, 4, 0, 4, 2, 1, 0, 0, 0, 0, 0, 1),
+    (0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 3, 1, 0, 0, 0, 2, 0, 2, 1, 0, 0, 1, 2, 1, 0, 1, 1, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 3, 1, 0, 0, 0, 0, 0, 1, 0, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 2),
+    (0, 4, 0, 4, 0, 4, 0, 3, 0, 4, 4, 3, 4, 2, 4, 3, 2, 0, 4, 4, 4, 3, 5, 3, 5, 3, 3, 2, 4, 2, 4, 3, 4, 3, 1, 4, 0, 2, 3, 4, 4, 4, 3, 3, 3, 4, 4, 4, 3, 4, 1, 3, 4, 3, 2, 1, 2, 1, 3, 3, 3, 4, 4, 3, 3, 5, 0, 4, 0, 3, 0, 4, 3, 3, 3, 2, 1, 0, 3, 0, 0, 3, 3),
+    (0, 4, 0, 3, 0, 3, 0, 3, 0, 3, 5, 5, 3, 3, 3, 3, 4, 3, 4, 3, 3, 3, 4, 4, 4, 3, 3, 3, 3, 4, 3, 5, 3, 3, 1, 3, 2, 4, 5, 5, 5, 5, 4, 3, 4, 5, 5, 3, 2, 2, 3, 3, 3, 3, 2, 3, 3, 1, 2, 3, 2, 4, 3, 3, 3, 4, 0, 4, 0, 2, 0, 4, 3, 2, 2, 1, 2, 0, 3, 0, 0, 4, 1),
 )
+# fmt: on
 
-class JapaneseContextAnalysis(object):
+
+class JapaneseContextAnalysis:
     NUM_OF_CATEGORY = 6
     DONT_KNOW = -1
     ENOUGH_REL_THRESHOLD = 100
     MAX_REL_THRESHOLD = 1000
     MINIMUM_DATA_THRESHOLD = 4
 
-    def __init__(self):
-        self._total_rel = None
-        self._rel_sample = None
-        self._need_to_skip_char_num = None
-        self._last_char_order = None
-        self._done = None
+    def __init__(self) -> None:
+        self._total_rel = 0
+        self._rel_sample: Listint = 
+        self._need_to_skip_char_num = 0
+        self._last_char_order = -1
+        self._done = False

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/langbulgarianmodel.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/langbulgarianmodel.py Changed

@@ -1,9 +1,5 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-
 from chardet.sbcharsetprober import SingleByteCharSetModel
 
-
 # 3: Positive
 # 2: Likely
 # 1: Unlikely
@@ -4115,536 +4111,539 @@
 
 # Character Mapping Table(s):
 ISO_8859_5_BULGARIAN_CHAR_TO_ORDER = {
-     0: 255,  # '\x00'
-     1: 255,  # '\x01'
-     2: 255,  # '\x02'
-     3: 255,  # '\x03'
-     4: 255,  # '\x04'
-     5: 255,  # '\x05'
-     6: 255,  # '\x06'
-     7: 255,  # '\x07'
-     8: 255,  # '\x08'
-     9: 255,  # '\t'
-     10: 254,  # '\n'
-     11: 255,  # '\x0b'
-     12: 255,  # '\x0c'
-     13: 254,  # '\r'
-     14: 255,  # '\x0e'
-     15: 255,  # '\x0f'
-     16: 255,  # '\x10'
-     17: 255,  # '\x11'
-     18: 255,  # '\x12'
-     19: 255,  # '\x13'
-     20: 255,  # '\x14'
-     21: 255,  # '\x15'
-     22: 255,  # '\x16'
-     23: 255,  # '\x17'
-     24: 255,  # '\x18'
-     25: 255,  # '\x19'
-     26: 255,  # '\x1a'
-     27: 255,  # '\x1b'
-     28: 255,  # '\x1c'
-     29: 255,  # '\x1d'
-     30: 255,  # '\x1e'
-     31: 255,  # '\x1f'
-     32: 253,  # ' '
-     33: 253,  # '!'
-     34: 253,  # '"'
-     35: 253,  # '#'
-     36: 253,  # '$'
-     37: 253,  # '%'
-     38: 253,  # '&'
-     39: 253,  # "'"
-     40: 253,  # '('
-     41: 253,  # ')'
-     42: 253,  # '*'
-     43: 253,  # '+'
-     44: 253,  # ','
-     45: 253,  # '-'
-     46: 253,  # '.'
-     47: 253,  # '/'
-     48: 252,  # '0'
-     49: 252,  # '1'
-     50: 252,  # '2'
-     51: 252,  # '3'
-     52: 252,  # '4'
-     53: 252,  # '5'
-     54: 252,  # '6'
-     55: 252,  # '7'
-     56: 252,  # '8'
-     57: 252,  # '9'
-     58: 253,  # ':'
-     59: 253,  # ';'
-     60: 253,  # '<'
-     61: 253,  # '='
-     62: 253,  # '>'
-     63: 253,  # '?'
-     64: 253,  # '@'
-     65: 77,  # 'A'
-     66: 90,  # 'B'
-     67: 99,  # 'C'
-     68: 100,  # 'D'
-     69: 72,  # 'E'
-     70: 109,  # 'F'
-     71: 107,  # 'G'
-     72: 101,  # 'H'
-     73: 79,  # 'I'
-     74: 185,  # 'J'
-     75: 81,  # 'K'
-     76: 102,  # 'L'
-     77: 76,  # 'M'
-     78: 94,  # 'N'
-     79: 82,  # 'O'
-     80: 110,  # 'P'
-     81: 186,  # 'Q'
-     82: 108,  # 'R'
-     83: 91,  # 'S'
-     84: 74,  # 'T'
-     85: 119,  # 'U'
-     86: 84,  # 'V'
-     87: 96,  # 'W'
-     88: 111,  # 'X'
-     89: 187,  # 'Y'
-     90: 115,  # 'Z'
-     91: 253,  # ''
-     92: 253,  # '\\'
-     93: 253,  # ''
-     94: 253,  # '^'
-     95: 253,  # '_'
-     96: 253,  # '`'
-     97: 65,  # 'a'
-     98: 69,  # 'b'
-     99: 70,  # 'c'
-     100: 66,  # 'd'
-     101: 63,  # 'e'
-     102: 68,  # 'f'
-     103: 112,  # 'g'
-     104: 103,  # 'h'
-     105: 92,  # 'i'
-     106: 194,  # 'j'
-     107: 104,  # 'k'
-     108: 95,  # 'l'
-     109: 86,  # 'm'
-     110: 87,  # 'n'
-     111: 71,  # 'o'
-     112: 116,  # 'p'
-     113: 195,  # 'q'
-     114: 85,  # 'r'
-     115: 93,  # 's'
-     116: 97,  # 't'
-     117: 113,  # 'u'
-     118: 196,  # 'v'
-     119: 197,  # 'w'
-     120: 198,  # 'x'
-     121: 199,  # 'y'
-     122: 200,  # 'z'
-     123: 253,  # '{'
-     124: 253,  # '|'
-     125: 253,  # '}'
-     126: 253,  # '~'
-     127: 253,  # '\x7f'
-     128: 194,  # '\x80'
-     129: 195,  # '\x81'
-     130: 196,  # '\x82'
-     131: 197,  # '\x83'
-     132: 198,  # '\x84'
-     133: 199,  # '\x85'
-     134: 200,  # '\x86'
-     135: 201,  # '\x87'
-     136: 202,  # '\x88'
-     137: 203,  # '\x89'
-     138: 204,  # '\x8a'
-     139: 205,  # '\x8b'
-     140: 206,  # '\x8c'
-     141: 207,  # '\x8d'
-     142: 208,  # '\x8e'
-     143: 209,  # '\x8f'
-     144: 210,  # '\x90'
-     145: 211,  # '\x91'
-     146: 212,  # '\x92'
-     147: 213,  # '\x93'
-     148: 214,  # '\x94'
-     149: 215,  # '\x95'
-     150: 216,  # '\x96'
-     151: 217,  # '\x97'
-     152: 218,  # '\x98'
-     153: 219,  # '\x99'
-     154: 220,  # '\x9a'
-     155: 221,  # '\x9b'
-     156: 222,  # '\x9c'
-     157: 223,  # '\x9d'
-     158: 224,  # '\x9e'
-     159: 225,  # '\x9f'
-     160: 81,  # '\xa0'
-     161: 226,  # 'Ё'
-     162: 227,  # 'Ђ'
-     163: 228,  # 'Ѓ'
-     164: 229,  # 'Є'
-     165: 230,  # 'Ѕ'
-     166: 105,  # 'І'
-     167: 231,  # 'Ї'
-     168: 232,  # 'Ј'
-     169: 233,  # 'Љ'
-     170: 234,  # 'Њ'
-     171: 235,  # 'Ћ'
-     172: 236,  # 'Ќ'
-     173: 45,  # '\xad'
-     174: 237,  # 'Ў'
-     175: 238,  # 'Џ'
-     176: 31,  # 'А'
-     177: 32,  # 'Б'
-     178: 35,  # 'В'
-     179: 43,  # 'Г'
-     180: 37,  # 'Д'
-     181: 44,  # 'Е'
-     182: 55,  # 'Ж'
-     183: 47,  # 'З'
-     184: 40,  # 'И'
-     185: 59,  # 'Й'

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/langgreekmodel.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/langgreekmodel.py Changed

@@ -1,9 +1,5 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-
 from chardet.sbcharsetprober import SingleByteCharSetModel
 
-
 # 3: Positive
 # 2: Likely
 # 1: Unlikely
@@ -3863,536 +3859,539 @@
 
 # Character Mapping Table(s):
 WINDOWS_1253_GREEK_CHAR_TO_ORDER = {
-     0: 255,  # '\x00'
-     1: 255,  # '\x01'
-     2: 255,  # '\x02'
-     3: 255,  # '\x03'
-     4: 255,  # '\x04'
-     5: 255,  # '\x05'
-     6: 255,  # '\x06'
-     7: 255,  # '\x07'
-     8: 255,  # '\x08'
-     9: 255,  # '\t'
-     10: 254,  # '\n'
-     11: 255,  # '\x0b'
-     12: 255,  # '\x0c'
-     13: 254,  # '\r'
-     14: 255,  # '\x0e'
-     15: 255,  # '\x0f'
-     16: 255,  # '\x10'
-     17: 255,  # '\x11'
-     18: 255,  # '\x12'
-     19: 255,  # '\x13'
-     20: 255,  # '\x14'
-     21: 255,  # '\x15'
-     22: 255,  # '\x16'
-     23: 255,  # '\x17'
-     24: 255,  # '\x18'
-     25: 255,  # '\x19'
-     26: 255,  # '\x1a'
-     27: 255,  # '\x1b'
-     28: 255,  # '\x1c'
-     29: 255,  # '\x1d'
-     30: 255,  # '\x1e'
-     31: 255,  # '\x1f'
-     32: 253,  # ' '
-     33: 253,  # '!'
-     34: 253,  # '"'
-     35: 253,  # '#'
-     36: 253,  # '$'
-     37: 253,  # '%'
-     38: 253,  # '&'
-     39: 253,  # "'"
-     40: 253,  # '('
-     41: 253,  # ')'
-     42: 253,  # '*'
-     43: 253,  # '+'
-     44: 253,  # ','
-     45: 253,  # '-'
-     46: 253,  # '.'
-     47: 253,  # '/'
-     48: 252,  # '0'
-     49: 252,  # '1'
-     50: 252,  # '2'
-     51: 252,  # '3'
-     52: 252,  # '4'
-     53: 252,  # '5'
-     54: 252,  # '6'
-     55: 252,  # '7'
-     56: 252,  # '8'
-     57: 252,  # '9'
-     58: 253,  # ':'
-     59: 253,  # ';'
-     60: 253,  # '<'
-     61: 253,  # '='
-     62: 253,  # '>'
-     63: 253,  # '?'
-     64: 253,  # '@'
-     65: 82,  # 'A'
-     66: 100,  # 'B'
-     67: 104,  # 'C'
-     68: 94,  # 'D'
-     69: 98,  # 'E'
-     70: 101,  # 'F'
-     71: 116,  # 'G'
-     72: 102,  # 'H'
-     73: 111,  # 'I'
-     74: 187,  # 'J'
-     75: 117,  # 'K'
-     76: 92,  # 'L'
-     77: 88,  # 'M'
-     78: 113,  # 'N'
-     79: 85,  # 'O'
-     80: 79,  # 'P'
-     81: 118,  # 'Q'
-     82: 105,  # 'R'
-     83: 83,  # 'S'
-     84: 67,  # 'T'
-     85: 114,  # 'U'
-     86: 119,  # 'V'
-     87: 95,  # 'W'
-     88: 99,  # 'X'
-     89: 109,  # 'Y'
-     90: 188,  # 'Z'
-     91: 253,  # ''
-     92: 253,  # '\\'
-     93: 253,  # ''
-     94: 253,  # '^'
-     95: 253,  # '_'
-     96: 253,  # '`'
-     97: 72,  # 'a'
-     98: 70,  # 'b'
-     99: 80,  # 'c'
-     100: 81,  # 'd'
-     101: 60,  # 'e'
-     102: 96,  # 'f'
-     103: 93,  # 'g'
-     104: 89,  # 'h'
-     105: 68,  # 'i'
-     106: 120,  # 'j'
-     107: 97,  # 'k'
-     108: 77,  # 'l'
-     109: 86,  # 'm'
-     110: 69,  # 'n'
-     111: 55,  # 'o'
-     112: 78,  # 'p'
-     113: 115,  # 'q'
-     114: 65,  # 'r'
-     115: 66,  # 's'
-     116: 58,  # 't'
-     117: 76,  # 'u'
-     118: 106,  # 'v'
-     119: 103,  # 'w'
-     120: 87,  # 'x'
-     121: 107,  # 'y'
-     122: 112,  # 'z'
-     123: 253,  # '{'
-     124: 253,  # '|'
-     125: 253,  # '}'
-     126: 253,  # '~'
-     127: 253,  # '\x7f'
-     128: 255,  # '€'
-     129: 255,  # None
-     130: 255,  # '‚'
-     131: 255,  # 'ƒ'
-     132: 255,  # '„'
-     133: 255,  # '…'
-     134: 255,  # '†'
-     135: 255,  # '‡'
-     136: 255,  # None
-     137: 255,  # '‰'
-     138: 255,  # None
-     139: 255,  # '‹'
-     140: 255,  # None
-     141: 255,  # None
-     142: 255,  # None
-     143: 255,  # None
-     144: 255,  # None
-     145: 255,  # '‘'
-     146: 255,  # '’'
-     147: 255,  # '“'
-     148: 255,  # '”'
-     149: 255,  # '•'
-     150: 255,  # '–'
-     151: 255,  # '—'
-     152: 255,  # None
-     153: 255,  # '™'
-     154: 255,  # None
-     155: 255,  # '›'
-     156: 255,  # None
-     157: 255,  # None
-     158: 255,  # None
-     159: 255,  # None
-     160: 253,  # '\xa0'
-     161: 233,  # '΅'
-     162: 61,  # 'Ά'
-     163: 253,  # '£'
-     164: 253,  # '¤'
-     165: 253,  # '¥'
-     166: 253,  # '¦'
-     167: 253,  # '§'
-     168: 253,  # '¨'
-     169: 253,  # '©'
-     170: 253,  # None
-     171: 253,  # '«'
-     172: 253,  # '¬'
-     173: 74,  # '\xad'
-     174: 253,  # '®'
-     175: 253,  # '―'
-     176: 253,  # '°'
-     177: 253,  # '±'
-     178: 253,  # '²'
-     179: 253,  # '³'
-     180: 247,  # '΄'
-     181: 253,  # 'µ'
-     182: 253,  # '¶'
-     183: 36,  # '·'
-     184: 46,  # 'Έ'
-     185: 71,  # 'Ή'

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/langhebrewmodel.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/langhebrewmodel.py Changed

@@ -1,9 +1,5 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-
 from chardet.sbcharsetprober import SingleByteCharSetModel
 
-
 # 3: Positive
 # 2: Likely
 # 1: Unlikely
@@ -4115,269 +4111,270 @@
 
 # Character Mapping Table(s):
 WINDOWS_1255_HEBREW_CHAR_TO_ORDER = {
-     0: 255,  # '\x00'
-     1: 255,  # '\x01'
-     2: 255,  # '\x02'
-     3: 255,  # '\x03'
-     4: 255,  # '\x04'
-     5: 255,  # '\x05'
-     6: 255,  # '\x06'
-     7: 255,  # '\x07'
-     8: 255,  # '\x08'
-     9: 255,  # '\t'
-     10: 254,  # '\n'
-     11: 255,  # '\x0b'
-     12: 255,  # '\x0c'
-     13: 254,  # '\r'
-     14: 255,  # '\x0e'
-     15: 255,  # '\x0f'
-     16: 255,  # '\x10'
-     17: 255,  # '\x11'
-     18: 255,  # '\x12'
-     19: 255,  # '\x13'
-     20: 255,  # '\x14'
-     21: 255,  # '\x15'
-     22: 255,  # '\x16'
-     23: 255,  # '\x17'
-     24: 255,  # '\x18'
-     25: 255,  # '\x19'
-     26: 255,  # '\x1a'
-     27: 255,  # '\x1b'
-     28: 255,  # '\x1c'
-     29: 255,  # '\x1d'
-     30: 255,  # '\x1e'
-     31: 255,  # '\x1f'
-     32: 253,  # ' '
-     33: 253,  # '!'
-     34: 253,  # '"'
-     35: 253,  # '#'
-     36: 253,  # '$'
-     37: 253,  # '%'
-     38: 253,  # '&'
-     39: 253,  # "'"
-     40: 253,  # '('
-     41: 253,  # ')'
-     42: 253,  # '*'
-     43: 253,  # '+'
-     44: 253,  # ','
-     45: 253,  # '-'
-     46: 253,  # '.'
-     47: 253,  # '/'
-     48: 252,  # '0'
-     49: 252,  # '1'
-     50: 252,  # '2'
-     51: 252,  # '3'
-     52: 252,  # '4'
-     53: 252,  # '5'
-     54: 252,  # '6'
-     55: 252,  # '7'
-     56: 252,  # '8'
-     57: 252,  # '9'
-     58: 253,  # ':'
-     59: 253,  # ';'
-     60: 253,  # '<'
-     61: 253,  # '='
-     62: 253,  # '>'
-     63: 253,  # '?'
-     64: 253,  # '@'
-     65: 69,  # 'A'
-     66: 91,  # 'B'
-     67: 79,  # 'C'
-     68: 80,  # 'D'
-     69: 92,  # 'E'
-     70: 89,  # 'F'
-     71: 97,  # 'G'
-     72: 90,  # 'H'
-     73: 68,  # 'I'
-     74: 111,  # 'J'
-     75: 112,  # 'K'
-     76: 82,  # 'L'
-     77: 73,  # 'M'
-     78: 95,  # 'N'
-     79: 85,  # 'O'
-     80: 78,  # 'P'
-     81: 121,  # 'Q'
-     82: 86,  # 'R'
-     83: 71,  # 'S'
-     84: 67,  # 'T'
-     85: 102,  # 'U'
-     86: 107,  # 'V'
-     87: 84,  # 'W'
-     88: 114,  # 'X'
-     89: 103,  # 'Y'
-     90: 115,  # 'Z'
-     91: 253,  # ''
-     92: 253,  # '\\'
-     93: 253,  # ''
-     94: 253,  # '^'
-     95: 253,  # '_'
-     96: 253,  # '`'
-     97: 50,  # 'a'
-     98: 74,  # 'b'
-     99: 60,  # 'c'
-     100: 61,  # 'd'
-     101: 42,  # 'e'
-     102: 76,  # 'f'
-     103: 70,  # 'g'
-     104: 64,  # 'h'
-     105: 53,  # 'i'
-     106: 105,  # 'j'
-     107: 93,  # 'k'
-     108: 56,  # 'l'
-     109: 65,  # 'm'
-     110: 54,  # 'n'
-     111: 49,  # 'o'
-     112: 66,  # 'p'
-     113: 110,  # 'q'
-     114: 51,  # 'r'
-     115: 43,  # 's'
-     116: 44,  # 't'
-     117: 63,  # 'u'
-     118: 81,  # 'v'
-     119: 77,  # 'w'
-     120: 98,  # 'x'
-     121: 75,  # 'y'
-     122: 108,  # 'z'
-     123: 253,  # '{'
-     124: 253,  # '|'
-     125: 253,  # '}'
-     126: 253,  # '~'
-     127: 253,  # '\x7f'
-     128: 124,  # '€'
-     129: 202,  # None
-     130: 203,  # '‚'
-     131: 204,  # 'ƒ'
-     132: 205,  # '„'
-     133: 40,  # '…'
-     134: 58,  # '†'
-     135: 206,  # '‡'
-     136: 207,  # 'ˆ'
-     137: 208,  # '‰'
-     138: 209,  # None
-     139: 210,  # '‹'
-     140: 211,  # None
-     141: 212,  # None
-     142: 213,  # None
-     143: 214,  # None
-     144: 215,  # None
-     145: 83,  # '‘'
-     146: 52,  # '’'
-     147: 47,  # '“'
-     148: 46,  # '”'
-     149: 72,  # '•'
-     150: 32,  # '–'
-     151: 94,  # '—'
-     152: 216,  # '˜'
-     153: 113,  # '™'
-     154: 217,  # None
-     155: 109,  # '›'
-     156: 218,  # None
-     157: 219,  # None
-     158: 220,  # None
-     159: 221,  # None
-     160: 34,  # '\xa0'
-     161: 116,  # '¡'
-     162: 222,  # '¢'
-     163: 118,  # '£'
-     164: 100,  # '₪'
-     165: 223,  # '¥'
-     166: 224,  # '¦'
-     167: 117,  # '§'
-     168: 119,  # '¨'
-     169: 104,  # '©'
-     170: 125,  # '×'
-     171: 225,  # '«'
-     172: 226,  # '¬'
-     173: 87,  # '\xad'
-     174: 99,  # '®'
-     175: 227,  # '¯'
-     176: 106,  # '°'
-     177: 122,  # '±'
-     178: 123,  # '²'
-     179: 228,  # '³'
-     180: 55,  # '´'
-     181: 229,  # 'µ'
-     182: 230,  # '¶'
-     183: 101,  # '·'
-     184: 231,  # '¸'
-     185: 232,  # '¹'

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/langhungarianmodel.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/langhungarianmodel.py Changed

@@ -1,9 +1,5 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-
 from chardet.sbcharsetprober import SingleByteCharSetModel
 
-
 # 3: Positive
 # 2: Likely
 # 1: Unlikely
@@ -4115,536 +4111,539 @@
 
 # Character Mapping Table(s):
 WINDOWS_1250_HUNGARIAN_CHAR_TO_ORDER = {
-     0: 255,  # '\x00'
-     1: 255,  # '\x01'
-     2: 255,  # '\x02'
-     3: 255,  # '\x03'
-     4: 255,  # '\x04'
-     5: 255,  # '\x05'
-     6: 255,  # '\x06'
-     7: 255,  # '\x07'
-     8: 255,  # '\x08'
-     9: 255,  # '\t'
-     10: 254,  # '\n'
-     11: 255,  # '\x0b'
-     12: 255,  # '\x0c'
-     13: 254,  # '\r'
-     14: 255,  # '\x0e'
-     15: 255,  # '\x0f'
-     16: 255,  # '\x10'
-     17: 255,  # '\x11'
-     18: 255,  # '\x12'
-     19: 255,  # '\x13'
-     20: 255,  # '\x14'
-     21: 255,  # '\x15'
-     22: 255,  # '\x16'
-     23: 255,  # '\x17'
-     24: 255,  # '\x18'
-     25: 255,  # '\x19'
-     26: 255,  # '\x1a'
-     27: 255,  # '\x1b'
-     28: 255,  # '\x1c'
-     29: 255,  # '\x1d'
-     30: 255,  # '\x1e'
-     31: 255,  # '\x1f'
-     32: 253,  # ' '
-     33: 253,  # '!'
-     34: 253,  # '"'
-     35: 253,  # '#'
-     36: 253,  # '$'
-     37: 253,  # '%'
-     38: 253,  # '&'
-     39: 253,  # "'"
-     40: 253,  # '('
-     41: 253,  # ')'
-     42: 253,  # '*'
-     43: 253,  # '+'
-     44: 253,  # ','
-     45: 253,  # '-'
-     46: 253,  # '.'
-     47: 253,  # '/'
-     48: 252,  # '0'
-     49: 252,  # '1'
-     50: 252,  # '2'
-     51: 252,  # '3'
-     52: 252,  # '4'
-     53: 252,  # '5'
-     54: 252,  # '6'
-     55: 252,  # '7'
-     56: 252,  # '8'
-     57: 252,  # '9'
-     58: 253,  # ':'
-     59: 253,  # ';'
-     60: 253,  # '<'
-     61: 253,  # '='
-     62: 253,  # '>'
-     63: 253,  # '?'
-     64: 253,  # '@'
-     65: 28,  # 'A'
-     66: 40,  # 'B'
-     67: 54,  # 'C'
-     68: 45,  # 'D'
-     69: 32,  # 'E'
-     70: 50,  # 'F'
-     71: 49,  # 'G'
-     72: 38,  # 'H'
-     73: 39,  # 'I'
-     74: 53,  # 'J'
-     75: 36,  # 'K'
-     76: 41,  # 'L'
-     77: 34,  # 'M'
-     78: 35,  # 'N'
-     79: 47,  # 'O'
-     80: 46,  # 'P'
-     81: 72,  # 'Q'
-     82: 43,  # 'R'
-     83: 33,  # 'S'
-     84: 37,  # 'T'
-     85: 57,  # 'U'
-     86: 48,  # 'V'
-     87: 64,  # 'W'
-     88: 68,  # 'X'
-     89: 55,  # 'Y'
-     90: 52,  # 'Z'
-     91: 253,  # ''
-     92: 253,  # '\\'
-     93: 253,  # ''
-     94: 253,  # '^'
-     95: 253,  # '_'
-     96: 253,  # '`'
-     97: 2,  # 'a'
-     98: 18,  # 'b'
-     99: 26,  # 'c'
-     100: 17,  # 'd'
-     101: 1,  # 'e'
-     102: 27,  # 'f'
-     103: 12,  # 'g'
-     104: 20,  # 'h'
-     105: 9,  # 'i'
-     106: 22,  # 'j'
-     107: 7,  # 'k'
-     108: 6,  # 'l'
-     109: 13,  # 'm'
-     110: 4,  # 'n'
-     111: 8,  # 'o'
-     112: 23,  # 'p'
-     113: 67,  # 'q'
-     114: 10,  # 'r'
-     115: 5,  # 's'
-     116: 3,  # 't'
-     117: 21,  # 'u'
-     118: 19,  # 'v'
-     119: 65,  # 'w'
-     120: 62,  # 'x'
-     121: 16,  # 'y'
-     122: 11,  # 'z'
-     123: 253,  # '{'
-     124: 253,  # '|'
-     125: 253,  # '}'
-     126: 253,  # '~'
-     127: 253,  # '\x7f'
-     128: 161,  # '€'
-     129: 162,  # None
-     130: 163,  # '‚'
-     131: 164,  # None
-     132: 165,  # '„'
-     133: 166,  # '…'
-     134: 167,  # '†'
-     135: 168,  # '‡'
-     136: 169,  # None
-     137: 170,  # '‰'
-     138: 171,  # 'Š'
-     139: 172,  # '‹'
-     140: 173,  # 'Ś'
-     141: 174,  # 'Ť'
-     142: 175,  # 'Ž'
-     143: 176,  # 'Ź'
-     144: 177,  # None
-     145: 178,  # '‘'
-     146: 179,  # '’'
-     147: 180,  # '“'
-     148: 78,  # '”'
-     149: 181,  # '•'
-     150: 69,  # '–'
-     151: 182,  # '—'
-     152: 183,  # None
-     153: 184,  # '™'
-     154: 185,  # 'š'
-     155: 186,  # '›'
-     156: 187,  # 'ś'
-     157: 188,  # 'ť'
-     158: 189,  # 'ž'
-     159: 190,  # 'ź'
-     160: 191,  # '\xa0'
-     161: 192,  # 'ˇ'
-     162: 193,  # '˘'
-     163: 194,  # 'Ł'
-     164: 195,  # '¤'
-     165: 196,  # 'Ą'
-     166: 197,  # '¦'
-     167: 76,  # '§'
-     168: 198,  # '¨'
-     169: 199,  # '©'
-     170: 200,  # 'Ş'
-     171: 201,  # '«'
-     172: 202,  # '¬'
-     173: 203,  # '\xad'
-     174: 204,  # '®'
-     175: 205,  # 'Ż'
-     176: 81,  # '°'
-     177: 206,  # '±'
-     178: 207,  # '˛'
-     179: 208,  # 'ł'
-     180: 209,  # '´'
-     181: 210,  # 'µ'
-     182: 211,  # '¶'
-     183: 212,  # '·'
-     184: 213,  # '¸'
-     185: 214,  # 'ą'

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/langrussianmodel.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/langrussianmodel.py Changed

@@ -1,9 +1,5 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-
 from chardet.sbcharsetprober import SingleByteCharSetModel
 
-
 # 3: Positive
 # 2: Likely
 # 1: Unlikely
@@ -4115,1604 +4111,1615 @@
 
 # Character Mapping Table(s):
 IBM866_RUSSIAN_CHAR_TO_ORDER = {
-     0: 255,  # '\x00'
-     1: 255,  # '\x01'
-     2: 255,  # '\x02'
-     3: 255,  # '\x03'
-     4: 255,  # '\x04'
-     5: 255,  # '\x05'
-     6: 255,  # '\x06'
-     7: 255,  # '\x07'
-     8: 255,  # '\x08'
-     9: 255,  # '\t'
-     10: 254,  # '\n'
-     11: 255,  # '\x0b'
-     12: 255,  # '\x0c'
-     13: 254,  # '\r'
-     14: 255,  # '\x0e'
-     15: 255,  # '\x0f'
-     16: 255,  # '\x10'
-     17: 255,  # '\x11'
-     18: 255,  # '\x12'
-     19: 255,  # '\x13'
-     20: 255,  # '\x14'
-     21: 255,  # '\x15'
-     22: 255,  # '\x16'
-     23: 255,  # '\x17'
-     24: 255,  # '\x18'
-     25: 255,  # '\x19'
-     26: 255,  # '\x1a'
-     27: 255,  # '\x1b'
-     28: 255,  # '\x1c'
-     29: 255,  # '\x1d'
-     30: 255,  # '\x1e'
-     31: 255,  # '\x1f'
-     32: 253,  # ' '
-     33: 253,  # '!'
-     34: 253,  # '"'
-     35: 253,  # '#'
-     36: 253,  # '$'
-     37: 253,  # '%'
-     38: 253,  # '&'
-     39: 253,  # "'"
-     40: 253,  # '('
-     41: 253,  # ')'
-     42: 253,  # '*'
-     43: 253,  # '+'
-     44: 253,  # ','
-     45: 253,  # '-'
-     46: 253,  # '.'
-     47: 253,  # '/'
-     48: 252,  # '0'
-     49: 252,  # '1'
-     50: 252,  # '2'
-     51: 252,  # '3'
-     52: 252,  # '4'
-     53: 252,  # '5'
-     54: 252,  # '6'
-     55: 252,  # '7'
-     56: 252,  # '8'
-     57: 252,  # '9'
-     58: 253,  # ':'
-     59: 253,  # ';'
-     60: 253,  # '<'
-     61: 253,  # '='
-     62: 253,  # '>'
-     63: 253,  # '?'
-     64: 253,  # '@'
-     65: 142,  # 'A'
-     66: 143,  # 'B'
-     67: 144,  # 'C'
-     68: 145,  # 'D'
-     69: 146,  # 'E'
-     70: 147,  # 'F'
-     71: 148,  # 'G'
-     72: 149,  # 'H'
-     73: 150,  # 'I'
-     74: 151,  # 'J'
-     75: 152,  # 'K'
-     76: 74,  # 'L'
-     77: 153,  # 'M'
-     78: 75,  # 'N'
-     79: 154,  # 'O'
-     80: 155,  # 'P'
-     81: 156,  # 'Q'
-     82: 157,  # 'R'
-     83: 158,  # 'S'
-     84: 159,  # 'T'
-     85: 160,  # 'U'
-     86: 161,  # 'V'
-     87: 162,  # 'W'
-     88: 163,  # 'X'
-     89: 164,  # 'Y'
-     90: 165,  # 'Z'
-     91: 253,  # ''
-     92: 253,  # '\\'
-     93: 253,  # ''
-     94: 253,  # '^'
-     95: 253,  # '_'
-     96: 253,  # '`'
-     97: 71,  # 'a'
-     98: 172,  # 'b'
-     99: 66,  # 'c'
-     100: 173,  # 'd'
-     101: 65,  # 'e'
-     102: 174,  # 'f'
-     103: 76,  # 'g'
-     104: 175,  # 'h'
-     105: 64,  # 'i'
-     106: 176,  # 'j'
-     107: 177,  # 'k'
-     108: 77,  # 'l'
-     109: 72,  # 'm'
-     110: 178,  # 'n'
-     111: 69,  # 'o'
-     112: 67,  # 'p'
-     113: 179,  # 'q'
-     114: 78,  # 'r'
-     115: 73,  # 's'
-     116: 180,  # 't'
-     117: 181,  # 'u'
-     118: 79,  # 'v'
-     119: 182,  # 'w'
-     120: 183,  # 'x'
-     121: 184,  # 'y'
-     122: 185,  # 'z'
-     123: 253,  # '{'
-     124: 253,  # '|'
-     125: 253,  # '}'
-     126: 253,  # '~'
-     127: 253,  # '\x7f'
-     128: 37,  # 'А'
-     129: 44,  # 'Б'
-     130: 33,  # 'В'
-     131: 46,  # 'Г'
-     132: 41,  # 'Д'
-     133: 48,  # 'Е'
-     134: 56,  # 'Ж'
-     135: 51,  # 'З'
-     136: 42,  # 'И'
-     137: 60,  # 'Й'
-     138: 36,  # 'К'
-     139: 49,  # 'Л'
-     140: 38,  # 'М'
-     141: 31,  # 'Н'
-     142: 34,  # 'О'
-     143: 35,  # 'П'
-     144: 45,  # 'Р'
-     145: 32,  # 'С'
-     146: 40,  # 'Т'
-     147: 52,  # 'У'
-     148: 53,  # 'Ф'
-     149: 55,  # 'Х'
-     150: 58,  # 'Ц'
-     151: 50,  # 'Ч'
-     152: 57,  # 'Ш'
-     153: 63,  # 'Щ'
-     154: 70,  # 'Ъ'
-     155: 62,  # 'Ы'
-     156: 61,  # 'Ь'
-     157: 47,  # 'Э'
-     158: 59,  # 'Ю'
-     159: 43,  # 'Я'
-     160: 3,  # 'а'
-     161: 21,  # 'б'
-     162: 10,  # 'в'
-     163: 19,  # 'г'
-     164: 13,  # 'д'
-     165: 2,  # 'е'
-     166: 24,  # 'ж'
-     167: 20,  # 'з'
-     168: 4,  # 'и'
-     169: 23,  # 'й'
-     170: 11,  # 'к'
-     171: 8,  # 'л'
-     172: 12,  # 'м'
-     173: 5,  # 'н'
-     174: 1,  # 'о'
-     175: 15,  # 'п'
-     176: 191,  # '░'
-     177: 192,  # '▒'
-     178: 193,  # '▓'
-     179: 194,  # '│'
-     180: 195,  # '┤'
-     181: 196,  # '╡'
-     182: 197,  # '╢'
-     183: 198,  # '╖'
-     184: 199,  # '╕'
-     185: 200,  # '╣'

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/langthaimodel.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/langthaimodel.py Changed

@@ -1,9 +1,5 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-
 from chardet.sbcharsetprober import SingleByteCharSetModel
 
-
 # 3: Positive
 # 2: Likely
 # 1: Unlikely
@@ -4115,269 +4111,270 @@
 
 # Character Mapping Table(s):
 TIS_620_THAI_CHAR_TO_ORDER = {
-     0: 255,  # '\x00'
-     1: 255,  # '\x01'
-     2: 255,  # '\x02'
-     3: 255,  # '\x03'
-     4: 255,  # '\x04'
-     5: 255,  # '\x05'
-     6: 255,  # '\x06'
-     7: 255,  # '\x07'
-     8: 255,  # '\x08'
-     9: 255,  # '\t'
-     10: 254,  # '\n'
-     11: 255,  # '\x0b'
-     12: 255,  # '\x0c'
-     13: 254,  # '\r'
-     14: 255,  # '\x0e'
-     15: 255,  # '\x0f'
-     16: 255,  # '\x10'
-     17: 255,  # '\x11'
-     18: 255,  # '\x12'
-     19: 255,  # '\x13'
-     20: 255,  # '\x14'
-     21: 255,  # '\x15'
-     22: 255,  # '\x16'
-     23: 255,  # '\x17'
-     24: 255,  # '\x18'
-     25: 255,  # '\x19'
-     26: 255,  # '\x1a'
-     27: 255,  # '\x1b'
-     28: 255,  # '\x1c'
-     29: 255,  # '\x1d'
-     30: 255,  # '\x1e'
-     31: 255,  # '\x1f'
-     32: 253,  # ' '
-     33: 253,  # '!'
-     34: 253,  # '"'
-     35: 253,  # '#'
-     36: 253,  # '$'
-     37: 253,  # '%'
-     38: 253,  # '&'
-     39: 253,  # "'"
-     40: 253,  # '('
-     41: 253,  # ')'
-     42: 253,  # '*'
-     43: 253,  # '+'
-     44: 253,  # ','
-     45: 253,  # '-'
-     46: 253,  # '.'
-     47: 253,  # '/'
-     48: 252,  # '0'
-     49: 252,  # '1'
-     50: 252,  # '2'
-     51: 252,  # '3'
-     52: 252,  # '4'
-     53: 252,  # '5'
-     54: 252,  # '6'
-     55: 252,  # '7'
-     56: 252,  # '8'
-     57: 252,  # '9'
-     58: 253,  # ':'
-     59: 253,  # ';'
-     60: 253,  # '<'
-     61: 253,  # '='
-     62: 253,  # '>'
-     63: 253,  # '?'
-     64: 253,  # '@'
-     65: 182,  # 'A'
-     66: 106,  # 'B'
-     67: 107,  # 'C'
-     68: 100,  # 'D'
-     69: 183,  # 'E'
-     70: 184,  # 'F'
-     71: 185,  # 'G'
-     72: 101,  # 'H'
-     73: 94,  # 'I'
-     74: 186,  # 'J'
-     75: 187,  # 'K'
-     76: 108,  # 'L'
-     77: 109,  # 'M'
-     78: 110,  # 'N'
-     79: 111,  # 'O'
-     80: 188,  # 'P'
-     81: 189,  # 'Q'
-     82: 190,  # 'R'
-     83: 89,  # 'S'
-     84: 95,  # 'T'
-     85: 112,  # 'U'
-     86: 113,  # 'V'
-     87: 191,  # 'W'
-     88: 192,  # 'X'
-     89: 193,  # 'Y'
-     90: 194,  # 'Z'
-     91: 253,  # ''
-     92: 253,  # '\\'
-     93: 253,  # ''
-     94: 253,  # '^'
-     95: 253,  # '_'
-     96: 253,  # '`'
-     97: 64,  # 'a'
-     98: 72,  # 'b'
-     99: 73,  # 'c'
-     100: 114,  # 'd'
-     101: 74,  # 'e'
-     102: 115,  # 'f'
-     103: 116,  # 'g'
-     104: 102,  # 'h'
-     105: 81,  # 'i'
-     106: 201,  # 'j'
-     107: 117,  # 'k'
-     108: 90,  # 'l'
-     109: 103,  # 'm'
-     110: 78,  # 'n'
-     111: 82,  # 'o'
-     112: 96,  # 'p'
-     113: 202,  # 'q'
-     114: 91,  # 'r'
-     115: 79,  # 's'
-     116: 84,  # 't'
-     117: 104,  # 'u'
-     118: 105,  # 'v'
-     119: 97,  # 'w'
-     120: 98,  # 'x'
-     121: 92,  # 'y'
-     122: 203,  # 'z'
-     123: 253,  # '{'
-     124: 253,  # '|'
-     125: 253,  # '}'
-     126: 253,  # '~'
-     127: 253,  # '\x7f'
-     128: 209,  # '\x80'
-     129: 210,  # '\x81'
-     130: 211,  # '\x82'
-     131: 212,  # '\x83'
-     132: 213,  # '\x84'
-     133: 88,  # '\x85'
-     134: 214,  # '\x86'
-     135: 215,  # '\x87'
-     136: 216,  # '\x88'
-     137: 217,  # '\x89'
-     138: 218,  # '\x8a'
-     139: 219,  # '\x8b'
-     140: 220,  # '\x8c'
-     141: 118,  # '\x8d'
-     142: 221,  # '\x8e'
-     143: 222,  # '\x8f'
-     144: 223,  # '\x90'
-     145: 224,  # '\x91'
-     146: 99,  # '\x92'
-     147: 85,  # '\x93'
-     148: 83,  # '\x94'
-     149: 225,  # '\x95'
-     150: 226,  # '\x96'
-     151: 227,  # '\x97'
-     152: 228,  # '\x98'
-     153: 229,  # '\x99'
-     154: 230,  # '\x9a'
-     155: 231,  # '\x9b'
-     156: 232,  # '\x9c'
-     157: 233,  # '\x9d'
-     158: 234,  # '\x9e'
-     159: 235,  # '\x9f'
-     160: 236,  # None
-     161: 5,  # 'ก'
-     162: 30,  # 'ข'
-     163: 237,  # 'ฃ'
-     164: 24,  # 'ค'
-     165: 238,  # 'ฅ'
-     166: 75,  # 'ฆ'
-     167: 8,  # 'ง'
-     168: 26,  # 'จ'
-     169: 52,  # 'ฉ'
-     170: 34,  # 'ช'
-     171: 51,  # 'ซ'
-     172: 119,  # 'ฌ'
-     173: 47,  # 'ญ'
-     174: 58,  # 'ฎ'
-     175: 57,  # 'ฏ'
-     176: 49,  # 'ฐ'
-     177: 53,  # 'ฑ'
-     178: 55,  # 'ฒ'
-     179: 43,  # 'ณ'
-     180: 20,  # 'ด'
-     181: 19,  # 'ต'
-     182: 44,  # 'ถ'
-     183: 14,  # 'ท'
-     184: 48,  # 'ธ'
-     185: 3,  # 'น'

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/langturkishmodel.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/langturkishmodel.py Changed

@@ -1,9 +1,5 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-
 from chardet.sbcharsetprober import SingleByteCharSetModel
 
-
 # 3: Positive
 # 2: Likely
 # 1: Unlikely
@@ -4115,269 +4111,270 @@
 
 # Character Mapping Table(s):
 ISO_8859_9_TURKISH_CHAR_TO_ORDER = {
-     0: 255,  # '\x00'
-     1: 255,  # '\x01'
-     2: 255,  # '\x02'
-     3: 255,  # '\x03'
-     4: 255,  # '\x04'
-     5: 255,  # '\x05'
-     6: 255,  # '\x06'
-     7: 255,  # '\x07'
-     8: 255,  # '\x08'
-     9: 255,  # '\t'
-     10: 255,  # '\n'
-     11: 255,  # '\x0b'
-     12: 255,  # '\x0c'
-     13: 255,  # '\r'
-     14: 255,  # '\x0e'
-     15: 255,  # '\x0f'
-     16: 255,  # '\x10'
-     17: 255,  # '\x11'
-     18: 255,  # '\x12'
-     19: 255,  # '\x13'
-     20: 255,  # '\x14'
-     21: 255,  # '\x15'
-     22: 255,  # '\x16'
-     23: 255,  # '\x17'
-     24: 255,  # '\x18'
-     25: 255,  # '\x19'
-     26: 255,  # '\x1a'
-     27: 255,  # '\x1b'
-     28: 255,  # '\x1c'
-     29: 255,  # '\x1d'
-     30: 255,  # '\x1e'
-     31: 255,  # '\x1f'
-     32: 255,  # ' '
-     33: 255,  # '!'
-     34: 255,  # '"'
-     35: 255,  # '#'
-     36: 255,  # '$'
-     37: 255,  # '%'
-     38: 255,  # '&'
-     39: 255,  # "'"
-     40: 255,  # '('
-     41: 255,  # ')'
-     42: 255,  # '*'
-     43: 255,  # '+'
-     44: 255,  # ','
-     45: 255,  # '-'
-     46: 255,  # '.'
-     47: 255,  # '/'
-     48: 255,  # '0'
-     49: 255,  # '1'
-     50: 255,  # '2'
-     51: 255,  # '3'
-     52: 255,  # '4'
-     53: 255,  # '5'
-     54: 255,  # '6'
-     55: 255,  # '7'
-     56: 255,  # '8'
-     57: 255,  # '9'
-     58: 255,  # ':'
-     59: 255,  # ';'
-     60: 255,  # '<'
-     61: 255,  # '='
-     62: 255,  # '>'
-     63: 255,  # '?'
-     64: 255,  # '@'
-     65: 23,  # 'A'
-     66: 37,  # 'B'
-     67: 47,  # 'C'
-     68: 39,  # 'D'
-     69: 29,  # 'E'
-     70: 52,  # 'F'
-     71: 36,  # 'G'
-     72: 45,  # 'H'
-     73: 53,  # 'I'
-     74: 60,  # 'J'
-     75: 16,  # 'K'
-     76: 49,  # 'L'
-     77: 20,  # 'M'
-     78: 46,  # 'N'
-     79: 42,  # 'O'
-     80: 48,  # 'P'
-     81: 69,  # 'Q'
-     82: 44,  # 'R'
-     83: 35,  # 'S'
-     84: 31,  # 'T'
-     85: 51,  # 'U'
-     86: 38,  # 'V'
-     87: 62,  # 'W'
-     88: 65,  # 'X'
-     89: 43,  # 'Y'
-     90: 56,  # 'Z'
-     91: 255,  # ''
-     92: 255,  # '\\'
-     93: 255,  # ''
-     94: 255,  # '^'
-     95: 255,  # '_'
-     96: 255,  # '`'
-     97: 1,  # 'a'
-     98: 21,  # 'b'
-     99: 28,  # 'c'
-     100: 12,  # 'd'
-     101: 2,  # 'e'
-     102: 18,  # 'f'
-     103: 27,  # 'g'
-     104: 25,  # 'h'
-     105: 3,  # 'i'
-     106: 24,  # 'j'
-     107: 10,  # 'k'
-     108: 5,  # 'l'
-     109: 13,  # 'm'
-     110: 4,  # 'n'
-     111: 15,  # 'o'
-     112: 26,  # 'p'
-     113: 64,  # 'q'
-     114: 7,  # 'r'
-     115: 8,  # 's'
-     116: 9,  # 't'
-     117: 14,  # 'u'
-     118: 32,  # 'v'
-     119: 57,  # 'w'
-     120: 58,  # 'x'
-     121: 11,  # 'y'
-     122: 22,  # 'z'
-     123: 255,  # '{'
-     124: 255,  # '|'
-     125: 255,  # '}'
-     126: 255,  # '~'
-     127: 255,  # '\x7f'
-     128: 180,  # '\x80'
-     129: 179,  # '\x81'
-     130: 178,  # '\x82'
-     131: 177,  # '\x83'
-     132: 176,  # '\x84'
-     133: 175,  # '\x85'
-     134: 174,  # '\x86'
-     135: 173,  # '\x87'
-     136: 172,  # '\x88'
-     137: 171,  # '\x89'
-     138: 170,  # '\x8a'
-     139: 169,  # '\x8b'
-     140: 168,  # '\x8c'
-     141: 167,  # '\x8d'
-     142: 166,  # '\x8e'
-     143: 165,  # '\x8f'
-     144: 164,  # '\x90'
-     145: 163,  # '\x91'
-     146: 162,  # '\x92'
-     147: 161,  # '\x93'
-     148: 160,  # '\x94'
-     149: 159,  # '\x95'
-     150: 101,  # '\x96'
-     151: 158,  # '\x97'
-     152: 157,  # '\x98'
-     153: 156,  # '\x99'
-     154: 155,  # '\x9a'
-     155: 154,  # '\x9b'
-     156: 153,  # '\x9c'
-     157: 152,  # '\x9d'
-     158: 151,  # '\x9e'
-     159: 106,  # '\x9f'
-     160: 150,  # '\xa0'
-     161: 149,  # '¡'
-     162: 148,  # '¢'
-     163: 147,  # '£'
-     164: 146,  # '¤'
-     165: 145,  # '¥'
-     166: 144,  # '¦'
-     167: 100,  # '§'
-     168: 143,  # '¨'
-     169: 142,  # '©'
-     170: 141,  # 'ª'
-     171: 140,  # '«'
-     172: 139,  # '¬'
-     173: 138,  # '\xad'
-     174: 137,  # '®'
-     175: 136,  # '¯'
-     176: 94,  # '°'
-     177: 80,  # '±'
-     178: 93,  # '²'
-     179: 135,  # '³'
-     180: 105,  # '´'
-     181: 134,  # 'µ'
-     182: 133,  # '¶'
-     183: 63,  # '·'
-     184: 132,  # '¸'
-     185: 131,  # '¹'

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/latin1prober.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/latin1prober.py Changed

@@ -26,6 +26,8 @@
 # 02110-1301  USA
 ######################### END LICENSE BLOCK #########################
 
+from typing import List, Union
+
 from .charsetprober import CharSetProber
 from .enums import ProbingState
 
@@ -41,6 +43,7 @@
 ASO = 7  # accent small other
 CLASS_NUM = 8  # total classes
 
+# fmt: off
 Latin1_CharToClass = (
     OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH,   # 00 - 07
     OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH,   # 08 - 0F
@@ -91,34 +94,34 @@
     0,  3,  1,  3,  1,  1,  1,  3,  # ASV
     0,  3,  1,  3,  1,  1,  3,  3,  # ASO
 )
+# fmt: on
 
 
 class Latin1Prober(CharSetProber):
-    def __init__(self):
-        super(Latin1Prober, self).__init__()
-        self._last_char_class = None
-        self._freq_counter = None
+    def __init__(self) -> None:
+        super().__init__()
+        self._last_char_class = OTH
+        self._freq_counter: Listint = 
         self.reset()
 
-    def reset(self):
+    def reset(self) -> None:
         self._last_char_class = OTH
         self._freq_counter = 0 * FREQ_CAT_NUM
-        CharSetProber.reset(self)
+        super().reset()
 
     @property
-    def charset_name(self):
+    def charset_name(self) -> str:
         return "ISO-8859-1"
 
     @property
-    def language(self):
+    def language(self) -> str:
         return ""
 
-    def feed(self, byte_str):
-        byte_str = self.filter_with_english_letters(byte_str)
+    def feed(self, byte_str: Unionbytes, bytearray) -> ProbingState:
+        byte_str = self.remove_xml_tags(byte_str)
         for c in byte_str:
             char_class = Latin1_CharToClassc
-            freq = Latin1ClassModel(self._last_char_class * CLASS_NUM)
-                                    + char_class
+            freq = Latin1ClassModel(self._last_char_class * CLASS_NUM) + char_class
             if freq == 0:
                 self._state = ProbingState.NOT_ME
                 break
@@ -127,19 +130,18 @@
 
         return self.state
 
-    def get_confidence(self):
+    def get_confidence(self) -> float:
         if self.state == ProbingState.NOT_ME:
             return 0.01
 
         total = sum(self._freq_counter)
-        if total < 0.01:
-            confidence = 0.0
-        else:
-            confidence = ((self._freq_counter3 - self._freq_counter1 * 20.0)
-                          / total)
-        if confidence < 0.0:
-            confidence = 0.0
+        confidence = (
+            0.0
+            if total < 0.01
+            else (self._freq_counter3 - self._freq_counter1 * 20.0) / total
+        )
+        confidence = max(confidence, 0.0)
         # lower the confidence of latin1 so that other more accurate
         # detector can take priority.
-        confidence = confidence * 0.73
+        confidence *= 0.73
         return confidence

_service:tar_scm:chardet-5.1.0.tar.gz/chardet/macromanprober.py Added

@@ -0,0 +1,162 @@
+######################## BEGIN LICENSE BLOCK ########################
+# This code was modified from latin1prober.py by Rob Speer <rob@lumino.so>.
+# The Original Code is Mozilla Universal charset detector code.
+#
+# The Initial Developer of the Original Code is
+# Netscape Communications Corporation.
+# Portions created by the Initial Developer are Copyright (C) 2001
+# the Initial Developer. All Rights Reserved.
+#
+# Contributor(s):
+#   Rob Speer - adapt to MacRoman encoding
+#   Mark Pilgrim - port to Python
+#   Shy Shalom - original C code
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
+# 02110-1301  USA
+######################### END LICENSE BLOCK #########################
+
+from typing import List, Union
+
+from .charsetprober import CharSetProber
+from .enums import ProbingState
+
+FREQ_CAT_NUM = 4
+
+UDF = 0  # undefined
+OTH = 1  # other
+ASC = 2  # ascii capital letter
+ASS = 3  # ascii small letter
+ACV = 4  # accent capital vowel
+ACO = 5  # accent capital other
+ASV = 6  # accent small vowel
+ASO = 7  # accent small other
+ODD = 8  # character that is unlikely to appear
+CLASS_NUM = 9  # total classes
+
+# The change from Latin1 is that we explicitly look for extended characters
+# that are infrequently-occurring symbols, and consider them to always be
+# improbable. This should let MacRoman get out of the way of more likely
+# encodings in most situations.
+
+# fmt: off
+MacRoman_CharToClass = (
+    OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH,  # 00 - 07
+    OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH,  # 08 - 0F
+    OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH,  # 10 - 17
+    OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH,  # 18 - 1F
+    OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH,  # 20 - 27
+    OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH,  # 28 - 2F
+    OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH,  # 30 - 37
+    OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH,  # 38 - 3F
+    OTH, ASC, ASC, ASC, ASC, ASC, ASC, ASC,  # 40 - 47
+    ASC, ASC, ASC, ASC, ASC, ASC, ASC, ASC,  # 48 - 4F
+    ASC, ASC, ASC, ASC, ASC, ASC, ASC, ASC,  # 50 - 57
+    ASC, ASC, ASC, OTH, OTH, OTH, OTH, OTH,  # 58 - 5F
+    OTH, ASS, ASS, ASS, ASS, ASS, ASS, ASS,  # 60 - 67
+    ASS, ASS, ASS, ASS, ASS, ASS, ASS, ASS,  # 68 - 6F
+    ASS, ASS, ASS, ASS, ASS, ASS, ASS, ASS,  # 70 - 77
+    ASS, ASS, ASS, OTH, OTH, OTH, OTH, OTH,  # 78 - 7F
+    ACV, ACV, ACO, ACV, ACO, ACV, ACV, ASV,  # 80 - 87
+    ASV, ASV, ASV, ASV, ASV, ASO, ASV, ASV,  # 88 - 8F
+    ASV, ASV, ASV, ASV, ASV, ASV, ASO, ASV,  # 90 - 97
+    ASV, ASV, ASV, ASV, ASV, ASV, ASV, ASV,  # 98 - 9F
+    OTH, OTH, OTH, OTH, OTH, OTH, OTH, ASO,  # A0 - A7
+    OTH, OTH, ODD, ODD, OTH, OTH, ACV, ACV,  # A8 - AF
+    OTH, OTH, OTH, OTH, OTH, OTH, OTH, OTH,  # B0 - B7
+    OTH, OTH, OTH, OTH, OTH, OTH, ASV, ASV,  # B8 - BF
+    OTH, OTH, ODD, OTH, ODD, OTH, OTH, OTH,  # C0 - C7
+    OTH, OTH, OTH, ACV, ACV, ACV, ACV, ASV,  # C8 - CF
+    OTH, OTH, OTH, OTH, OTH, OTH, OTH, ODD,  # D0 - D7
+    ASV, ACV, ODD, OTH, OTH, OTH, OTH, OTH,  # D8 - DF
+    OTH, OTH, OTH, OTH, OTH, ACV, ACV, ACV,  # E0 - E7
+    ACV, ACV, ACV, ACV, ACV, ACV, ACV, ACV,  # E8 - EF
+    ODD, ACV, ACV, ACV, ACV, ASV, ODD, ODD,  # F0 - F7
+    ODD, ODD, ODD, ODD, ODD, ODD, ODD, ODD,  # F8 - FF
+)
+
+# 0 : illegal
+# 1 : very unlikely
+# 2 : normal
+# 3 : very likely
+MacRomanClassModel = (
+# UDF OTH ASC ASS ACV ACO ASV ASO ODD
+    0,  0,  0,  0,  0,  0,  0,  0,  0,  # UDF
+    0,  3,  3,  3,  3,  3,  3,  3,  1,  # OTH
+    0,  3,  3,  3,  3,  3,  3,  3,  1,  # ASC
+    0,  3,  3,  3,  1,  1,  3,  3,  1,  # ASS
+    0,  3,  3,  3,  1,  2,  1,  2,  1,  # ACV
+    0,  3,  3,  3,  3,  3,  3,  3,  1,  # ACO
+    0,  3,  1,  3,  1,  1,  1,  3,  1,  # ASV
+    0,  3,  1,  3,  1,  1,  3,  3,  1,  # ASO
+    0,  1,  1,  1,  1,  1,  1,  1,  1,  # ODD
+)
+# fmt: on
+
+
+class MacRomanProber(CharSetProber):
+    def __init__(self) -> None:
+        super().__init__()
+        self._last_char_class = OTH
+        self._freq_counter: Listint = 
+        self.reset()
+
+    def reset(self) -> None:
+        self._last_char_class = OTH
+        self._freq_counter = 0 * FREQ_CAT_NUM
+
+        # express the prior that MacRoman is a somewhat rare encoding;
+        # this can be done by starting out in a slightly improbable state
+        # that must be overcome
+        self._freq_counter2 = 10
+
+        super().reset()
+
+    @property
+    def charset_name(self) -> str:
+        return "MacRoman"
+
+    @property
+    def language(self) -> str:
+        return ""
+
+    def feed(self, byte_str: Unionbytes, bytearray) -> ProbingState:
+        byte_str = self.remove_xml_tags(byte_str)
+        for c in byte_str:
+            char_class = MacRoman_CharToClassc
+            freq = MacRomanClassModel(self._last_char_class * CLASS_NUM) + char_class
+            if freq == 0:
+                self._state = ProbingState.NOT_ME
+                break
+            self._freq_counterfreq += 1
+            self._last_char_class = char_class
+
+        return self.state
+
+    def get_confidence(self) -> float:
+        if self.state == ProbingState.NOT_ME:
+            return 0.01
+
+        total = sum(self._freq_counter)
+        confidence = (
+            0.0
+            if total < 0.01
+            else (self._freq_counter3 - self._freq_counter1 * 20.0) / total
+        )
+        confidence = max(confidence, 0.0)
+        # lower the confidence of MacRoman so that other more accurate
+        # detector can take priority.
+        confidence *= 0.73
+        return confidence

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/mbcharsetprober.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/mbcharsetprober.py Changed

@@ -27,8 +27,12 @@
 # 02110-1301  USA
 ######################### END LICENSE BLOCK #########################
 
+from typing import Optional, Union
+
+from .chardistribution import CharDistributionAnalysis
 from .charsetprober import CharSetProber
-from .enums import ProbingState, MachineState
+from .codingstatemachine import CodingStateMachine
+from .enums import LanguageFilter, MachineState, ProbingState
 
 
 class MultiByteCharSetProber(CharSetProber):
@@ -36,56 +40,56 @@
     MultiByteCharSetProber
     """
 
-    def __init__(self, lang_filter=None):
-        super(MultiByteCharSetProber, self).__init__(lang_filter=lang_filter)
-        self.distribution_analyzer = None
-        self.coding_sm = None
-        self._last_char = 0, 0
+    def __init__(self, lang_filter: LanguageFilter = LanguageFilter.NONE) -> None:
+        super().__init__(lang_filter=lang_filter)
+        self.distribution_analyzer: OptionalCharDistributionAnalysis = None
+        self.coding_sm: OptionalCodingStateMachine = None
+        self._last_char = bytearray(b"\0\0")
 
-    def reset(self):
-        super(MultiByteCharSetProber, self).reset()
+    def reset(self) -> None:
+        super().reset()
         if self.coding_sm:
             self.coding_sm.reset()
         if self.distribution_analyzer:
             self.distribution_analyzer.reset()
-        self._last_char = 0, 0
-
-    @property
-    def charset_name(self):
-        raise NotImplementedError
+        self._last_char = bytearray(b"\0\0")
 
-    @property
-    def language(self):
-        raise NotImplementedError
+    def feed(self, byte_str: Unionbytes, bytearray) -> ProbingState:
+        assert self.coding_sm is not None
+        assert self.distribution_analyzer is not None
 
-    def feed(self, byte_str):
-        for i in range(len(byte_str)):
-            coding_state = self.coding_sm.next_state(byte_stri)
+        for i, byte in enumerate(byte_str):
+            coding_state = self.coding_sm.next_state(byte)
             if coding_state == MachineState.ERROR:
-                self.logger.debug('%s %s prober hit error at byte %s',
-                                  self.charset_name, self.language, i)
+                self.logger.debug(
+                    "%s %s prober hit error at byte %s",
+                    self.charset_name,
+                    self.language,
+                    i,
+                )
                 self._state = ProbingState.NOT_ME
                 break
-            elif coding_state == MachineState.ITS_ME:
+            if coding_state == MachineState.ITS_ME:
                 self._state = ProbingState.FOUND_IT
                 break
-            elif coding_state == MachineState.START:
+            if coding_state == MachineState.START:
                 char_len = self.coding_sm.get_current_charlen()
                 if i == 0:
-                    self._last_char1 = byte_str0
+                    self._last_char1 = byte
                     self.distribution_analyzer.feed(self._last_char, char_len)
                 else:
-                    self.distribution_analyzer.feed(byte_stri - 1:i + 1,
-                                                    char_len)
+                    self.distribution_analyzer.feed(byte_stri - 1 : i + 1, char_len)
 
         self._last_char0 = byte_str-1
 
         if self.state == ProbingState.DETECTING:
-            if (self.distribution_analyzer.got_enough_data() and
-                    (self.get_confidence() > self.SHORTCUT_THRESHOLD)):
+            if self.distribution_analyzer.got_enough_data() and (
+                self.get_confidence() > self.SHORTCUT_THRESHOLD
+            ):
                 self._state = ProbingState.FOUND_IT
 
         return self.state
 
-    def get_confidence(self):
+    def get_confidence(self) -> float:
+        assert self.distribution_analyzer is not None
         return self.distribution_analyzer.get_confidence()

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/mbcsgroupprober.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/mbcsgroupprober.py Changed

@@ -27,20 +27,22 @@
 # 02110-1301  USA
 ######################### END LICENSE BLOCK #########################
 
+from .big5prober import Big5Prober
 from .charsetgroupprober import CharSetGroupProber
-from .utf8prober import UTF8Prober
-from .sjisprober import SJISProber
+from .cp949prober import CP949Prober
+from .enums import LanguageFilter
 from .eucjpprober import EUCJPProber
-from .gb2312prober import GB2312Prober
 from .euckrprober import EUCKRProber
-from .cp949prober import CP949Prober
-from .big5prober import Big5Prober
 from .euctwprober import EUCTWProber
+from .gb2312prober import GB2312Prober
+from .johabprober import JOHABProber
+from .sjisprober import SJISProber
+from .utf8prober import UTF8Prober
 
 
 class MBCSGroupProber(CharSetGroupProber):
-    def __init__(self, lang_filter=None):
-        super(MBCSGroupProber, self).__init__(lang_filter=lang_filter)
+    def __init__(self, lang_filter: LanguageFilter = LanguageFilter.NONE) -> None:
+        super().__init__(lang_filter=lang_filter)
         self.probers = 
             UTF8Prober(),
             SJISProber(),
@@ -49,6 +51,7 @@
             EUCKRProber(),
             CP949Prober(),
             Big5Prober(),
-            EUCTWProber()
+            EUCTWProber(),
+            JOHABProber(),
         
         self.reset()

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/mbcssm.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/mbcssm.py Changed

@@ -25,43 +25,45 @@
 # 02110-1301  USA
 ######################### END LICENSE BLOCK #########################
 
+from .codingstatemachinedict import CodingStateMachineDict
 from .enums import MachineState
 
 # BIG5
 
+# fmt: off
 BIG5_CLS = (
-    1,1,1,1,1,1,1,1,  # 00 - 07    #allow 0x00 as legal value
-    1,1,1,1,1,1,0,0,  # 08 - 0f
-    1,1,1,1,1,1,1,1,  # 10 - 17
-    1,1,1,0,1,1,1,1,  # 18 - 1f
-    1,1,1,1,1,1,1,1,  # 20 - 27
-    1,1,1,1,1,1,1,1,  # 28 - 2f
-    1,1,1,1,1,1,1,1,  # 30 - 37
-    1,1,1,1,1,1,1,1,  # 38 - 3f
-    2,2,2,2,2,2,2,2,  # 40 - 47
-    2,2,2,2,2,2,2,2,  # 48 - 4f
-    2,2,2,2,2,2,2,2,  # 50 - 57
-    2,2,2,2,2,2,2,2,  # 58 - 5f
-    2,2,2,2,2,2,2,2,  # 60 - 67
-    2,2,2,2,2,2,2,2,  # 68 - 6f
-    2,2,2,2,2,2,2,2,  # 70 - 77
-    2,2,2,2,2,2,2,1,  # 78 - 7f
-    4,4,4,4,4,4,4,4,  # 80 - 87
-    4,4,4,4,4,4,4,4,  # 88 - 8f
-    4,4,4,4,4,4,4,4,  # 90 - 97
-    4,4,4,4,4,4,4,4,  # 98 - 9f
-    4,3,3,3,3,3,3,3,  # a0 - a7
-    3,3,3,3,3,3,3,3,  # a8 - af
-    3,3,3,3,3,3,3,3,  # b0 - b7
-    3,3,3,3,3,3,3,3,  # b8 - bf
-    3,3,3,3,3,3,3,3,  # c0 - c7
-    3,3,3,3,3,3,3,3,  # c8 - cf
-    3,3,3,3,3,3,3,3,  # d0 - d7
-    3,3,3,3,3,3,3,3,  # d8 - df
-    3,3,3,3,3,3,3,3,  # e0 - e7
-    3,3,3,3,3,3,3,3,  # e8 - ef
-    3,3,3,3,3,3,3,3,  # f0 - f7
-    3,3,3,3,3,3,3,0  # f8 - ff
+    1, 1, 1, 1, 1, 1, 1, 1,  # 00 - 07    #allow 0x00 as legal value
+    1, 1, 1, 1, 1, 1, 0, 0,  # 08 - 0f
+    1, 1, 1, 1, 1, 1, 1, 1,  # 10 - 17
+    1, 1, 1, 0, 1, 1, 1, 1,  # 18 - 1f
+    1, 1, 1, 1, 1, 1, 1, 1,  # 20 - 27
+    1, 1, 1, 1, 1, 1, 1, 1,  # 28 - 2f
+    1, 1, 1, 1, 1, 1, 1, 1,  # 30 - 37
+    1, 1, 1, 1, 1, 1, 1, 1,  # 38 - 3f
+    2, 2, 2, 2, 2, 2, 2, 2,  # 40 - 47
+    2, 2, 2, 2, 2, 2, 2, 2,  # 48 - 4f
+    2, 2, 2, 2, 2, 2, 2, 2,  # 50 - 57
+    2, 2, 2, 2, 2, 2, 2, 2,  # 58 - 5f
+    2, 2, 2, 2, 2, 2, 2, 2,  # 60 - 67
+    2, 2, 2, 2, 2, 2, 2, 2,  # 68 - 6f
+    2, 2, 2, 2, 2, 2, 2, 2,  # 70 - 77
+    2, 2, 2, 2, 2, 2, 2, 1,  # 78 - 7f
+    4, 4, 4, 4, 4, 4, 4, 4,  # 80 - 87
+    4, 4, 4, 4, 4, 4, 4, 4,  # 88 - 8f
+    4, 4, 4, 4, 4, 4, 4, 4,  # 90 - 97
+    4, 4, 4, 4, 4, 4, 4, 4,  # 98 - 9f
+    4, 3, 3, 3, 3, 3, 3, 3,  # a0 - a7
+    3, 3, 3, 3, 3, 3, 3, 3,  # a8 - af
+    3, 3, 3, 3, 3, 3, 3, 3,  # b0 - b7
+    3, 3, 3, 3, 3, 3, 3, 3,  # b8 - bf
+    3, 3, 3, 3, 3, 3, 3, 3,  # c0 - c7
+    3, 3, 3, 3, 3, 3, 3, 3,  # c8 - cf
+    3, 3, 3, 3, 3, 3, 3, 3,  # d0 - d7
+    3, 3, 3, 3, 3, 3, 3, 3,  # d8 - df
+    3, 3, 3, 3, 3, 3, 3, 3,  # e0 - e7
+    3, 3, 3, 3, 3, 3, 3, 3,  # e8 - ef
+    3, 3, 3, 3, 3, 3, 3, 3,  # f0 - f7
+    3, 3, 3, 3, 3, 3, 3, 0  # f8 - ff
 )
 
 BIG5_ST = (
@@ -69,34 +71,37 @@
     MachineState.ERROR,MachineState.ERROR,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ITS_ME,MachineState.ERROR,#08-0f
     MachineState.ERROR,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START#10-17
 )
+# fmt: on
 
 BIG5_CHAR_LEN_TABLE = (0, 1, 1, 2, 0)
 
-BIG5_SM_MODEL = {'class_table': BIG5_CLS,
-                 'class_factor': 5,
-                 'state_table': BIG5_ST,
-                 'char_len_table': BIG5_CHAR_LEN_TABLE,
-                 'name': 'Big5'}
+BIG5_SM_MODEL: CodingStateMachineDict = {
+    "class_table": BIG5_CLS,
+    "class_factor": 5,
+    "state_table": BIG5_ST,
+    "char_len_table": BIG5_CHAR_LEN_TABLE,
+    "name": "Big5",
+}
 
 # CP949
-
+# fmt: off
 CP949_CLS  = (
-    1,1,1,1,1,1,1,1, 1,1,1,1,1,1,0,0,  # 00 - 0f
-    1,1,1,1,1,1,1,1, 1,1,1,0,1,1,1,1,  # 10 - 1f
-    1,1,1,1,1,1,1,1, 1,1,1,1,1,1,1,1,  # 20 - 2f
-    1,1,1,1,1,1,1,1, 1,1,1,1,1,1,1,1,  # 30 - 3f
-    1,4,4,4,4,4,4,4, 4,4,4,4,4,4,4,4,  # 40 - 4f
-    4,4,5,5,5,5,5,5, 5,5,5,1,1,1,1,1,  # 50 - 5f
-    1,5,5,5,5,5,5,5, 5,5,5,5,5,5,5,5,  # 60 - 6f
-    5,5,5,5,5,5,5,5, 5,5,5,1,1,1,1,1,  # 70 - 7f
-    0,6,6,6,6,6,6,6, 6,6,6,6,6,6,6,6,  # 80 - 8f
-    6,6,6,6,6,6,6,6, 6,6,6,6,6,6,6,6,  # 90 - 9f
-    6,7,7,7,7,7,7,7, 7,7,7,7,7,8,8,8,  # a0 - af
-    7,7,7,7,7,7,7,7, 7,7,7,7,7,7,7,7,  # b0 - bf
-    7,7,7,7,7,7,9,2, 2,3,2,2,2,2,2,2,  # c0 - cf
-    2,2,2,2,2,2,2,2, 2,2,2,2,2,2,2,2,  # d0 - df
-    2,2,2,2,2,2,2,2, 2,2,2,2,2,2,2,2,  # e0 - ef
-    2,2,2,2,2,2,2,2, 2,2,2,2,2,2,2,0,  # f0 - ff
+    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0,  # 00 - 0f
+    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1,  # 10 - 1f
+    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,  # 20 - 2f
+    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,  # 30 - 3f
+    1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,  # 40 - 4f
+    4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 5, 1, 1, 1, 1, 1,  # 50 - 5f
+    1, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,  # 60 - 6f
+    5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 1, 1, 1, 1, 1,  # 70 - 7f
+    0, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,  # 80 - 8f
+    6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6,  # 90 - 9f
+    6, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 8, 8, 8,  # a0 - af
+    7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7,  # b0 - bf
+    7, 7, 7, 7, 7, 7, 9, 2, 2, 3, 2, 2, 2, 2, 2, 2,  # c0 - cf
+    2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,  # d0 - df
+    2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,  # e0 - ef
+    2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0,  # f0 - ff
 )
 
 CP949_ST = (
@@ -109,50 +114,53 @@
     MachineState.ERROR,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.START, # 5
     MachineState.ERROR,MachineState.START,MachineState.START,MachineState.START,MachineState.START,MachineState.ERROR,MachineState.ERROR,MachineState.START,MachineState.START,MachineState.START, # 6
 )
+# fmt: on
 
 CP949_CHAR_LEN_TABLE = (0, 1, 2, 0, 1, 1, 2, 2, 0, 2)
 
-CP949_SM_MODEL = {'class_table': CP949_CLS,
-                  'class_factor': 10,
-                  'state_table': CP949_ST,
-                  'char_len_table': CP949_CHAR_LEN_TABLE,
-                  'name': 'CP949'}
+CP949_SM_MODEL: CodingStateMachineDict = {
+    "class_table": CP949_CLS,
+    "class_factor": 10,
+    "state_table": CP949_ST,
+    "char_len_table": CP949_CHAR_LEN_TABLE,
+    "name": "CP949",
+}
 
 # EUC-JP
-
+# fmt: off
 EUCJP_CLS = (
-    4,4,4,4,4,4,4,4,  # 00 - 07
-    4,4,4,4,4,4,5,5,  # 08 - 0f
-    4,4,4,4,4,4,4,4,  # 10 - 17
-    4,4,4,5,4,4,4,4,  # 18 - 1f
-    4,4,4,4,4,4,4,4,  # 20 - 27
-    4,4,4,4,4,4,4,4,  # 28 - 2f
-    4,4,4,4,4,4,4,4,  # 30 - 37
-    4,4,4,4,4,4,4,4,  # 38 - 3f
-    4,4,4,4,4,4,4,4,  # 40 - 47
-    4,4,4,4,4,4,4,4,  # 48 - 4f
-    4,4,4,4,4,4,4,4,  # 50 - 57
-    4,4,4,4,4,4,4,4,  # 58 - 5f
-    4,4,4,4,4,4,4,4,  # 60 - 67
-    4,4,4,4,4,4,4,4,  # 68 - 6f
-    4,4,4,4,4,4,4,4,  # 70 - 77
-    4,4,4,4,4,4,4,4,  # 78 - 7f
-    5,5,5,5,5,5,5,5,  # 80 - 87
-    5,5,5,5,5,5,1,3,  # 88 - 8f
-    5,5,5,5,5,5,5,5,  # 90 - 97
-    5,5,5,5,5,5,5,5,  # 98 - 9f
-    5,2,2,2,2,2,2,2,  # a0 - a7
-    2,2,2,2,2,2,2,2,  # a8 - af
-    2,2,2,2,2,2,2,2,  # b0 - b7
-    2,2,2,2,2,2,2,2,  # b8 - bf
-    2,2,2,2,2,2,2,2,  # c0 - c7
-    2,2,2,2,2,2,2,2,  # c8 - cf
-    2,2,2,2,2,2,2,2,  # d0 - d7
-    2,2,2,2,2,2,2,2,  # d8 - df
-    0,0,0,0,0,0,0,0,  # e0 - e7
-    0,0,0,0,0,0,0,0,  # e8 - ef
-    0,0,0,0,0,0,0,0,  # f0 - f7
-    0,0,0,0,0,0,0,5  # f8 - ff
+    4, 4, 4, 4, 4, 4, 4, 4,  # 00 - 07
+    4, 4, 4, 4, 4, 4, 5, 5,  # 08 - 0f
+    4, 4, 4, 4, 4, 4, 4, 4,  # 10 - 17
+    4, 4, 4, 5, 4, 4, 4, 4,  # 18 - 1f
+    4, 4, 4, 4, 4, 4, 4, 4,  # 20 - 27

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/metadata/languages.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/metadata/languages.py Changed

@@ -1,19 +1,17 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
 """
 Metadata about languages used by our model training code for our
 SingleByteCharSetProbers.  Could be used for other things in the future.
 
 This code is based on the language metadata from the uchardet project.
 """
-from __future__ import absolute_import, print_function
 
 from string import ascii_letters
+from typing import List, Optional
 
+# TODO: Add Ukrainian (KOI8-U)
 
-# TODO: Add Ukranian (KOI8-U)
 
-class Language(object):
+class Language:
     """Metadata about a language useful for training models
 
     :ivar name: The human name for the language, in English.
@@ -33,9 +31,17 @@
                             Wikipedia for training data.
     :type wiki_start_pages: list of str
     """
-    def __init__(self, name=None, iso_code=None, use_ascii=True, charsets=None,
-                 alphabet=None, wiki_start_pages=None):
-        super(Language, self).__init__()
+
+    def __init__(
+        self,
+        name: Optionalstr = None,
+        iso_code: Optionalstr = None,
+        use_ascii: bool = True,
+        charsets: OptionalListstr = None,
+        alphabet: Optionalstr = None,
+        wiki_start_pages: OptionalListstr = None,
+    ) -> None:
+        super().__init__()
         self.name = name
         self.iso_code = iso_code
         self.use_ascii = use_ascii
@@ -46,265 +52,301 @@
             else:
                 alphabet = ascii_letters
         elif not alphabet:
-            raise ValueError('Must supply alphabet if use_ascii is False')
-        self.alphabet = ''.join(sorted(set(alphabet))) if alphabet else None
+            raise ValueError("Must supply alphabet if use_ascii is False")
+        self.alphabet = "".join(sorted(set(alphabet))) if alphabet else None
         self.wiki_start_pages = wiki_start_pages
 
-    def __repr__(self):
-        return '{}({})'.format(self.__class__.__name__,
-                               ', '.join('{}={!r}'.format(k, v)
-                                         for k, v in self.__dict__.items()
-                                         if not k.startswith('_')))
+    def __repr__(self) -> str:
+        param_str = ", ".join(
+            f"{k}={v!r}" for k, v in self.__dict__.items() if not k.startswith("_")
+        )
+        return f"{self.__class__.__name__}({param_str})"
 
 
-LANGUAGES = {'Arabic': Language(name='Arabic',
-                                iso_code='ar',
-                                use_ascii=False,
-                                # We only support encodings that use isolated
-                                # forms, because the current recommendation is
-                                # that the rendering system handles presentation
-                                # forms. This means we purposefully skip IBM864.
-                                charsets='ISO-8859-6', 'WINDOWS-1256',
-                                          'CP720', 'CP864',
-                                alphabet=u'ءآأؤإئابةتثجحخدذرزسشصضطظعغػؼؽؾؿـفقكلمنهوىيًٌٍَُِّ',
-                                wiki_start_pages=u'الصفحة_الرئيسية'),
-             'Belarusian': Language(name='Belarusian',
-                                    iso_code='be',
-                                    use_ascii=False,
-                                    charsets='ISO-8859-5', 'WINDOWS-1251',
-                                              'IBM866', 'MacCyrillic',
-                                    alphabet=(u'АБВГДЕЁЖЗІЙКЛМНОПРСТУЎФХЦЧШЫЬЭЮЯ'
-                                              u'абвгдеёжзійклмнопрстуўфхцчшыьэюяʼ'),
-                                    wiki_start_pages=u'Галоўная_старонка'),
-             'Bulgarian': Language(name='Bulgarian',
-                                   iso_code='bg',
-                                   use_ascii=False,
-                                   charsets='ISO-8859-5', 'WINDOWS-1251',
-                                             'IBM855',
-                                   alphabet=(u'АБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЬЮЯ'
-                                             u'абвгдежзийклмнопрстуфхцчшщъьюя'),
-                                   wiki_start_pages=u'Начална_страница'),
-             'Czech': Language(name='Czech',
-                               iso_code='cz',
-                               use_ascii=True,
-                               charsets='ISO-8859-2', 'WINDOWS-1250',
-                               alphabet=u'áčďéěíňóřšťúůýžÁČĎÉĚÍŇÓŘŠŤÚŮÝŽ',
-                               wiki_start_pages=u'Hlavní_strana'),
-             'Danish': Language(name='Danish',
-                                iso_code='da',
-                                use_ascii=True,
-                                charsets='ISO-8859-1', 'ISO-8859-15',
-                                          'WINDOWS-1252',
-                                alphabet=u'æøåÆØÅ',
-                                wiki_start_pages=u'Forside'),
-             'German': Language(name='German',
-                                iso_code='de',
-                                use_ascii=True,
-                                charsets='ISO-8859-1', 'WINDOWS-1252',
-                                alphabet=u'äöüßÄÖÜ',
-                                wiki_start_pages=u'Wikipedia:Hauptseite'),
-             'Greek': Language(name='Greek',
-                               iso_code='el',
-                               use_ascii=False,
-                               charsets='ISO-8859-7', 'WINDOWS-1253',
-                               alphabet=(u'αβγδεζηθικλμνξοπρσςτυφχψωάέήίόύώ'
-                                         u'ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΣΤΥΦΧΨΩΆΈΉΊΌΎΏ'),
-                               wiki_start_pages=u'Πύλη:Κύρια'),
-             'English': Language(name='English',
-                                 iso_code='en',
-                                 use_ascii=True,
-                                 charsets='ISO-8859-1', 'WINDOWS-1252',
-                                 wiki_start_pages=u'Main_Page'),
-             'Esperanto': Language(name='Esperanto',
-                                   iso_code='eo',
-                                   # Q, W, X, and Y not used at all
-                                   use_ascii=False,
-                                   charsets='ISO-8859-3',
-                                   alphabet=(u'abcĉdefgĝhĥijĵklmnoprsŝtuŭvz'
-                                             u'ABCĈDEFGĜHĤIJĴKLMNOPRSŜTUŬVZ'),
-                                   wiki_start_pages=u'Vikipedio:Ĉefpaĝo'),
-             'Spanish': Language(name='Spanish',
-                                 iso_code='es',
-                                 use_ascii=True,
-                                 charsets='ISO-8859-1', 'ISO-8859-15',
-                                           'WINDOWS-1252',
-                                 alphabet=u'ñáéíóúüÑÁÉÍÓÚÜ',
-                                 wiki_start_pages=u'Wikipedia:Portada'),
-             'Estonian': Language(name='Estonian',
-                                  iso_code='et',
-                                  use_ascii=False,
-                                  charsets='ISO-8859-4', 'ISO-8859-13',
-                                            'WINDOWS-1257',
-                                  # C, F, Š, Q, W, X, Y, Z, Ž are only for
-                                  # loanwords
-                                  alphabet=(u'ABDEGHIJKLMNOPRSTUVÕÄÖÜ'
-                                            u'abdeghijklmnoprstuvõäöü'),
-                                  wiki_start_pages=u'Esileht'),
-             'Finnish': Language(name='Finnish',
-                                 iso_code='fi',
-                                 use_ascii=True,
-                                 charsets='ISO-8859-1', 'ISO-8859-15',
-                                           'WINDOWS-1252',
-                                 alphabet=u'ÅÄÖŠŽåäöšž',
-                                 wiki_start_pages=u'Wikipedia:Etusivu'),
-             'French': Language(name='French',
-                                iso_code='fr',
-                                use_ascii=True,
-                                charsets='ISO-8859-1', 'ISO-8859-15',
-                                          'WINDOWS-1252',
-                                alphabet=u'œàâçèéîïùûêŒÀÂÇÈÉÎÏÙÛÊ',
-                                wiki_start_pages=u'Wikipédia:Accueil_principal',
-                                                  u'Bœuf (animal)'),
-             'Hebrew': Language(name='Hebrew',
-                                iso_code='he',
-                                use_ascii=False,
-                                charsets='ISO-8859-8', 'WINDOWS-1255',
-                                alphabet=u'אבגדהוזחטיךכלםמןנסעףפץצקרשתװױײ',
-                                wiki_start_pages=u'עמוד_ראשי'),
-             'Croatian': Language(name='Croatian',
-                                  iso_code='hr',
-                                  # Q, W, X, Y are only used for foreign words.
-                                  use_ascii=False,
-                                  charsets='ISO-8859-2', 'WINDOWS-1250',
-                                  alphabet=(u'abcčćdđefghijklmnoprsštuvzž'
-                                            u'ABCČĆDĐEFGHIJKLMNOPRSŠTUVZŽ'),
-                                  wiki_start_pages=u'Glavna_stranica'),
-             'Hungarian': Language(name='Hungarian',
-                                   iso_code='hu',
-                                   # Q, W, X, Y are only used for foreign words.
-                                   use_ascii=False,
-                                   charsets='ISO-8859-2', 'WINDOWS-1250',
-                                   alphabet=(u'abcdefghijklmnoprstuvzáéíóöőúüű'
-                                             u'ABCDEFGHIJKLMNOPRSTUVZÁÉÍÓÖŐÚÜŰ'),
-                                   wiki_start_pages=u'Kezdőlap'),
-             'Italian': Language(name='Italian',
-                                 iso_code='it',
-                                 use_ascii=True,
-                                 charsets='ISO-8859-1', 'ISO-8859-15',
-                                           'WINDOWS-1252',
-                                 alphabet=u'ÀÈÉÌÒÓÙàèéìòóù',
-                                 wiki_start_pages=u'Pagina_principale'),
-             'Lithuanian': Language(name='Lithuanian',
-                                    iso_code='lt',
-                                    use_ascii=False,
-                                    charsets='ISO-8859-13', 'WINDOWS-1257',
-                                              'ISO-8859-4',
-                                    # Q, W, and X not used at all
-                                    alphabet=(u'AĄBCČDEĘĖFGHIĮYJKLMNOPRSŠTUŲŪVZŽ'

_service:tar_scm:chardet-5.1.0.tar.gz/chardet/py.typed Added

_service:tar_scm:chardet-5.1.0.tar.gz/chardet/resultdict.py Added

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/sbcharsetprober.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/sbcharsetprober.py Changed

@@ -26,70 +26,77 @@
 # 02110-1301  USA
 ######################### END LICENSE BLOCK #########################
 
-from collections import namedtuple
+from typing import Dict, List, NamedTuple, Optional, Union
 
 from .charsetprober import CharSetProber
 from .enums import CharacterCategory, ProbingState, SequenceLikelihood
 
 
-SingleByteCharSetModel = namedtuple('SingleByteCharSetModel',
-                                    'charset_name',
-                                     'language',
-                                     'char_to_order_map',
-                                     'language_model',
-                                     'typical_positive_ratio',
-                                     'keep_ascii_letters',
-                                     'alphabet')
+class SingleByteCharSetModel(NamedTuple):
+    charset_name: str
+    language: str
+    char_to_order_map: Dictint, int
+    language_model: Dictint, Dictint, int
+    typical_positive_ratio: float
+    keep_ascii_letters: bool
+    alphabet: str
 
 
 class SingleByteCharSetProber(CharSetProber):
     SAMPLE_SIZE = 64
-    SB_ENOUGH_REL_THRESHOLD = 1024  #  0.25 * SAMPLE_SIZE^2
+    SB_ENOUGH_REL_THRESHOLD = 1024  # 0.25 * SAMPLE_SIZE^2
     POSITIVE_SHORTCUT_THRESHOLD = 0.95
     NEGATIVE_SHORTCUT_THRESHOLD = 0.05
 
-    def __init__(self, model, reversed=False, name_prober=None):
-        super(SingleByteCharSetProber, self).__init__()
+    def __init__(
+        self,
+        model: SingleByteCharSetModel,
+        is_reversed: bool = False,
+        name_prober: OptionalCharSetProber = None,
+    ) -> None:
+        super().__init__()
         self._model = model
         # TRUE if we need to reverse every pair in the model lookup
-        self._reversed = reversed
+        self._reversed = is_reversed
         # Optional auxiliary prober for name decision
         self._name_prober = name_prober
-        self._last_order = None
-        self._seq_counters = None
-        self._total_seqs = None
-        self._total_char = None
-        self._freq_char = None
+        self._last_order = 255
+        self._seq_counters: Listint = 
+        self._total_seqs = 0
+        self._total_char = 0
+        self._control_char = 0
+        self._freq_char = 0
         self.reset()
 
-    def reset(self):
-        super(SingleByteCharSetProber, self).reset()
+    def reset(self) -> None:
+        super().reset()
         # char order of last character
         self._last_order = 255
         self._seq_counters = 0 * SequenceLikelihood.get_num_categories()
         self._total_seqs = 0
         self._total_char = 0
+        self._control_char = 0
         # characters that fall in our sampling range
         self._freq_char = 0
 
     @property
-    def charset_name(self):
+    def charset_name(self) -> Optionalstr:
         if self._name_prober:
             return self._name_prober.charset_name
-        else:
-            return self._model.charset_name
+        return self._model.charset_name
 
     @property
-    def language(self):
+    def language(self) -> Optionalstr:
         if self._name_prober:
             return self._name_prober.language
-        else:
-            return self._model.language
+        return self._model.language
 
-    def feed(self, byte_str):
+    def feed(self, byte_str: Unionbytes, bytearray) -> ProbingState:
         # TODO: Make filter_international_words keep things in self.alphabet
         if not self._model.keep_ascii_letters:
             byte_str = self.filter_international_words(byte_str)
+        else:
+            byte_str = self.remove_xml_tags(byte_str)
         if not byte_str:
             return self.state
         char_to_order_map = self._model.char_to_order_map
@@ -103,9 +110,6 @@
             #      _total_char purposes.
             if order < CharacterCategory.CONTROL:
                 self._total_char += 1
-            # TODO: Follow uchardet's lead and discount confidence for frequent
-            #       control characters.
-            #       See https://github.com/BYVoid/uchardet/commit/55b4f23971db61
             if order < self.SAMPLE_SIZE:
                 self._freq_char += 1
                 if self._last_order < self.SAMPLE_SIZE:
@@ -122,23 +126,36 @@
             if self._total_seqs > self.SB_ENOUGH_REL_THRESHOLD:
                 confidence = self.get_confidence()
                 if confidence > self.POSITIVE_SHORTCUT_THRESHOLD:
-                    self.logger.debug('%s confidence = %s, we have a winner',
-                                      charset_name, confidence)
+                    self.logger.debug(
+                        "%s confidence = %s, we have a winner", charset_name, confidence
+                    )
                     self._state = ProbingState.FOUND_IT
                 elif confidence < self.NEGATIVE_SHORTCUT_THRESHOLD:
-                    self.logger.debug('%s confidence = %s, below negative '
-                                      'shortcut threshhold %s', charset_name,
-                                      confidence,
-                                      self.NEGATIVE_SHORTCUT_THRESHOLD)
+                    self.logger.debug(
+                        "%s confidence = %s, below negative shortcut threshold %s",
+                        charset_name,
+                        confidence,
+                        self.NEGATIVE_SHORTCUT_THRESHOLD,
+                    )
                     self._state = ProbingState.NOT_ME
 
         return self.state
 
-    def get_confidence(self):
+    def get_confidence(self) -> float:
         r = 0.01
         if self._total_seqs > 0:
-            r = ((1.0 * self._seq_countersSequenceLikelihood.POSITIVE) /
-                 self._total_seqs / self._model.typical_positive_ratio)
+            r = (
+                (
+                    self._seq_countersSequenceLikelihood.POSITIVE
+                    + 0.25 * self._seq_countersSequenceLikelihood.LIKELY
+                )
+                / self._total_seqs
+                / self._model.typical_positive_ratio
+            )
+            # The more control characters (proportionnaly to the size
+            # of the text), the less confident we become in the current
+            # charset.
+            r = r * (self._total_char - self._control_char) / self._total_char
             r = r * self._freq_char / self._total_char
             if r >= 1.0:
                 r = 0.99

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/sbcsgroupprober.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/sbcsgroupprober.py Changed

@@ -28,33 +28,38 @@
 
 from .charsetgroupprober import CharSetGroupProber
 from .hebrewprober import HebrewProber
-from .langbulgarianmodel import (ISO_8859_5_BULGARIAN_MODEL,
-                                 WINDOWS_1251_BULGARIAN_MODEL)
+from .langbulgarianmodel import ISO_8859_5_BULGARIAN_MODEL, WINDOWS_1251_BULGARIAN_MODEL
 from .langgreekmodel import ISO_8859_7_GREEK_MODEL, WINDOWS_1253_GREEK_MODEL
 from .langhebrewmodel import WINDOWS_1255_HEBREW_MODEL
+
 # from .langhungarianmodel import (ISO_8859_2_HUNGARIAN_MODEL,
 #                                  WINDOWS_1250_HUNGARIAN_MODEL)
-from .langrussianmodel import (IBM855_RUSSIAN_MODEL, IBM866_RUSSIAN_MODEL,
-                               ISO_8859_5_RUSSIAN_MODEL, KOI8_R_RUSSIAN_MODEL,
-                               MACCYRILLIC_RUSSIAN_MODEL,
-                               WINDOWS_1251_RUSSIAN_MODEL)
+from .langrussianmodel import (
+    IBM855_RUSSIAN_MODEL,
+    IBM866_RUSSIAN_MODEL,
+    ISO_8859_5_RUSSIAN_MODEL,
+    KOI8_R_RUSSIAN_MODEL,
+    MACCYRILLIC_RUSSIAN_MODEL,
+    WINDOWS_1251_RUSSIAN_MODEL,
+)
 from .langthaimodel import TIS_620_THAI_MODEL
 from .langturkishmodel import ISO_8859_9_TURKISH_MODEL
 from .sbcharsetprober import SingleByteCharSetProber
 
 
 class SBCSGroupProber(CharSetGroupProber):
-    def __init__(self):
-        super(SBCSGroupProber, self).__init__()
+    def __init__(self) -> None:
+        super().__init__()
         hebrew_prober = HebrewProber()
-        logical_hebrew_prober = SingleByteCharSetProber(WINDOWS_1255_HEBREW_MODEL,
-                                                        False, hebrew_prober)
+        logical_hebrew_prober = SingleByteCharSetProber(
+            WINDOWS_1255_HEBREW_MODEL, is_reversed=False, name_prober=hebrew_prober
+        )
         # TODO: See if using ISO-8859-8 Hebrew model works better here, since
         #       it's actually the visual one
-        visual_hebrew_prober = SingleByteCharSetProber(WINDOWS_1255_HEBREW_MODEL,
-                                                       True, hebrew_prober)
-        hebrew_prober.set_model_probers(logical_hebrew_prober,
-                                        visual_hebrew_prober)
+        visual_hebrew_prober = SingleByteCharSetProber(
+            WINDOWS_1255_HEBREW_MODEL, is_reversed=True, name_prober=hebrew_prober
+        )
+        hebrew_prober.set_model_probers(logical_hebrew_prober, visual_hebrew_prober)
         # TODO: ORDER MATTERS HERE. I changed the order vs what was in master
         #       and several tests failed that did not before. Some thought
         #       should be put into the ordering, and we should consider making

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/sjisprober.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/sjisprober.py Changed

@@ -25,68 +25,81 @@
 # 02110-1301  USA
 ######################### END LICENSE BLOCK #########################
 
-from .mbcharsetprober import MultiByteCharSetProber
-from .codingstatemachine import CodingStateMachine
+from typing import Union
+
 from .chardistribution import SJISDistributionAnalysis
+from .codingstatemachine import CodingStateMachine
+from .enums import MachineState, ProbingState
 from .jpcntx import SJISContextAnalysis
+from .mbcharsetprober import MultiByteCharSetProber
 from .mbcssm import SJIS_SM_MODEL
-from .enums import ProbingState, MachineState
 
 
 class SJISProber(MultiByteCharSetProber):
-    def __init__(self):
-        super(SJISProber, self).__init__()
+    def __init__(self) -> None:
+        super().__init__()
         self.coding_sm = CodingStateMachine(SJIS_SM_MODEL)
         self.distribution_analyzer = SJISDistributionAnalysis()
         self.context_analyzer = SJISContextAnalysis()
         self.reset()
 
-    def reset(self):
-        super(SJISProber, self).reset()
+    def reset(self) -> None:
+        super().reset()
         self.context_analyzer.reset()
 
     @property
-    def charset_name(self):
+    def charset_name(self) -> str:
         return self.context_analyzer.charset_name
 
     @property
-    def language(self):
+    def language(self) -> str:
         return "Japanese"
 
-    def feed(self, byte_str):
-        for i in range(len(byte_str)):
-            coding_state = self.coding_sm.next_state(byte_stri)
+    def feed(self, byte_str: Unionbytes, bytearray) -> ProbingState:
+        assert self.coding_sm is not None
+        assert self.distribution_analyzer is not None
+
+        for i, byte in enumerate(byte_str):
+            coding_state = self.coding_sm.next_state(byte)
             if coding_state == MachineState.ERROR:
-                self.logger.debug('%s %s prober hit error at byte %s',
-                                  self.charset_name, self.language, i)
+                self.logger.debug(
+                    "%s %s prober hit error at byte %s",
+                    self.charset_name,
+                    self.language,
+                    i,
+                )
                 self._state = ProbingState.NOT_ME
                 break
-            elif coding_state == MachineState.ITS_ME:
+            if coding_state == MachineState.ITS_ME:
                 self._state = ProbingState.FOUND_IT
                 break
-            elif coding_state == MachineState.START:
+            if coding_state == MachineState.START:
                 char_len = self.coding_sm.get_current_charlen()
                 if i == 0:
-                    self._last_char1 = byte_str0
-                    self.context_analyzer.feed(self._last_char2 - char_len:,
-                                               char_len)
+                    self._last_char1 = byte
+                    self.context_analyzer.feed(
+                        self._last_char2 - char_len :, char_len
+                    )
                     self.distribution_analyzer.feed(self._last_char, char_len)
                 else:
-                    self.context_analyzer.feed(byte_stri + 1 - char_len:i + 3
-                                                        - char_len, char_len)
-                    self.distribution_analyzer.feed(byte_stri - 1:i + 1,
-                                                    char_len)
+                    self.context_analyzer.feed(
+                        byte_stri + 1 - char_len : i + 3 - char_len, char_len
+                    )
+                    self.distribution_analyzer.feed(byte_stri - 1 : i + 1, char_len)
 
         self._last_char0 = byte_str-1
 
         if self.state == ProbingState.DETECTING:
-            if (self.context_analyzer.got_enough_data() and
-               (self.get_confidence() > self.SHORTCUT_THRESHOLD)):
+            if self.context_analyzer.got_enough_data() and (
+                self.get_confidence() > self.SHORTCUT_THRESHOLD
+            ):
                 self._state = ProbingState.FOUND_IT
 
         return self.state
 
-    def get_confidence(self):
+    def get_confidence(self) -> float:
+        assert self.distribution_analyzer is not None
+
         context_conf = self.context_analyzer.get_confidence()
         distrib_conf = self.distribution_analyzer.get_confidence()
         return max(context_conf, distrib_conf)

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/universaldetector.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/universaldetector.py Changed

@@ -39,16 +39,21 @@
 import codecs
 import logging
 import re
+from typing import List, Optional, Union
 
 from .charsetgroupprober import CharSetGroupProber
+from .charsetprober import CharSetProber
 from .enums import InputState, LanguageFilter, ProbingState
 from .escprober import EscCharSetProber
 from .latin1prober import Latin1Prober
+from .macromanprober import MacRomanProber
 from .mbcsgroupprober import MBCSGroupProber
+from .resultdict import ResultDict
 from .sbcsgroupprober import SBCSGroupProber
+from .utf1632prober import UTF1632Prober
 
 
-class UniversalDetector(object):
+class UniversalDetector:
     """
     The ``UniversalDetector`` class underlies the ``chardet.detect`` function
     and coordinates all of the different charset probers.
@@ -66,49 +71,87 @@
     """
 
     MINIMUM_THRESHOLD = 0.20
-    HIGH_BYTE_DETECTOR = re.compile(b'\x80-\xFF')
-    ESC_DETECTOR = re.compile(b'(\033|~{)')
-    WIN_BYTE_DETECTOR = re.compile(b'\x80-\x9F')
-    ISO_WIN_MAP = {'iso-8859-1': 'Windows-1252',
-                   'iso-8859-2': 'Windows-1250',
-                   'iso-8859-5': 'Windows-1251',
-                   'iso-8859-6': 'Windows-1256',
-                   'iso-8859-7': 'Windows-1253',
-                   'iso-8859-8': 'Windows-1255',
-                   'iso-8859-9': 'Windows-1254',
-                   'iso-8859-13': 'Windows-1257'}
-
-    def __init__(self, lang_filter=LanguageFilter.ALL):
-        self._esc_charset_prober = None
-        self._charset_probers = 
-        self.result = None
-        self.done = None
-        self._got_data = None
-        self._input_state = None
-        self._last_char = None
+    HIGH_BYTE_DETECTOR = re.compile(b"\x80-\xFF")
+    ESC_DETECTOR = re.compile(b"(\033|~{)")
+    WIN_BYTE_DETECTOR = re.compile(b"\x80-\x9F")
+    ISO_WIN_MAP = {
+        "iso-8859-1": "Windows-1252",
+        "iso-8859-2": "Windows-1250",
+        "iso-8859-5": "Windows-1251",
+        "iso-8859-6": "Windows-1256",
+        "iso-8859-7": "Windows-1253",
+        "iso-8859-8": "Windows-1255",
+        "iso-8859-9": "Windows-1254",
+        "iso-8859-13": "Windows-1257",
+    }
+    # Based on https://encoding.spec.whatwg.org/#names-and-labels
+    # but altered to match Python names for encodings and remove mappings
+    # that break tests.
+    LEGACY_MAP = {
+        "ascii": "Windows-1252",
+        "iso-8859-1": "Windows-1252",
+        "tis-620": "ISO-8859-11",
+        "iso-8859-9": "Windows-1254",
+        "gb2312": "GB18030",
+        "euc-kr": "CP949",
+        "utf-16le": "UTF-16",
+    }
+
+    def __init__(
+        self,
+        lang_filter: LanguageFilter = LanguageFilter.ALL,
+        should_rename_legacy: bool = False,
+    ) -> None:
+        self._esc_charset_prober: OptionalEscCharSetProber = None
+        self._utf1632_prober: OptionalUTF1632Prober = None
+        self._charset_probers: ListCharSetProber = 
+        self.result: ResultDict = {
+            "encoding": None,
+            "confidence": 0.0,
+            "language": None,
+        }
+        self.done = False
+        self._got_data = False
+        self._input_state = InputState.PURE_ASCII
+        self._last_char = b""
         self.lang_filter = lang_filter
         self.logger = logging.getLogger(__name__)
-        self._has_win_bytes = None
+        self._has_win_bytes = False
+        self.should_rename_legacy = should_rename_legacy
         self.reset()
 
-    def reset(self):
+    @property
+    def input_state(self) -> int:
+        return self._input_state
+
+    @property
+    def has_win_bytes(self) -> bool:
+        return self._has_win_bytes
+
+    @property
+    def charset_probers(self) -> ListCharSetProber:
+        return self._charset_probers
+
+    def reset(self) -> None:
         """
         Reset the UniversalDetector and all of its probers back to their
         initial states.  This is called by ``__init__``, so you only need to
         call this directly in between analyses of different documents.
         """
-        self.result = {'encoding': None, 'confidence': 0.0, 'language': None}
+        self.result = {"encoding": None, "confidence": 0.0, "language": None}
         self.done = False
         self._got_data = False
         self._has_win_bytes = False
         self._input_state = InputState.PURE_ASCII
-        self._last_char = b''
+        self._last_char = b""
         if self._esc_charset_prober:
             self._esc_charset_prober.reset()
+        if self._utf1632_prober:
+            self._utf1632_prober.reset()
         for prober in self._charset_probers:
             prober.reset()
 
-    def feed(self, byte_str):
+    def feed(self, byte_str: Unionbytes, bytearray) -> None:
         """
         Takes a chunk of a document and feeds it through all of the relevant
         charset probers.
@@ -125,7 +168,7 @@
         if self.done:
             return
 
-        if not len(byte_str):
+        if not byte_str:
             return
 
         if not isinstance(byte_str, bytearray):
@@ -136,35 +179,38 @@
             # If the data starts with BOM, we know it is UTF
             if byte_str.startswith(codecs.BOM_UTF8):
                 # EF BB BF  UTF-8 with BOM
-                self.result = {'encoding': "UTF-8-SIG",
-                               'confidence': 1.0,
-                               'language': ''}
-            elif byte_str.startswith((codecs.BOM_UTF32_LE,
-                                      codecs.BOM_UTF32_BE)):
+                self.result = {
+                    "encoding": "UTF-8-SIG",
+                    "confidence": 1.0,
+                    "language": "",
+                }
+            elif byte_str.startswith((codecs.BOM_UTF32_LE, codecs.BOM_UTF32_BE)):
                 # FF FE 00 00  UTF-32, little-endian BOM
                 # 00 00 FE FF  UTF-32, big-endian BOM
-                self.result = {'encoding': "UTF-32",
-                               'confidence': 1.0,
-                               'language': ''}
-            elif byte_str.startswith(b'\xFE\xFF\x00\x00'):
+                self.result = {"encoding": "UTF-32", "confidence": 1.0, "language": ""}
+            elif byte_str.startswith(b"\xFE\xFF\x00\x00"):
                 # FE FF 00 00  UCS-4, unusual octet order BOM (3412)
-                self.result = {'encoding': "X-ISO-10646-UCS-4-3412",
-                               'confidence': 1.0,
-                               'language': ''}
-            elif byte_str.startswith(b'\x00\x00\xFF\xFE'):
+                self.result = {
+                    # TODO: This encoding is not supported by Python. Should remove?
+                    "encoding": "X-ISO-10646-UCS-4-3412",
+                    "confidence": 1.0,
+                    "language": "",
+                }
+            elif byte_str.startswith(b"\x00\x00\xFF\xFE"):
                 # 00 00 FF FE  UCS-4, unusual octet order BOM (2143)
-                self.result = {'encoding': "X-ISO-10646-UCS-4-2143",
-                               'confidence': 1.0,
-                               'language': ''}
+                self.result = {
+                    # TODO: This encoding is not supported by Python. Should remove?
+                    "encoding": "X-ISO-10646-UCS-4-2143",
+                    "confidence": 1.0,
+                    "language": "",
+                }
             elif byte_str.startswith((codecs.BOM_LE, codecs.BOM_BE)):
                 # FF FE  UTF-16, little endian BOM
                 # FE FF  UTF-16, big endian BOM
-                self.result = {'encoding': "UTF-16",
-                               'confidence': 1.0,
-                               'language': ''}
+                self.result = {"encoding": "UTF-16", "confidence": 1.0, "language": ""}
 
             self._got_data = True
-            if self.result'encoding' is not None:

_service:tar_scm:chardet-5.1.0.tar.gz/chardet/utf1632prober.py Added

@@ -0,0 +1,225 @@
+######################## BEGIN LICENSE BLOCK ########################
+#
+# Contributor(s):
+#   Jason Zavaglia
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
+# 02110-1301  USA
+######################### END LICENSE BLOCK #########################
+from typing import List, Union
+
+from .charsetprober import CharSetProber
+from .enums import ProbingState
+
+
+class UTF1632Prober(CharSetProber):
+    """
+    This class simply looks for occurrences of zero bytes, and infers
+    whether the file is UTF16 or UTF32 (low-endian or big-endian)
+    For instance, files looking like ( \0 \0 \0 nonzero )+
+    have a good probability to be UTF32BE.  Files looking like ( \0 nonzero )+
+    may be guessed to be UTF16BE, and inversely for little-endian varieties.
+    """
+
+    # how many logical characters to scan before feeling confident of prediction
+    MIN_CHARS_FOR_DETECTION = 20
+    # a fixed constant ratio of expected zeros or non-zeros in modulo-position.
+    EXPECTED_RATIO = 0.94
+
+    def __init__(self) -> None:
+        super().__init__()
+        self.position = 0
+        self.zeros_at_mod = 0 * 4
+        self.nonzeros_at_mod = 0 * 4
+        self._state = ProbingState.DETECTING
+        self.quad = 0, 0, 0, 0
+        self.invalid_utf16be = False
+        self.invalid_utf16le = False
+        self.invalid_utf32be = False
+        self.invalid_utf32le = False
+        self.first_half_surrogate_pair_detected_16be = False
+        self.first_half_surrogate_pair_detected_16le = False
+        self.reset()
+
+    def reset(self) -> None:
+        super().reset()
+        self.position = 0
+        self.zeros_at_mod = 0 * 4
+        self.nonzeros_at_mod = 0 * 4
+        self._state = ProbingState.DETECTING
+        self.invalid_utf16be = False
+        self.invalid_utf16le = False
+        self.invalid_utf32be = False
+        self.invalid_utf32le = False
+        self.first_half_surrogate_pair_detected_16be = False
+        self.first_half_surrogate_pair_detected_16le = False
+        self.quad = 0, 0, 0, 0
+
+    @property
+    def charset_name(self) -> str:
+        if self.is_likely_utf32be():
+            return "utf-32be"
+        if self.is_likely_utf32le():
+            return "utf-32le"
+        if self.is_likely_utf16be():
+            return "utf-16be"
+        if self.is_likely_utf16le():
+            return "utf-16le"
+        # default to something valid
+        return "utf-16"
+
+    @property
+    def language(self) -> str:
+        return ""
+
+    def approx_32bit_chars(self) -> float:
+        return max(1.0, self.position / 4.0)
+
+    def approx_16bit_chars(self) -> float:
+        return max(1.0, self.position / 2.0)
+
+    def is_likely_utf32be(self) -> bool:
+        approx_chars = self.approx_32bit_chars()
+        return approx_chars >= self.MIN_CHARS_FOR_DETECTION and (
+            self.zeros_at_mod0 / approx_chars > self.EXPECTED_RATIO
+            and self.zeros_at_mod1 / approx_chars > self.EXPECTED_RATIO
+            and self.zeros_at_mod2 / approx_chars > self.EXPECTED_RATIO
+            and self.nonzeros_at_mod3 / approx_chars > self.EXPECTED_RATIO
+            and not self.invalid_utf32be
+        )
+
+    def is_likely_utf32le(self) -> bool:
+        approx_chars = self.approx_32bit_chars()
+        return approx_chars >= self.MIN_CHARS_FOR_DETECTION and (
+            self.nonzeros_at_mod0 / approx_chars > self.EXPECTED_RATIO
+            and self.zeros_at_mod1 / approx_chars > self.EXPECTED_RATIO
+            and self.zeros_at_mod2 / approx_chars > self.EXPECTED_RATIO
+            and self.zeros_at_mod3 / approx_chars > self.EXPECTED_RATIO
+            and not self.invalid_utf32le
+        )
+
+    def is_likely_utf16be(self) -> bool:
+        approx_chars = self.approx_16bit_chars()
+        return approx_chars >= self.MIN_CHARS_FOR_DETECTION and (
+            (self.nonzeros_at_mod1 + self.nonzeros_at_mod3) / approx_chars
+            > self.EXPECTED_RATIO
+            and (self.zeros_at_mod0 + self.zeros_at_mod2) / approx_chars
+            > self.EXPECTED_RATIO
+            and not self.invalid_utf16be
+        )
+
+    def is_likely_utf16le(self) -> bool:
+        approx_chars = self.approx_16bit_chars()
+        return approx_chars >= self.MIN_CHARS_FOR_DETECTION and (
+            (self.nonzeros_at_mod0 + self.nonzeros_at_mod2) / approx_chars
+            > self.EXPECTED_RATIO
+            and (self.zeros_at_mod1 + self.zeros_at_mod3) / approx_chars
+            > self.EXPECTED_RATIO
+            and not self.invalid_utf16le
+        )
+
+    def validate_utf32_characters(self, quad: Listint) -> None:
+        """
+        Validate if the quad of bytes is valid UTF-32.
+
+        UTF-32 is valid in the range 0x00000000 - 0x0010FFFF
+        excluding 0x0000D800 - 0x0000DFFF
+
+        https://en.wikipedia.org/wiki/UTF-32
+        """
+        if (
+            quad0 != 0
+            or quad1 > 0x10
+            or (quad0 == 0 and quad1 == 0 and 0xD8 <= quad2 <= 0xDF)
+        ):
+            self.invalid_utf32be = True
+        if (
+            quad3 != 0
+            or quad2 > 0x10
+            or (quad3 == 0 and quad2 == 0 and 0xD8 <= quad1 <= 0xDF)
+        ):
+            self.invalid_utf32le = True
+
+    def validate_utf16_characters(self, pair: Listint) -> None:
+        """
+        Validate if the pair of bytes is  valid UTF-16.
+
+        UTF-16 is valid in the range 0x0000 - 0xFFFF excluding 0xD800 - 0xFFFF
+        with an exception for surrogate pairs, which must be in the range
+        0xD800-0xDBFF followed by 0xDC00-0xDFFF
+
+        https://en.wikipedia.org/wiki/UTF-16
+        """
+        if not self.first_half_surrogate_pair_detected_16be:
+            if 0xD8 <= pair0 <= 0xDB:
+                self.first_half_surrogate_pair_detected_16be = True
+            elif 0xDC <= pair0 <= 0xDF:
+                self.invalid_utf16be = True
+        else:
+            if 0xDC <= pair0 <= 0xDF:
+                self.first_half_surrogate_pair_detected_16be = False
+            else:
+                self.invalid_utf16be = True
+
+        if not self.first_half_surrogate_pair_detected_16le:
+            if 0xD8 <= pair1 <= 0xDB:
+                self.first_half_surrogate_pair_detected_16le = True
+            elif 0xDC <= pair1 <= 0xDF:
+                self.invalid_utf16le = True
+        else:
+            if 0xDC <= pair1 <= 0xDF:
+                self.first_half_surrogate_pair_detected_16le = False
+            else:
+                self.invalid_utf16le = True
+
+    def feed(self, byte_str: Unionbytes, bytearray) -> ProbingState:
+        for c in byte_str:
+            mod4 = self.position % 4
+            self.quadmod4 = c
+            if mod4 == 3:
+                self.validate_utf32_characters(self.quad)
+                self.validate_utf16_characters(self.quad0:2)
+                self.validate_utf16_characters(self.quad2:4)
+            if c == 0:
+                self.zeros_at_modmod4 += 1
+            else:
+                self.nonzeros_at_modmod4 += 1
+            self.position += 1

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/utf8prober.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/utf8prober.py Changed

@@ -25,45 +25,46 @@
 # 02110-1301  USA
 ######################### END LICENSE BLOCK #########################
 
+from typing import Union
+
 from .charsetprober import CharSetProber
-from .enums import ProbingState, MachineState
 from .codingstatemachine import CodingStateMachine
+from .enums import MachineState, ProbingState
 from .mbcssm import UTF8_SM_MODEL
 
 
-
 class UTF8Prober(CharSetProber):
     ONE_CHAR_PROB = 0.5
 
-    def __init__(self):
-        super(UTF8Prober, self).__init__()
+    def __init__(self) -> None:
+        super().__init__()
         self.coding_sm = CodingStateMachine(UTF8_SM_MODEL)
-        self._num_mb_chars = None
+        self._num_mb_chars = 0
         self.reset()
 
-    def reset(self):
-        super(UTF8Prober, self).reset()
+    def reset(self) -> None:
+        super().reset()
         self.coding_sm.reset()
         self._num_mb_chars = 0
 
     @property
-    def charset_name(self):
+    def charset_name(self) -> str:
         return "utf-8"
 
     @property
-    def language(self):
+    def language(self) -> str:
         return ""
 
-    def feed(self, byte_str):
+    def feed(self, byte_str: Unionbytes, bytearray) -> ProbingState:
         for c in byte_str:
             coding_state = self.coding_sm.next_state(c)
             if coding_state == MachineState.ERROR:
                 self._state = ProbingState.NOT_ME
                 break
-            elif coding_state == MachineState.ITS_ME:
+            if coding_state == MachineState.ITS_ME:
                 self._state = ProbingState.FOUND_IT
                 break
-            elif coding_state == MachineState.START:
+            if coding_state == MachineState.START:
                 if self.coding_sm.get_current_charlen() >= 2:
                     self._num_mb_chars += 1
 
@@ -73,10 +74,9 @@
 
         return self.state
 
-    def get_confidence(self):
+    def get_confidence(self) -> float:
         unlike = 0.99
         if self._num_mb_chars < 6:
-            unlike *= self.ONE_CHAR_PROB ** self._num_mb_chars
+            unlike *= self.ONE_CHAR_PROB**self._num_mb_chars
             return 1.0 - unlike
-        else:
-            return unlike
+        return unlike

_service:tar_scm:chardet-4.0.0.tar.gz/chardet/version.py -> _service:tar_scm:chardet-5.1.0.tar.gz/chardet/version.py Changed

_service:tar_scm:chardet-4.0.0.tar.gz/docs/conf.py -> _service:tar_scm:chardet-5.1.0.tar.gz/docs/conf.py Changed

@@ -1,6 +1,3 @@
-#!/usr/bin/env python3
-# -*- coding: utf-8 -*-
-#
 # chardet documentation build configuration file, created by
 # sphinx-quickstart on Thu Mar 27 00:17:49 2015.
 #
@@ -13,39 +10,40 @@
 # All configuration values have a default; values that are commented out
 # serve to show the default.
 
-import chardet
 import os
 
+import chardet
+
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
-#sys.path.insert(0, os.path.abspath('.'))
+# sys.path.insert(0, os.path.abspath('.'))
 
 # -- General configuration ------------------------------------------------
 
 # If your documentation needs a minimal Sphinx version, state it here.
-#needs_sphinx = '1.0'
+# needs_sphinx = '1.0'
 
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = 'sphinx.ext.autodoc', 'sphinx.ext.viewcode', 'sphinx.ext.mathjax'
+extensions = "sphinx.ext.autodoc", "sphinx.ext.viewcode", "sphinx.ext.mathjax"
 
 # Add any paths that contain templates here, relative to this directory.
-templates_path = '_templates'
+templates_path = "_templates"
 
 # The suffix of source filenames.
-source_suffix = '.rst'
+source_suffix = ".rst"
 
 # The encoding of source files.
-#source_encoding = 'utf-8-sig'
+# source_encoding = 'utf-8-sig'
 
 # The master toctree document.
-master_doc = 'index'
+master_doc = "index"
 
 # General information about the project.
-project = 'chardet'
-copyright = '2015, Mark Pilgrim, Dan Blanchard, Ian Cordasco'
+project = "chardet"
+copyright = "2015, Mark Pilgrim, Dan Blanchard, Ian Cordasco"  #  pylint: disable=redefined-builtin
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
@@ -58,172 +56,176 @@
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
-#language = None
+# language = None
 
 # There are two options for replacing |today|: either, you set today to some
 # non-false value, then it is used:
-#today = ''
+# today = ''
 # Else, today_fmt is used as the format for a strftime call.
-#today_fmt = '%B %d, %Y'
+# today_fmt = '%B %d, %Y'
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
-exclude_patterns = '_build'
+exclude_patterns = "_build"
 
 # The reST default role (used for this markup: `text`) to use for all
 # documents.
-#default_role = None
+# default_role = None
 
 # If true, '()' will be appended to :func: etc. cross-reference text.
-#add_function_parentheses = True
+# add_function_parentheses = True
 
 # If true, the current module name will be prepended to all description
 # unit titles (such as .. function::).
-#add_module_names = True
+# add_module_names = True
 
 # If true, sectionauthor and moduleauthor directives will be shown in the
 # output. They are ignored by default.
-#show_authors = False
+# show_authors = False
 
 # The name of the Pygments (syntax highlighting) style to use.
-pygments_style = 'sphinx'
+pygments_style = "sphinx"
 
 # A list of ignored prefixes for module index sorting.
-#modindex_common_prefix = 
+# modindex_common_prefix = 
 
 # If true, keep warnings as "system message" paragraphs in the built documents.
-#keep_warnings = False
+# keep_warnings = False
 
 
 # -- Options for HTML output ----------------------------------------------
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
-on_rtd = os.environ.get('READTHEDOCS', None) == 'True'
+on_rtd = os.environ.get("READTHEDOCS", None) == "True"
 if not on_rtd:
-    import sphinx_rtd_theme
-    html_theme = 'sphinx_rtd_theme'
+    import sphinx_rtd_theme  # pylint: disable=import-error
+
+    html_theme = "sphinx_rtd_theme"
     html_theme_path = sphinx_rtd_theme.get_html_theme_path()
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
 # documentation.
-#html_theme_options = {}
+# html_theme_options = {}
 
 # Add any paths that contain custom themes here, relative to this directory.
-#html_theme_path = 
+# html_theme_path = 
 
 # The name for this set of Sphinx documents.  If None, it defaults to
 # "<project> v<release> documentation".
-#html_title = None
+# html_title = None
 
 # A shorter title for the navigation bar.  Default is the same as html_title.
-#html_short_title = None
+# html_short_title = None
 
 # The name of an image file (relative to this directory) to place at the top
 # of the sidebar.
-#html_logo = None
+# html_logo = None
 
 # The name of an image file (within the static path) to use as favicon of the
 # docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
 # pixels large.
-#html_favicon = None
+# html_favicon = None
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = '_static'
+html_static_path = "_static"
 
 # Add any extra paths that contain custom files (such as robots.txt or
 # .htaccess) here, relative to this directory. These files are copied
 # directly to the root of the documentation.
-#html_extra_path = 
+# html_extra_path = 
 
 # If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
 # using the given strftime format.
-#html_last_updated_fmt = '%b %d, %Y'
+# html_last_updated_fmt = '%b %d, %Y'
 
 # If true, SmartyPants will be used to convert quotes and dashes to
 # typographically correct entities.
-#html_use_smartypants = True
+# html_use_smartypants = True
 
 # Custom sidebar templates, maps document names to template names.
-#html_sidebars = {}
+# html_sidebars = {}
 
 # Additional templates that should be rendered to pages, maps page names to
 # template names.
-#html_additional_pages = {}
+# html_additional_pages = {}
 
 # If false, no module index is generated.
-#html_domain_indices = True
+# html_domain_indices = True
 
 # If false, no index is generated.
-#html_use_index = True
+# html_use_index = True
 
 # If true, the index is split into individual pages for each letter.
-#html_split_index = False
+# html_split_index = False
 
 # If true, links to the reST sources are added to the pages.
-#html_show_sourcelink = True
+# html_show_sourcelink = True

_service:tar_scm:chardet-4.0.0.tar.gz/docs/faq.rst -> _service:tar_scm:chardet-5.1.0.tar.gz/docs/faq.rst Changed

_service:tar_scm:chardet-4.0.0.tar.gz/docs/usage.rst -> _service:tar_scm:chardet-5.1.0.tar.gz/docs/usage.rst Changed

@@ -17,8 +17,8 @@
 
 .. code:: python
 
-    >>> import urllib
-    >>> rawdata = urllib.urlopen('http://yahoo.co.jp/').read()
+    >>> import urllib.request
+    >>> rawdata = urllib.request.urlopen('http://yahoo.co.jp/').read()
     >>> import chardet
     >>> chardet.detect(rawdata)
     {'encoding': 'EUC-JP', 'confidence': 0.99}
@@ -47,17 +47,17 @@
 
 .. code:: python
 
-    import urllib
+    import urllib.request
     from chardet.universaldetector import UniversalDetector
 
-    usock = urllib.urlopen('http://yahoo.co.jp/')
+    usock = urllib.request.urlopen('http://yahoo.co.jp/')
     detector = UniversalDetector()
     for line in usock.readlines():
         detector.feed(line)
         if detector.done: break
     detector.close()
     usock.close()
-    print detector.result
+    print(detector.result)
 
 .. code:: python
 
@@ -79,10 +79,10 @@
 
     detector = UniversalDetector()
     for filename in glob.glob('*.xml'):
-        print filename.ljust(60),
+        print(filename.ljust(60), end='')
         detector.reset()
-        for line in file(filename, 'rb'):
+        for line in open(filename, 'rb'):
             detector.feed(line)
             if detector.done: break
         detector.close()
-        print detector.result
+        print(detector.result)

_service:tar_scm:chardet-5.1.0.tar.gz/pyproject.toml Added

_service:tar_scm:chardet-4.0.0.tar.gz/setup.cfg -> _service:tar_scm:chardet-5.1.0.tar.gz/setup.cfg Changed

@@ -1,5 +1,47 @@
-bdist_wheel
-universal = 1
+metadata
+name = chardet
+version = attr: chardet.version.__version__
+description = Universal encoding detector for Python 3
+long_description = file: README.rst
+author = Mark Pilgrim
+author_email = mark@diveintomark.org
+maintainer = Daniel Blanchard
+maintainer_email = dan.blanchard@gmail.com
+url = https://github.com/chardet/chardet
+license = LGPL
+keywords = encoding, i18n, xml
+project_urls = 
+	Documentation = https://chardet.readthedocs.io/
+	GitHub Project = https://github.com/chardet/chardet
+	Issue Tracker = https://github.com/chardet/chardet/issues
+classifiers = 
+	Development Status :: 5 - Production/Stable
+	Intended Audience :: Developers
+	License :: OSI Approved :: GNU Lesser General Public License v2 or later (LGPLv2+)
+	Operating System :: OS Independent
+	Programming Language :: Python
+	Programming Language :: Python :: 3
+	Programming Language :: Python :: 3.7
+	Programming Language :: Python :: 3.8
+	Programming Language :: Python :: 3.9
+	Programming Language :: Python :: 3.10
+	Programming Language :: Python :: 3.11
+	Programming Language :: Python :: Implementation :: CPython
+	Programming Language :: Python :: Implementation :: PyPy
+	Topic :: Software Development :: Libraries :: Python Modules
+	Topic :: Text Processing :: Linguistic
+
+options
+include_package_data = true
+python_requires = >=3.7
+packages = find:
+
+options.package_data
+chardet = py.typed
+
+options.entry_points
+console_scripts = 
+	chardetect = chardet.cli.chardetect:main
 
 tool:pytest
 addopts = -v

_service:tar_scm:chardet-4.0.0.tar.gz/test.py -> _service:tar_scm:chardet-5.1.0.tar.gz/test.py Changed

@@ -5,46 +5,56 @@
 :author: Ian Cordasco
 """
 
-from __future__ import with_statement
 
 import textwrap
 from difflib import ndiff
-from io import open
 from os import listdir
 from os.path import dirname, isdir, join, realpath, relpath, splitext
+from pprint import pformat
+from unicodedata import normalize
 
 try:
     import hypothesis.strategies as st
-    from hypothesis import given, assume, settings, Verbosity
+    from hypothesis import Verbosity, assume, given, settings
+
     HAVE_HYPOTHESIS = True
 except ImportError:
     HAVE_HYPOTHESIS = False
-import pytest
+import pytest  # pylint: disable=import-error
 
 import chardet
-
+from chardet.metadata.languages import LANGUAGES
 
 # TODO: Restore Hungarian encodings (iso-8859-2 and windows-1250) after we
 #       retrain model.
-MISSING_ENCODINGS = {'iso-8859-2', 'iso-8859-6', 'windows-1250',
-                     'windows-1254', 'windows-1256'}
-EXPECTED_FAILURES = {'tests/iso-8859-7-greek/disabled.gr.xml',
-                     'tests/iso-8859-9-turkish/divxplanet.com.xml',
-                     'tests/iso-8859-9-turkish/subtitle.srt',
-                     'tests/iso-8859-9-turkish/wikitop_tr_ISO-8859-9.txt'}
+MISSING_ENCODINGS = {
+    "iso-8859-2",
+    "iso-8859-6",
+    "windows-1250",
+    "windows-1254",
+    "windows-1256",
+}
+EXPECTED_FAILURES = {
+    "tests/iso-8859-9-turkish/_ude_1.txt",
+    "tests/iso-8859-9-turkish/_ude_2.txt",
+    "tests/iso-8859-9-turkish/divxplanet.com.xml",
+    "tests/iso-8859-9-turkish/subtitle.srt",
+    "tests/iso-8859-9-turkish/wikitop_tr_ISO-8859-9.txt",
+}
+
 
 def gen_test_params():
     """Yields tuples of paths and encodings to use for test_encoding_detection"""
-    base_path = relpath(join(dirname(realpath(__file__)), 'tests'))
+    base_path = relpath(join(dirname(realpath(__file__)), "tests"))
     for encoding in listdir(base_path):
         path = join(base_path, encoding)
         # Skip files in tests directory
         if not isdir(path):
             continue
-        # Remove language suffixes from encoding if pressent
+        # Remove language suffixes from encoding if present
         encoding = encoding.lower()
-        for postfix in '-arabic', '-bulgarian', '-cyrillic', '-greek',
-                        '-hebrew', '-hungarian', '-turkish':
+        for language in sorted(LANGUAGES.keys()):
+            postfix = "-" + language.lower()
             if encoding.endswith(postfix):
                 encoding = encoding.rpartition(postfix)0
                 break
@@ -54,7 +64,7 @@
         # Test encoding detection for each file we have of encoding for
         for file_name in listdir(path):
             ext = splitext(file_name)1.lower()
-            if ext not in '.html', '.txt', '.xml', '.srt':
+            if ext not in ".html", ".txt", ".xml", ".srt":
                 continue
             full_path = join(path, file_name)
             test_case = full_path, encoding
@@ -63,77 +73,157 @@
             yield test_case
 
 
-@pytest.mark.parametrize ('file_name, encoding', gen_test_params())
+@pytest.mark.parametrize("file_name, encoding", gen_test_params())
 def test_encoding_detection(file_name, encoding):
-    with open(file_name, 'rb') as f:
+    with open(file_name, "rb") as f:
         input_bytes = f.read()
         result = chardet.detect(input_bytes)
         try:
             expected_unicode = input_bytes.decode(encoding)
         except LookupError:
-            expected_unicode = ''
+            expected_unicode = ""
         try:
-            detected_unicode = input_bytes.decode(result'encoding')
+            detected_unicode = input_bytes.decode(result"encoding")
         except (LookupError, UnicodeDecodeError, TypeError):
-            detected_unicode = ''
+            detected_unicode = ""
     if result:
-        encoding_match = (result'encoding' or '').lower() == encoding
+        encoding_match = (result"encoding" or "").lower() == encoding
     else:
         encoding_match = False
     # Only care about mismatches that would actually result in different
     # behavior when decoding
+    expected_unicode = normalize("NFKC", expected_unicode)
+    detected_unicode = normalize("NFKC", detected_unicode)
     if not encoding_match and expected_unicode != detected_unicode:
-        wrapped_expected = '\n'.join(textwrap.wrap(expected_unicode, 100)) + '\n'
-        wrapped_detected = '\n'.join(textwrap.wrap(detected_unicode, 100)) + '\n'
-        diff = ''.join(ndiff(wrapped_expected.splitlines(True),
-                             wrapped_detected.splitlines(True)))
+        wrapped_expected = "\n".join(textwrap.wrap(expected_unicode, 100)) + "\n"
+        wrapped_detected = "\n".join(textwrap.wrap(detected_unicode, 100)) + "\n"
+        diff = "".join(
+            
+                line
+                for line in ndiff(
+                    wrapped_expected.splitlines(True), wrapped_detected.splitlines(True)
+                )
+                if not line.startswith(" ")
+            :20
+        )
+        all_encodings = chardet.detect_all(input_bytes, ignore_threshold=True)
     else:
-        diff = ''
+        diff = ""
         encoding_match = True
-    assert encoding_match, ("Expected %s, but got %s for %s.  Character "
-                            "differences: \n%s" % (encoding,
-                                                   result,
-                                                   file_name,
-                                                   diff))
+        all_encodings = result
+    assert encoding_match, (
+        f"Expected {encoding}, but got {result} for {file_name}.  First 20 "
+        f"lines with character differences: \n{diff}\n"
+        f"All encodings: {pformat(all_encodings)}"
+    )
+
+
+@pytest.mark.parametrize("file_name, encoding", gen_test_params())
+def test_encoding_detection_rename_legacy(file_name, encoding):
+    with open(file_name, "rb") as f:
+        input_bytes = f.read()
+        result = chardet.detect(input_bytes, should_rename_legacy=True)
+        try:
+            expected_unicode = input_bytes.decode(encoding)
+        except LookupError:
+            expected_unicode = ""
+        try:
+            detected_unicode = input_bytes.decode(result"encoding")
+        except (LookupError, UnicodeDecodeError, TypeError):
+            detected_unicode = ""
+    if result:
+        encoding_match = (result"encoding" or "").lower() == encoding
+    else:
+        encoding_match = False
+    # Only care about mismatches that would actually result in different
+    # behavior when decoding
+    expected_unicode = normalize("NFKD", expected_unicode)
+    detected_unicode = normalize("NFKD", detected_unicode)
+    if not encoding_match and expected_unicode != detected_unicode:
+        wrapped_expected = "\n".join(textwrap.wrap(expected_unicode, 100)) + "\n"
+        wrapped_detected = "\n".join(textwrap.wrap(detected_unicode, 100)) + "\n"
+        diff = "".join(
+            
+                line
+                for line in ndiff(
+                    wrapped_expected.splitlines(True), wrapped_detected.splitlines(True)
+                )
+                if not line.startswith(" ")
+            :20
+        )
+        all_encodings = chardet.detect_all(
+            input_bytes, ignore_threshold=True, should_rename_legacy=True
+        )
+    else:
+        diff = ""
+        encoding_match = True
+        all_encodings = result
+    assert encoding_match, (
+        f"Expected {encoding}, but got {result} for {file_name}.  First 20 "
+        f"lines of character differences: \n{diff}\n"
+        f"All encodings: {pformat(all_encodings)}"
+    )
 
 
 if HAVE_HYPOTHESIS:
+
     class JustALengthIssue(Exception):
         pass
 
-
     @pytest.mark.xfail

_service:tar_scm:chardet-5.1.0.tar.gz/tests/Johab Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/Johab/hlpro-readme.txt Added

@@ -0,0 +1,33 @@
+      To read this document, use common combined Hangul code or KSSM code.
+
+                              <<< ÐeaÏa¡ 3.0 >>>
+
+                                   ¹A¸b ·±·¥å
+
+                     Copyright (C) 1990, 1997 by Inkeon Lim
+                               All rights reserved
+
+
+o ¸á¸b¥ Åe ·A· ¬aÐw
+  * ¥¥ CDµA Í¡ÐqE ¡¡e ÐeaÏa¡ Åe Ìa·©i· ¸á¸b¥·e ¹A¸b¸a· ¡·A·³¡a.
+  * ¥¥ CDµA Í¡ÐqE ¡¡e ÐeaÏa¡ Åe Ìa·©i·e ¹A¸b¸a· Ðáa ´ô·¡ ´ááÐe
+    ÑwÈ¡e ·©¦ ¡e ¸åÁAi ¥¢¬aÐaµa Èa·¥µAA ¤Í¡Ði ® ´ô¯s¡a.
+    (µ: É·¯¥ wµA · ¦a, Èa a·¡§aá¡a Ïa¡aµA Í¡Ðq ¦a)
+  * ¬a¶w¸ae ÐeaÏa¡¡ ÄñÌa·©, ·ÇaE ¯©Ð Ìa·©(.EXE Ìa·©) £¹ ¯©ÐµA Ï©®¸â·a¡
+    ¶aAe ¦¢ Ìa·©(i© Ìa·© w) e·i ¸a·A³A ¤Í¡Ði ® ·¶·a¡a, a w¶µA
+    ÐeÐaµa ÐeaÏa¡ ·w¶w Ïa¡a·i ¬w´ó¸â·¥ ¶w¡¡¡ ¤Í¡Ði ® ·¶¯s¡a.
+  * ¡¯a Å¡aµA ÅÐe ¬aÐw·e hlprosrc.doci Àq¡Ða¡ ¤as¡a.
+
+o ÐeaÏa¡e ¡¯a¶w Ðei a·¡§aá¡·³¡a. ¬a¶w¸aa ¶¥Ðae aÏ¢ ¬a¶w¸a
+  ·¥ÈáÍA·¡¯a(GUI)i ¹A¸bÐa¡ ¶áÐe ¡i ¹A·Ðs¡a.
+
+o ÐeaÏa¡· ¡¡e ¢¬á Ìa·©(.DOC Ìa·©)·e ¬w¶w ¹¡ÐsÑw Å¡a(KSSM Å¡a)¡ ¸b¬÷A´á
+  ·¶¯s¡a.
+
+o ÐeaÏa¡· ¬éÃ¡ ¤w¤ó·e install.doc Ìa·©µA ¬é¡wA´á ·¶·a¡a, ¯©¹A ¬éÃ¡e ¤Ã¡
+  Ìa·©·¥ install.bati ¯©Ð¯¡Åa ®ÐÐs¡a. ¬éÃ¡Ða¡ ¸åµA ¤ea¯¡ install.doc
+  Ìa·©·i ·ª´á¥¡¡ ÐeaÏa¡i ¬éÃ¡Ða¡ ¤as¡a.
+
+o 286 Å¡a¡ ÂA¸âÑÁ¯¡Åa BC++ 3.0·a¡ ÄñÌa·©E a·¡§aá¡ Ìa·©i·¡ HLIB\BCLIB
+  ¡BÉ¡¡µA ¹A·S¡a. BC++ 3.0 ·¡¬w ¬a¶w¸ae ¬a¶w¸a ´e¬áµA ´åsE HLIB\LIB
+  ¯¥µA HLIB\BCLIB ¡BÉ¡¡i ÄñÌa·©á µ³eµA ¬é¸÷Ða¡ ¤as¡a.

_service:tar_scm:chardet-5.1.0.tar.gz/tests/Johab/iyagi-readme.txt Added

@@ -0,0 +1,456 @@
+===============================================================================
+                            ·¡´¡¡  éË¡ for Windows
+                                Çe¬aqÄñÏAÈá(º)
+                                   2001e 04¶©
+===============================================================================
+README Ìa·©·e ¬é¡w¬áa ¡¶ iµA Í¡ÐqA»¡ ´g·e ÂA¯¥ ¸÷¥¡i ¹A·Ðs¡a.
+
+----------------
+¬é¡w¬á ¬a¶w ¤w¤ó
+----------------
+
+¡A¡¡¸wµA¬á ·¡ ¢¬ái ¸i ¥¡a¡e Àw·i ÂAÑÁÐe ÒµA ¸a· º ¤aµA ÁAÇaÐa¯³¯¡µ¡.
+·¡ ¢¬ái ·¥áÐaa¡e Ìa·© ¡AAµA¬á ·¥ái ¬åÈÐa¯³¯¡µ¡.
+
+Àq¡ : ·¡ ¢¬á· ·©¦ ¶w·e ·¥ÈáU·¡´¡¡ 2000µA¬á ¸â¶wE ¡w·³¡a.
+
+----
+¡¢Àa
+----
+1. ¬éÃ¡Ða¡ ¸åµA
+2. ¬éÃ¡ e ¬é¡w
+3. Âa ¸÷¥¡
+
+----------------
+1. ¬éÃ¡Ða¡ ¸åµA
+----------------
+"·¡´¡¡ éË¡" ¹AÎ·i ¬éÃ¡Ða¡ ¶áÐ¬áe Windows 98 ¡e MEa ¬éÃ¡A´á ·¶e ¯¡¯aÉQ
+·¡  ·¶´á´¡  Ðs¡a. ¬éÃ¡i ¶áÐ Ï©¶aÐe Ðaa¡¯aÇa· ·e·e ¬éÃ¡¯¡ ¬åÈÐe µ³eµA 
+aa aa¡a, É·¯¥µA µeiÐaáa ¬a¶a w·i ·¡¶wÐaa¡e Âa ¬a´··¡ Ï©¶aÐs¡a.
+
+    * ·¥ÈáU ¡e PCÉ·¯¥ ¡w - ¡¡Q ¡e AËa¶¡Ça ·¥ÈáÍA·¡¯a ¸wÃ¡
+    * ·q¬÷ £¹ ¬a¶a ¡w - ¬a¶a ÄaaµÁ ¯aÏ¡Äá,  a·¡Ça
+    * CD-ROM ¬éÃ¡ - 2¤¢ ·¡¬w· CD-ROM aa·¡§a
+
+
+ÂA¡Ðe· ¯¡¯aÉQ ¶a ¬a´·
+-------------------------
+"·¡´¡¡ éË¡" ¹AÎ·e Windows 95/98/ME, Windows NT 4.0µA¬á ¯©ÐS¡a.
+
+    * 486DX, 66MHz ·¡¬w· Ïa¡A¬á 
+    * 24MB RAM(¡A¡¡¡a  g·i®¢ ¬÷w·¡ Ð·¬wQ)
+    * 10MB HDD(¬éÃ¡ µ³eµA aa a ·¡¬w·¡ Ï©¶aÐi ® ·¶¯s¡a.)
+    * VGA ·¡¬w· Ð¬w¡i a»¡e ¡¡¡Èá
+    *  a¶¯a ¡e Ñ¡ÑÅAe Í¡·¥Ë· ¸wÃ¡
+
+    ¶µwÁA¹Aa Windows 95·¥ w¶
+        * 486DX ·¡¬w· Ïa¡A¬á
+        * 8MB RAM ·¡¬w· ¡A¡¡¡
+
+µ¥a·¥·i É·Ð ¹A·(a¶¡a)Ae ¤á¸å·i ¬éÃ¡ Ða¡ ¶áÐ¬áe ¹A·E Ïa¡a· ´s
+Â·i Ð¹AÐa¡ ¶áÐe ´sÂ Ð¹A Ïa¡a·¡ Ï©¶aÐi ® ·¶¯s¡a.(µ¥a·¥ ·¸aa¯©µA
+¬á a¶¡a Ða¯© ® ·¶¯s¡a.)
+
+----------------
+2.¬éÃ¡ e ¬é¡w
+----------------
+·¥¯aÉ© Ïa¡a·i ¯©ÐÐe Ò· Á¸÷·i e¢Ó¡ ¡Ðs¡a.
+
+    ¬a¶w¥ ´¢ ÑÁ¡e
+        ¥¥ ¹AÎ· ¬éÃ¡ £¹ ¬a¶w wµA Ðe ¶w·³¡a.
+
+    ¬a¶w¸a ¸÷¥¡ ÑÁ¡e
+        ·³beµA ¬a¶w¸a ·¡q·i ·³bÐs¡a.
+
+    ¬éÃ¡ ¤w¤ó ÑÁ¡e
+        ¶¥Ðae ¬éÃ¡ ¤w¤ó·i ¬åÈÐs¡a.
+        ¬éÃ¡Ði Í©ái »¡¸÷Ði ® ·¶¯s¡a.
+        * ¬åÈ ¬éÃ¡i ¬åÈÐaµa ¬éÃ¡Ðw¡¢·i ¹¡¸÷Ði ® ·¶¯s¡a.(Ñe¸ ¦a)
+
+    ¬á§¡¯a ¬é¸÷ ÑÁ¡e
+        PCÉ·¯¥ ¬á§¡¯a· ´a·¡¡µÁ §¡£©¤åÑ¡i ·³bÐ  ® ·¶¯s¡a.
+
+    µei ¬é¸÷ ÑÁ¡e
+        É·¯¥µA µeiÐae ¤w¤óÁ Ï©¶aÐe ¬é¸÷·i Ði ® ·¶¯s¡a.
+        * ISDN ¸wÃ¡ ¬a¶w¸ae Âa ¬é¸÷·i Ði ® ·¶¯s¡a.
+        *  ¤w¯¢ ¡¡ ¬a¶w·¥ w¶ ¡¡Q·i ¶¬å·a¡ Ðs¡a.
+        * ·i ¶¬å ¬a¶wÐaa¡e ·¥ÈáU ¬a¶w¯¡ PPP ¸ó¢·i ¬åÈÐa»¡  a¯³¯¡µ¡.
+
+    ¬éÃ¡ µ³e »¡¸÷ ÑÁ¡e
+        ·¡´¡¡a ¬¬÷ ·¡¶AµA ·¡´¡¡ ¯©Ð´a·¡Å¥·i w¢Ði ® ·¶¯s¡a.
+        * ¨ae¯©ÐµA w¢·e IE 4.0 ·¡¬w·¡ ¬éÃ¡A´á ·¶´á´¡ Ðs¡a.
+        * ·¡´¡¡¶¡ai ¶å¡¶ ¡¥¥Íe»³¡¡ »¡¸÷Ði ® ·¶¯s¡a.
+        * MP3 Ìa·©·i ·¡´¡¡ MP3 µeº¡µA µeiÐi ® ·¶¯s¡a.
+
+    ¬éÃ¡ ´e ÑÁ¡e  
+        ¡¯aÇaµA ¬éÃ¡ Ìa·©·i ¥¢¬aÐs¡a.
+    ¬éÃ¡ ÒµA ·¡´¡¡i ¯©ÐÐaáa ¥¥ ¶w·i ¥© ® ·¶¯s¡a.
+
+
+------------
+3. Âa ¸÷¥¡
+------------
+
+ÑÅw ¬é¸÷(¤aq¸s·¡)
+-------------------
+"·¡´¡¡ éË¡" ¹AÎ· ÑÅw·e '¤aq¸s·¡'i É·Ð ¬é¸÷Ði ® ·¶·a¡a, ¤aq¸s·¡e ¡A
+AµA¬á ¬é¸÷-ÑÅw¬é¸÷ Ðw¡¢·i ¬åÈÐaáa Alt+1Ç¡i q - 'Alt'Ç¡i e
+¬wÈµA¬á ®¸a '1'Ç¡i q - ·a¡°á ¯©ÐÐi ® ·¶¯s¡a.
+
+·©¦ ÑÅw·e Ëb¸÷Ðe ¬á§¡¯aa ¸åÑÅAe ·±¯¡¸â·¥ ¬a¶wÑÅw·a¡ ¤aq¸s·¡µA¬á ¬é¸÷
+Ðae õ·¡ ´a¡a ¥¡¡ ¡AAa ¬é¸÷-¸åÑÅ ¡AAµA¬á »¡¸÷Ð´¡ Ðs¡a.
+
+¤aq¸s·¡· ¡Èa-¡sµA '¸ó¢¯¡ Ñe¸ Èá£¡é ¬a¶w'µA ÁAÇa Îa¯¡i Ða¡e ¨ae
+µei·¡a ÀéµA ·¶e ¸ó¢¤åÑ¡¡ ¬á§¡¯aµA µeiÐi , ¤aq¸s·¡· ÑÁ¡eµA ¬é¸÷
+Ðe 'Èá£¡é ¹·A'i ¬a¶wÐs¡a. ¥i¸â·a¡ ¬é¸÷Ðe Èá£¡é·i ¬a¶wÐa»¡ ´g¯s¡a.
+
+¤aq¸s·¡· ÑÁ¡e-¡sµA¬á '¡´âÐi ÑÁ¡e ®'i ÇaAÐa¡e Ði ®¢ ·¡´¡¡a ¯©
+ÐI  ¥¡a  g·e ¡A¡¡¡i ¬a¶wÐaA S¡a. µaá Àw·i ¶¡ ¶å¡¶i ¬a¶wÐa
+¯© µAe awÐe ¸âA ¬á¸÷Ðae õ·¡ ·A¡Ðs¡a.
+
+
+·¥ÈáU ¸ó¢Á Àé
+------------------
+É·¯¥ µei¬é¸÷Á µei ·A¢µA aa ·¥ÈáU eÂi ö·i  ·¸b·¡ as¡a. É·
+¯¥µA µeiA»¡ ´g·e ¬wÈµA¬á ¸åÑÁ¬åÁ ¡¡Q/ISDN·i É·Ð µeiÐae ÑÅw·¡a¡e ·¡
+´¡¡e ·¥Í¡¬³·i É·Ð ·¥ÈáUµA µei·i ¯¡¡Ðs¡a. ·¡µAe Ðw ¬á§¡¯aa ·Aa
+¬á§¡¯a·±·i ´eÐae Àw·¡ s¡a.
+
+Àå¡´e/Ða·¡ÉI/a¶¡µÁ {·e É·¯¥ ¬á§¡¯aµA µeiE ¬wÈµA¬á ·¥ÈáU eÂi 
+a¡e ·¡´¡¡e Ðw ¬á§¡¯aµA¬á ¹A·Ðae ·¥ÈáU ¸ó¢ ¡AAi É·Ð ·¥ÈáUµA µei
+·i ¯¡¡Ða¡a, ¬á§¡¯a ¶aqµA Ðe ´ee Ða»¡ ´g¯s¡a.
+
+·¡´¡¡ Àé·i É·Ð ¹A·Ae ·©¦ ¬á§¡¯ae ·Aa¸÷¥¡ ¬á§¡¯a·³¡a. ·Aa¸÷¥¡ ¬á
+§¡¯a·¥ w¶µAe ¬á§¡¯a ¶aq·i ´eÐae ÑÁ¡ea»¡ e µei·i ¯¡¡Ða¡a, ¬á§¡¯a ¬a
+¶wµA Ðe i¸÷·e ¬a¶w¸aa »¢¸ó 'Y' ¡e 'N'i ·³bÐ´¡ Ðs¡a.
+
+
+¡ÏaËa¡¡Q(Ñ¢·e ¶å ¡¡Q) ¬a¶w
+-----------------------------
+¡ÏaËa¡¡Q Ñ¢·e ¶å¡¡Q·¡a ¦¡¶e ¡¡Q·e ·©¤e ¡¡QÁ i¡ A·¡Èá Àá¡i ¡¡
+Q·¡  ´a¥ ÄñÏAÈáa Àá¡Ða¡¢ ¤iE ¡¡Q·a¡ ¡ÏaËa ¡¡Q·i ·¡¶wÐae w¶µA
+e Àå¡´e/Ða·¡ÉI/a¶¡ wµA¬á ¹A·Ðae ¡AA PPP ¬á§¡¯aa »¡¶¥A»¡ ´g·i ® 
+·¶¯s¡a.
+
+¹A¹¡´óÁAa  ·sÀái É·Ð ¥¡a ¸÷ÑÂÐa¡ ¬wAÐe ¸÷¥¡i ´è·a¯© ® ·¶·a¡a, ·¡´¡
+¡i  É·Ðe ·¥ÈáU µeiµA¬á¡ Àá·q µei¦Èá PPP¡ µei·i ¯¡¡Ðae w¶µA e ·¥
+ÈáU ¬á§¡¯ai ·¡¶wÐi ® ·¶¯s¡a.
+
+·©¦ ¡ÏaËa¡¡QµA¬á µei·i ¹·aÐe ÒµA¡ µei ¬wÈ¡ Îa¯¡Ae w¶, ¡AAµA¬á
+¬é¸÷-ÑÅw ¬é¸÷-É·¯¥ ¬é¸÷·i Àa¡ ¬åÈÐe ÒµA ¤aq¸s·¡· É·¯¥ Ðw¡¢µA
+¬á '¡ÏaËa ¡¡Q'µA ÁAÇa Îa¯¡i Ða¯³¯¡µ¡.
+
+
+Òé(Wheel)  a¶¯a
+----------------
+Ñe¸ Òé  a¶¯aµA¬á ¹A·Ðae ¦a ¡w·e É·¯¥ÀwÁ ·¡´¡¡¶¡aµA¬á e »¡¶¥S¡a.
+Àéµwµb·e Òé  a¶¯ai »¡¶¥Ða¡ ´áa¶ ¹¡a »¡¶¥A»¡ ´g¯s¡a.
+
+ÀéµwµbµA¬á Òé  a¶¯ai ¬a¶wÐaµa ¯aÇa©Ðae w¶, É·¯¥Àw·¡ ¯aÇa© S¡a.
+
+
+Ëb®Ðe Ç¡ ·¸b
+--------------
+·©¤e PCÉ·¯¥ ¬á§¡¯ai ·¡¶wÐae w¶, ·¥¬áËaÇ¡i a¡e ·¥¯¢E ¬á§¡¯aa ¡ww´á
+i ®Àa¸â·a¡ ¤e¸å¯¡Ç¡¡, ·¥¯¢E ¬á§¡¯aa ¡ww´áe 'Alt+Insert'i q·a¡°á
+·³bÐi ® ·¶¯s¡a.
+
+Å¥Ëa©(CTRL)Ç¡i e ¬wÈµA¬á ¤wÐ·Ç¡i a¡e ·¥¯¢E ¬á§¡¯aa ¡ww´ái ·¡·
+Ði ® ·¶¯s¡a. ·¥¯¢E ¬á§¡¯aa ¡ww´áe 'CTRL+Enter'i q·a¡°á ·³bÐi ®
+·¶¯s¡a.
+
+¡Ðe 'CTRL+Home'Ç¡e ·¥¯¢E ¡ww·¡a ¬á§¡¯a· Àá·q·i ¤e¸å¯¡Ç¡¡ 'CTRL+End'e
+ a»¡ b·i ¤e¸å ·¡·¯¡Ç³¡a. 
+
+eeÐe ¡¶ i¡ ·¡·Ðaa¡e 'Ctrl+Shift+F1'Ç¡i a¯³¯¡µ¡. ·¡´¡¡i Àá·q¯©Ð
+Ðaµa 'eeÐe ¡¶ i'·¡ e w¶µA¡ ·¡ Ç¡i a¡e ·¡·Ði ® ·¶¯s¡a.  
+
+Àq¡¡ '¡¡ 98'·i ·¡¶wÐae w¶, ·¡ ¡w·i ·¡¶wÐa¡e ¥¡a É·¯¥ ¬á§¡¯ai ¨a
+aA ·¡¶wÐi ® ·¶¯s¡a.
+
+
+ISDN ¸wÃ¡ Åe
+--------------
+·©¦  ISDN ¸wÃ¡·i ·¡¶wÐaµa ·¥ÈáU·¡a PCÉ·¯¥ ¬á§¡¯aµA µei·i ¯¡¡Ðae w¶µA 
+¸ó¢·¡ ¸i A»¡ ´ge w¶a ·¶¯s¡a. ·¡ w¶µAe ÂA¡¢¡i 115200BPS¡ ¥ew
+Ð ¬a¶wÐa¯¡¡e Ði I ®¡ ·¶¯s¡a.
+
+Ñw  ISDN  ¸wÃ¡· w¶µA É·¯¥ µei¸åµA ¥i¡· aa·¡¤ái ¶¡´¡ Ðae w¶µA 
+¶å¡¶· ¡¡Q w¢¸÷¥¡µA Ðw ¸wÃ¡a ¤ieA»¡ ´gea¡e ·¡´¡¡µA¬áe µei¸wÃ¡
+¡  ¬a¶wÐi ® ´ô¯s¡a.(e, ·¡´¡¡i ¡¡Q¸ó¢ ·¡¶A· ¡w·i ·¡¶wÐa¡¸a Ði w
+¶µAe ¬a¶wÐa¯© ® ·¶¯s¡a.)
+
+
+¶Ó §aa¶¸á· ·¡·Ç¡ Åe
+-------------------------
+"·¡´¡¡ éË¡" ¹AÎ·e PC¶w µA£IA·¡Èái »¥¥¡E ¡®·i É·Ð ¶Ó ¡wa»¡ µA£IA
+·¡e  Ðº¡¢  ¤iE Ïa¡a·³¡a. §aa¶¸áµAe ¬a¶w¸aa i®A(Ç¡¥¡a, ¸a
+Ìe)i  ¸÷·Ð ºe ¡w·¡ ´ôáa ¹AÐe¸â·¡»¡ e µA£IA·¡ÈáµAe ¡¥¥¸â·a¡ »¡¶¥ 
+Ðs¡a. a¸w ¢¹Aa Ae õ·e 'CTRL'Ç¡¡ µA£IA·¡ÈáµA¬áe ¯©¹A É·¯¥µA¬á ¬a¶w
+Ða¡  ¢µA µa¡µA ·©¤e ¡w·i ¸÷·Ð ¡e Ï©¶aÐe ¸÷¬w¸â·¥ É·¯¥·i Ða»¡ ¡µÐi 
+®¡ ·¶¯s¡a.
+
+
+·¡´¡¡ aw Åe
+---------------
+·¡´¡¡ aw·e  É·¯¥ÀwµA¬á ¬a¶wÐi ® ·¶e e·¡ Íe»³¡·³¡a. ·¡´¡¡¯aÇ¥·i ¬a¶w
+Ðae w¶µA ·¡´¡¡ aw· Äá¬áa ¸i ¥¡·¡»¡ ´g·i ® ·¶¯s¡a.
+
+·¡´¡¡ awµA¬á Ëb¸÷Ðe i¸ai ·³bÐae w¶, ÂÀåÐae ¢¸w·¡ aÈai ® ·¶¯s¡

_service:tar_scm:chardet-5.1.0.tar.gz/tests/Johab/mdir-doc.txt Added

@@ -0,0 +1,1224 @@
+
+
+    ÙÆÙÆÙÆ                    ÙÆÙÆÙÆ       ÙÆÙÆÙÆÙÆÙÆÙÆÙÆÙÆÙÆ
+    ÙÆÙÆÙÆÙÆ                ÙÆÙÆÙÆÙÆ         ÙÆÙÆ ÙÆÙÆ ÙÆÙÆ
+    ÙÆÙÆÙÆÙÆ                ÙÆÙÆÙÆÙÆ         ÙÆÙÆ ÙÆÙÆ ÙÆÙÆ
+    ÙÆÙÆÙÆÙÆÙÆ            ÙÆÙÆÙÆÙÆÙÆ         ÙÆÙÆ ÙÆÙÆ ÙÆÙÆ
+    ÙÆÙÆÙÆÙÆÙÆÙÆ        ÙÆÙÆÙÆÙÆÙÆÙÆ         ÙÆÙÆ ÙÆÙÆ ÙÆÙÆ
+    ÙÆÙÆÙÆÙÆÙÆÙÆÙÆ    ÙÆÙÆÙÆÙÆÙÆÙÆÙÆ         ÙÆÙÆ ÙÆÙÆ ÙÆÙÆ
+    ÙÇÙÇÙÇ  ÙÇÙÇÙÇ    ÙÇÙÇÙÇ  ÙÇÙÇÙÇ         ÙÇÙÇ ÙÇÙÇ ÙÇÙÇ
+    ÙÇÙÇÙÇ    ÙÇÙÇÙÇÙÇÙÇÙÇ    ÙÇÙÇÙÇ         ÙÇÙÇ ÙÇÙÇ ÙÇÙÇ
+    ÙÇÙÇÙÇ    ÙÇÙÇÙÇÙÇÙÇÙÇ    ÙÇÙÇÙÇ         ÙÇÙÇ ÙÇÙÇ ÙÇÙÇ
+    ÙÇÙÇÙÇ      ÙÇÙÇÙÇÙÇ      ÙÇÙÇÙÇ         ÙÇÙÇ ÙÇÙÇ ÙÇÙÇ
+    ÙÇÙÇÙÇ        ÙÇÙÇ        ÙÇÙÇÙÇ         ÙÇÙÇ ÙÇÙÇ ÙÇÙÇ
+    ÙÇÙÇÙÇ                    ÙÇÙÇÙÇ         ÙÇÙÇ ÙÇÙÇ ÙÇÙÇ    ·©¤e¶wÚ?w¢Ìe
+    ÙÇÙÇÙÇ                    ÙÇÙÇÙÇ         ÙÇÙÇ ÙÇÙÇ ÙÇÙÇ    ÙF¬a¶w¬é¡w¬áÙG
+    ÙÈÙÈÙÈ                    ÙÈÙÈÙÈ         ÙÈÙÈ ÙÈÙÈ ÙÈÙÈ
+    ÙÈÙÈÙÈ                    ÙÈÙÈÙÈ         ÙÈÙÈ ÙÈÙÈ ÙÈÙÈ     ÜÏ, 1991-1998
+    ÙÈÙÈÙÈ                    ÙÈÙÈÙÈ         ÙÈÙÈ ÙÈÙÈ ÙÈÙÈ     MDIR COMPUTER
+    ÙÈÙÈÙÈ                    ÙÈÙÈÙÈ         ÙÈÙÈ ÙÈÙÈ ÙÈÙÈ
+    ÙÈÙÈÙÈ                    ÙÈÙÈÙÈ         ÙÈÙÈ ÙÈÙÈ ÙÈÙÈ     ÐeÑw ¡¯a
+    ÙÈÙÈÙÈ                    ÙÈÙÈÙÈ dir   ÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈ   ¹A¸b¤åÑ¡ 3.10
+
+
+  Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡
+  ÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉÙÉ
+
+  ÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈ
+
+
+  ÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌ
+  ÙÌ                                                                        ÙÌ
+  ÙÌ MdirIII· á£¡´eµA Í¡ÐqE ¡¡e Ïa¡aÁ ¬é¡w¬áÌa·©wµA Ðe ¥Ðe·e ÙÌ
+  ÙÌ  ee·¡µAA ·¶·a¡a, ´ááÐe ·¡·A¡e ¬w´ó¸â·¥ ¡¢¸âµA ·¡¶wI ® ´ô¯s¡a. ÙÌ
+  ÙÌ ¬w´ó¸â·¥ ¡¢¸â·¡e MdirIII· ¬é¡w¬á£¹ Ïa¡a·i ·¡¶wÐaµa »¢/e¸ó¸â·a¡ ÙÌ
+  ÙÌ ·¡·¢·i ¥¡e ¡¡e Ð¶ái  iÐs¡a.                                      ÙÌ
+  ÙÌ                                                                        ÙÌ
+  ÙÌ MdirIII(Ñ¢·e M3 ¡e M·¡a¡¡ ¦s¡a)e (º) ÐeiÁ ÄñÏAÈá ÂÌe¦· ÙÌ
+  ÙÌ "´ás¡áÀg¡»¡a¶ '¡¯a¡ww´á º·¡¡'(ÂA¸÷Ðe,£¥w ¸á)"e PC Á¡¥¡¸a ÙÌ
+  ÙÌ i ¶áÐe eÐ¥¥µA ¡¯aÅUÁ ÐqA ¡/¹A·A¡ ·¶¯s¡a.  ee·¡· ¬á¡e¬w ÙÌ
+  ÙÌ Ðáb´ô·¡e eÐ¥¥µA ¦¢· ÑwÈ¡ ¹A·I ® ´ô·a¡a, b¹· ¢¬áÌa·©· ¢ ÙÌ
+  ÙÌ e¸å¸i qÐs¡a. ·¡i ¶á¤eÐi w¶ ¸á¸b¥¤óµA ¸áÁ¢A´á Àá¤éI ® ·¶·q ÙÌ
+  ÙÌ ·i ´iaa³¡a.                                                       ÙÌ
+  ÙÌ                                                                        ÙÌ
+  ÙÌ É·¯¥ w·i É·Ð ´ó¡aA´á ·©¤e¬a¶w¸ai·¡ ¸a·A¡·¡ ¬a¶wÐi ® ·¶Aq Ðáb ÙÌ
+  ÙÌ E ¤á¸å·i '·©¤e¶w'·¡a¡ Ða¡a ¸÷¯¢ w¢¸éÀai É·Ð ¬a¶wÐi ® ·¶e ¡¡e ÙÌ
+  ÙÌ ¡w·¡ µÅ¥bÐaA ¸b·Ðae ¤á¸å·i 'w¢Ìe'·¡a¡ Ðs¡a.                 ÙÌ
+  ÙÌ                                                                        ÙÌ
+  ÙÌ MdirIIIe ´á¶Á´á(ShareWare)·³¡a.  ·©¤e¶w·e ·¥¸â·¥ ¶w¡¡ ¬a¶wÐi  ÙÌ
+  ÙÌ w¶µE ·Ïa¡aÀáñ ¸a·A¡·¡ ¬a¶wÐi ® ·¶¯s¡a. w¢Ìe·e w¢¸ai ÙÌ
+  ÙÌ ¶áÐe õ·³¡a. b¹· Ìa·©·i ¢e ®¸÷Ða»¡  a¯³¯¡µ¡.                     ÙÌ
+  ÙÌ                                                                        ÙÌ
+  ÙÌ MdirIIIe ·¥¸â·¥ ¶w¡¡ e ¬a¶wÐi ® ·¶·a¡a,  ¦º·¡ ¤i¬Ðe »¢¸ó or ÙÌ
+  ÙÌ e¸ó¸â·¥ ´ááÐe Ï¡ÐµA Ðaµa ¹A¸b¸ae À·±·i »¡»¡ ´g¯s¡a. ¡Ðe ¹A¸b ÙÌ
+  ÙÌ ¸ae Ïa¡aµA Ðe ´ááÐe ¥¡»w¡ Ða»¡ ´g¯s¡a.                      ÙÌ
+  ÙÌ                                                                        ÙÌ
+  ÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌÙÌ
+
+
+
+  Û£Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Ùº Mdir III  A¯¡Ìe´e Ù¸Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¤
+  Û¢Ða·¡ÉIÁ a¶¡µA MdirIII i ¬a¶wÐa¯¡e ¦iÁ ¥¡a ¶¥ÑÉÐe ¸÷¥¡ aÑÅ·iÛ¢
+  Û¢¶áÐ A¯¡Ìe·¡  aeA´á ·¶¯s¡a.   MdirIIIi ¬awÐa¯¡e  g·e ¦iÁ ÐqAÛ¢
+  Û¢·¢¸ñ·i ¥¡a ®óA ÐiÐi ® ·¶·i õ·³¡a.                             Û¢
+  Û¢                                                                        Û¢
+  Û¢     Ða·¡ÉIµA¬áe º¸aa¯©· '¤i¸a Ðua·¥'·a¡ Àx´aµ¡¯¡¡e S¡a.     Û¢
+  Û¢                  Ða·¡ÉIµA¬áe 'go GLLINE'Ða¡e S¡a.                  Û¢
+  Û¢                                                                        Û¢
+  Û¢      a¶¡µA¬áe º¸aa¯©· "¤i¸aµÁ ÐqA"ae µµA ·¶¯s¡a.      Û¢
+  Û¢                 a¶¡µA¬áe 'go pdsprg'Ða¡e S¡a.                 Û¢
+  Û¦Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¥
+
+
+ÙÀ w¢µA ÅÐe ¸aAÐe ´ee {·¡ ¹A·Ae ORDER.DOCÌa·©·i Àq¡Ða¯³¯¡µ¡.
+
+
+ÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈ
+   Àa
+ÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈ
+
+  Û£Û¡Ù¹ Mdir IIIe ........... ¢´õµA, ´á¡µA ¬a¶wÐae ¢å·¥a? ...........
+  Û¢  Ù¹ Ìa·©¡¢¢ ............. ´áå ±i¡ A´á·¶© ......................
+  Û¢  Ù¹ ¹AÐe¬aÐw ............. ae ·¡å ÑÅwµA¬áe ·© ´eÐ ................
+  Û¢  Ù¹ M3· ¯©Ð ............ ¯¡··i é´á´¡ Â¤iÐa»¡ ......................
+  Û¢  Ù¹ ¬åÈ¬aÐw ............. ¡wwÐ ¬åÈ¬aÐw .............................
+  Û¢  Ù¹ ¢¸w·³b ............. Èa·¡Ï··e ¸÷ i ¯°´á ..........................
+  Û¢  Ù¹  a¶¯a ............... ¬´wºái ¶a¡Ða¸a ...........................
+  Û¢  Ù¹ ÑÁ¡e¬é¡w ............. e®ÐeA ¬é¡w·e ¢¯e ........................
+  Û¢  Ù¹ ¡A·¥ÑÁ¡e ............. ¸åÁA¸â·¥ ¬a¶w¤ó·¡aá .....................
+  Û¢  Ù¹ ¡BÉ¡¡ Å¡¡ ...... ¡BÉ¡¡i µá Å¡Ð? .......................
+  Û¢  Ù¹ ¡BÉ¡¡ ¨i¡ ¤a¡.. QCDe ¡ ¢¡´¡? ..............................
+  Û¢  Ù¹ ¡¯a¡ww ............. ¸÷ i ¡¯a¡ ai Ï©¶aa ´ô·ia? .............
+  Û¢  Ù¹ º¬â ................. Ìa·©µA ¬é¡w·i ¸÷ i¡ i ® ·¶¡? ............
+  Û¢  Ù¹ ÑÁ¡e¥¡Ñ¡ ............. ÑÁ¡eµA ¥¡Ñ¡ b·¡a¡ ¬éÃ¡Ða¡? .............
+  Û¢  Ù¹ ´sÂÌa·© ............. ´sÂÌa·© ae ¤ó aaÁa ºA ..............
+  Û¦Û¡Ù¹ Windows95............. Windows95µA¬á ¬a¶wÐi w¶ ...................
+
+
+ÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈ
+   Mdir IIIe
+ÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈ
+
+  Mdir IIIe Ïa¡a·i ÍeÐaA ¯©ÐÐaeA º¡¢¸â·¡ ·¶·a¡a, ´a¶á Ìa·©Á ¡B
+  É¡¡, ´sÂÌa·©·i ®óA Å¡Ði ® ·¶¡¢ ¡µÁºe Ïa¡a·³¡a.
+  ¹A¸b¸ae a·qÁ {·e ¬b·a¡ Ïa¡a·i  ei´ö¯s¡a.
+
+  o ÄñÏAÈái ¸åÑa ¡¡ae ¬aq¡ ®óA ¬a¶wÐaAq ¡¥¥¸â·¥ ¬a¶w¤ó·¡ eeÐs¡a.
+    µw¢¡A¯¡»¡i ¬a¶wÐeae ¸ñ e  ¹A¶AÐa¡e  ¸÷ i¡ ¬a¶wÐa¡ ®á¶ ¡¯a·¡a¡
+    qÓ¡  i³qa³¡a. (Ðei¡¯ai ¬a¶wÐi w¶ Ðei¡AA»¡a Îa¯¡S¡a.)
+    ´a¶á ÄñÏAÈái ´áa ¸÷¡  a º ´ae ¦iµAAe ¸b´ó ¢¡i ¼µaa©õ
+    ·³¡a. ¡¥¥¸â·a¡ ÑÁ¬iÎaÇ¡, µEÈáÇ¡, AltÇ¡, CtrlÇ¡, F10Ç¡ e ´i¡ a á»¡e
+    ÑÁ¡eÐae· ¸÷¥¡i Àq¡Ða¡e S¡a.  AltÇ¡a CtrlÇ¡i ¬i¼b á¥¡¯³¯¡µ¡.
+    b¡®· ¡s ¬a¶w¸ai ¶áÐe ¡w·e ´aµ ý»¡i ´g´v¯s¡a.  awÐa¡e ®óA
+    ¬a¶wÐi ® ·¶¡¢ ¢ Ï©¶aÐe ¡wi¡ e ¡¡´a´ö¯s¡a.
+
+  o ¸aº ¬a¶wÐae Ïa¡a·i Ì÷eÇ¡µA w¢Ðaµa ÍeÐaA ¯©ÐÐi ® ·¶¯s¡a.  ·¡
+    õ·e ÄñÏAÈá ¬a¶wµA ·¶´á¬á ÂAÐe· Òa·I·i ¹A·Ði õ·³¡a. Á· 28· Ì÷e
+    Ç¡µA b¡ ae Ïa¡a·i »¡¸÷Ði ® ·¶·a¡a, Ðaa· Ì÷eÇ¡µA µaá· Ïa¡
+    a·i w¢Ðae õ¡ awÐs¡a. ¸aº ¬a¶wÐae Ïa¡a·e ¶¥ÈáÃ¡¡~~
+
+  o ´sÂÌa·©·i ®óA Å¡Ði ® ·¶¯s¡a.  ÓeÓ¡ ¬a¶wÐae ¦¦· ´sÂÌa·©·i »¡
+    ¶¥Ðae õ·e ¢¥ ´sÂÌa·©¢· Ìa·©a»¡¡ eeÓ¡ ¯©Ð¯¡Ç¡¡ ¥©® ·¶¯s¡a.
+    ´sÂÌa·©·i  ei¡, Î¡, »¡¶e ¡¥¥¸â·¥ ¡w·¡»¡ e ¸å¢ ´sÂÌa·©Å¡¡µA
+    ¡µ»¡ ´gA Íe¡Ða¡ ¬a¶wÐa¡ ®óA  ei´ö¯s¡a.  É·¯¥·i Ðaáa ¸aai ¤´ó
+    Ðaew ´sÂÌa·©·i ¬a¶wÐae §¥¡a  g´a»¡£a¡  ¢´õ¥¡a ·A¶wÐe ¡w·¡ Iõ
+    {¯s¡a.  ¡Ðe "¸a·Î±´sÂÌa·©"¡ »¡¶¥Ða¡a, ÑÂ¸w¸ai ¸i¡µº ´sÂÌa·©¡
+    ¥iae ¡AAi Ñ¡ÂÐi Ï©¶a´ô·¡ ¯a¯a¡ ´i´a¬á Àá¡Ðs¡a. áÀg·eõ¡ ®óA~
+
+  o e®Ðe ´aqa¶·i ÂÐs¡a. ÑÁaÐa¡ ¶aeÐaA ¬÷Ði ®¡ ·¶»¡ e µ¡Ó¡a
+    µA ¸i i´áµ¡»¡ ´g´a¬á ¸w¯¡e ¬a¶w¯¡ Òa·I·¡ é´á»¡£a¡ eeÐe ÑÁ¡e¬÷µA
+    ¬á ÂAÐe iqÐaA  eia¡ ¡bÐ¯s¡a. ·¡µ×·¡¡e iqÐe ÑÁ¡eµA¬á~~
+
+  o Ïa¡a·i ¯©ÐÐae ¤w¤ó·¡ a´·Ða¡ ¹A·Ðae ¬aÐw¡  g¯s¡a.  ¡¯a·¡a
+    ¡e ¢´õ¥¡a¡ Ïa¡a·i ¯©ÐÐae ¡w·¡ wbÐ´¡ Ðs¡a. a¡¡ Ïa¡a
+    ¯©Ð¯¡ ¸ñ·A¡A¡¡¡a ¸â´á´¡ Ðs¡a.  "M3"e ¡A¡¡¡i ´aµ¸ñ·AÐa»¡ ´g·i®¡
+    ·¶·a¡a ¯aµÁÏ··i Ða¡e ´aº ¸â·e ¡A¡¡¡ e ¬a¶wÐa¡ a á»¡e ¯©ÐAe Ïa¡a
+    ·¡ ¬a¶wÐi ® ·¶¡¢ Ðº¡a. ¡¯a· ¹A´¢·i ñ´á¬á~~
+
+  o ¬a¶w¸aa ¬i·i »¡¸÷Ði ® ·¶¯s¡a.  ´a¶á Ìa·©· ÑÂ¸w¸a¥i¡ ¬i·i i¡
+    º®¡ ·¶´á¬á ¯©ÐÌa·©w·i ÐeµA Ìa´bÐi ® ·¶¯s¡a. ¬i·i  a·q¡~~
+
+
+ÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈ
+   Ìa·©¡¢¢
+ÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈÙÈ
+
+  Û®Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¯
+  ÛÙÈÙÈÙÈÙÈÙÈ Mdir IIIe  a·qÁ {·e Ìa·©i¡ ¬÷A´á ·¶¯s¡a ÙÈÙÈÙÈÙÈÙÈÛ
+  Û²Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¸Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û´
+  Û M.EXE        Û¢ Mdir III ¡A·¥ Ïa¡a                                 Û
+  Û M.DOC        Û¢ ¬a¶w¬é¡w¬á ( »¡q ¥¡¡ ¯¥ Ìa·© )                     Û
+  Û TIP.DOC      Û¢ MdirIIIÑÉ¶w Ë³(·A¶wÐe ¬a¶w¤w¤ó·i ®óA ¬é¡wÐe ¢¬á)     Û
+  Û MSET.EXE     Û¢ ÑÅw¬é¸÷ Ïa¡a                                      Û
+  Û MSET.DOC     Û¢ ÑÅw¬é¸÷ Ïa¡a ¬a¶w¬é¡w¬á                           Û
+  Û M.CFG        Û¢ ÑÅwÌa·© (Configuration file)                          Û
+  Û BLUE.COL     Û¢ ¬iÌa·© (Ìae¤aÈw· ¬i¸÷· Ìa·©)                    Û
+  Û M.HLP        Û¢ ¡¶ i Ìa·©        (¯©Ðº Àq¹¡Q)                     Û
+  Û M-J.HLP      Û¢ ¹¡ÐsÑw ¡¶ i Ìa·© (  "      "   )                     Û
+  Û M-K.HLP      Û¢ µÅ¬÷Ñw ¡¶ i Ìa·© (  "      "   )                     Û
+  Û MUPDATE.DOC  Û¢ ·¡¤å ÌeµA¬á ¬åE ¬aÐw·i ¸â´á Ìa·©                  Û
+  Û Q&A.DOC      Û¢ ¸aº ¢·¤h·e »©¢µA Ð ¸÷¡Ðe õ(»©¢Á s¥e ¡¡·q)  Û
+  Û MDIR.BAT     Û¢ DIR¡ww ÁA¶w ·©ÉÌa·©(Batch file)                    Û
+  Û MCD.BAT      Û¢ NCDÁA¶w ·©ÉÌa·©(Batch file)                         Û
+  Û·Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û«Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¹
+  Û M.ICO        Û¢ ¶å¡¶»a¶w ´a·¡Å¥ Ìa·©                                 Û
+  Û M.PIF        Û¢ ¶å¡¶»a¶w PIF    Ìa·©                                 Û
+  Û ORDER.DOC    Û¢ ¸÷¯¢¬a¶w¸a w¢´·¯¢ (w¢Ðaa Ði w¶ Àq¡¤aq)        Û
+  Û·Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û«Û¡Ùº ¥¡¹¡·AË©¡Ë¡ Ù¸Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¡Û¹
+  Û VV.EXE       Û¢ ¥¡a (Ðei¢¬áÌa·© ¥¡¡Ïa¡a)                       Û
+  Û VV.DOC       Û¢ ¥¡a ¬a¶w¬é¡w¬á                                        Û
+  Û MFF.EXE      Û¢ Mdir III FileFind (Ìa·©Àx¡ Ïa¡a)                  Û
+  Û MFF.DOC      Û¢ MFF  ¬a¶w¬é¡w¬á                                        Û
+  Û MF.EXE       Û¢ Mdir III Format   (Ïi¡Ï¡ ¡¯aÇa Í¡  Ïa¡a)        Û
+  Û MF.DOC       Û¢ MF   ¬a¶w¬é¡w¬á                                        Û
+  Û MCOPY.EXE    Û¢ Mdir III diskCOPY (Ïi¡Ï¡ ¡¯aÇa ¥¢¬a Ïa¡a)        Û
+  Û MCOPY.DOC    Û¢ MCOPY ¬a¶w¬é¡w¬á                                       Û
+  Û±Û¬Û¬Û¬Û¬Û¬Û¬Û¬ÛºÛ¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û°
+   ¤Í¡¯¡µAe ¤ea¯¡ ¶¥¥¥ ´sÂÌa·©·i ·¡¶wÐ º¯¡¡ ¤as¡a.¬é¡w¬áw· Ìa·©·¡
+   ´ô´á¬á ¹A¡ ¬a¶w·i Ða»¡ ¡µÐae w¶a ·¶¯s¡a.
+
+  Û®Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¯
+  ÛÙÈÙÈÙÈÙÈ ´ae Mdir III¬a¶wº ¸a··a¡ ¬¬÷Ae Ìa·©i·³¡a ÙÈÙÈÙÈÙÈÙÈÛ
+  Û²Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¸Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û´
+  Û M.HIS        Û¢ ¡¯a¡ww· Ó¡¯aÉ¡¡ Ìa·©(·¡¸å¡ww·i ¡´â¯¡Åa Ìa·©)   Û
+  Û _M_.BAT      Û¢ ¡A¡¡¡ ¸ñ·A´g¡ ¯©ÐÐi ¡ww ¤Ã¡Ìa·© (ËaµA ¬¬÷Q) Û
+  Û (M).BAT      Û¢ _M_.BATµÁ ÐqA ¬¬÷Ae ¡ww¤Ã¡Ìa·©.(TEMPµA ¬¬÷Q)   Û
+  Û MDIRTREE.DIR Û¢ MCD   Ëa¡¹¡ ¸÷¥¡Ìa·© (ËaµA ¬¬÷Q)                Û
+  Û MDIRTREE.?   Û¢ LAN¶w Ëa¡¹¡ ¸÷¥¡Ìa·© (TEMPµA ¬¬÷Q)                Û
+  Û *.MCD        Û¢ MCD   CDÈa·¡Ëi¶w Ëa¡¹¡ ¸÷¥¡Ìa·©(TEMP¡BÉ¡¡µA ¬¬÷)Û
+  Û M.$$$        Û¢ ¸b´óÀá¡i ¶áÐe ·±¯¡Ìa·©          (TEMP¡BÉ¡¡µA ¬¬÷)Û
+  Û M.$          Û¢ ´sÂÌa·© Àá¡i ¶áÐe ·±¯¡Ìa·©     (TEMP¡BÉ¡¡µA ¬¬÷)Û
+  Û ¦Ëb¸÷·¡q(*)Û¢ ¡¯aÇa ¯aµÁÏ·¶w Ìa·©(Hidden¢¬÷)  (TEMP¡BÉ¡¡µA ¬¬÷)Û
+  Û±Û¬Û¬Û¬Û¬Û¬Û¬Û¬ÛºÛ¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û¬Û°
+   <º> (*)Îa¯¡a A´á ·¶e Ìa·©·e Ï©¶a¯¡ ¬¬÷A´öa ¸a··a¡ »¡¶¡»¡e Ìa·©·³
+        ¡a.   ¶á Ìa·©·¡ »¡¶¡»¡»¡ ´g¡ q´a·¶e w¶e Ðw¸b´ó·¡ ¸÷¬w¸â·a¡
+        ¹·aA»¡ ´g´a¬á »¡¶»¡ ¡µÐe ¬wÈ·¡¡  ·¡¬wÐaA ¡¯aÇa· ¶w··¡ º´áa
+        e w¶ ÑÂ·¥Ð ¥¡¯¡¡ »¡¶A¶a.   e, XMSa EMS¡A¡¡¡a Â¦Ðe w¶e
+        ¡¯aÇa¡ ¯aµÁÏ··i Ða»¡ ´g·a£a¡ ·¡å Ñe¬w·e ´ô¯s¡a.
+        ¯aµÁÏ·Ë¥·e iÏa §aa¶(Ralf brown)· SPAWNOi ¬a¶wÐ¯s¡a.
+

_service:tar_scm:chardet-5.1.0.tar.gz/tests/MacRoman Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/MacRoman/ioreg_output.txt Added

@@ -0,0 +1,4000 @@
+    | | |   "IOReportLegend" = ({"IOReportGroupName"="Interrupt Statistics (by index)","IOReportChannels"=((5291294645182005248,4295098369,"               First Level Interrupt Handler Count"),(5291294645182005249,4295098369,"              Second Level Interrupt Handler Count"),(5291294645182005250,4295098369,"        First Level Interrupt Handler Time (MATUs)"),(5291294645182005251,4295098369,"   Second Level Interrupt Handler CPU Time (MATUs)"),(5291294645182005252,4295098369,"Second Level Interrupt Handler System Time (MATUs)")),"IOReportChannelInfo"={"IOReportChannelUnit"=0},"IOReportSubGroupName"="PWRB 0"})
+    | | |   "power-button-usagepage" = <0c00000000000000>
+    | | |   "DeviceOpenedByEventSystem" = Yes
+    | | |   "power-button-usage" = <4000000000000000>
+    | | |   "device-properties" = {"power-button-usage"=<4000000000000000>,"power-button-usagepage"=<0c00000000000000>}
+    | | |   "IOReportLegendPublic" = Yes
+    | | |   "IOInterruptControllers" = ("ACPIEventController")
+    | | |   "_STA" = 18446744073709551615
+    | | |   "IOInterruptSpecifiers" = (<0200000000000000>)
+    | | |   "name" = <"PNP0C0C">
+    | | |   "acpi-wake-type" = 1
+    | | | }
+    | | | 
+    | | +-o AppleACPIButton  <class AppleACPIButton, id 0x1000003b2, registered, matched, active, busy 0 (0 ms), retain 7>
+    | |   | {
+    | |   |   "IOMatchedAtBoot" = Yes
+    | |   |   "PrimaryUsagePage" = 12
+    | |   |   "IOUserClientClass" = "IOHIDEventServiceUserClient"
+    | |   |   "VersionNumber" = 0
+    | |   |   "VendorID" = 0
+    | |   |   "PressCountTrackingEnabled" = Yes
+    | |   |   "Built-In" = Yes
+    | |   |   "PressCountDoublePressTimeout" = 300000
+    | |   |   "power-button-usagepage" = 12
+    | |   |   "IONameMatched" = "PNP0C0C"
+    | |   |   "power-button-usage" = 64
+    | |   |   "ButtonDownMS" = 122
+    | |   |   "PressCountTriplePressTimeout" = 600000
+    | |   |   "ButtonEvents" = 1
+    | |   |   "ProductID" = 0
+    | |   |   "DeviceUsagePairs" = ({"DeviceUsagePage"=12,"DeviceUsage"=1})
+    | |   |   "HIDServiceGlobalModifiersUsage" = 2
+    | |   |   "ReportInterval" = 18000
+    | |   |   "VendorIDSource" = 0
+    | |   |   "HIDEventServiceProperties" = {"HIDMouseKeysOptionToggles"=0,"JitterNoClick"=1,"ActuateDetents"=1,"Dragging"=0,"HIDSlowKeysDelay"=0,"JitterNoMove"=1,"FlipLeftAndRightEdgeGestures"=No,"HIDTrackpadScrollAcceleration"=20480,"HIDInitialKeyRepeat"=500000000,"TrackpadThreeFingerHorizSwipeGesture"=2,"TrackpadThreeFingerDrag"=No,"MTGestureConfiguration"={"Version"=1,"Behaviors"=()},"MTGestureConfigurationOverride"={"Version"=1,"Behaviors"=({"BehaviorID"=1})},"HIDPointerAcceleration"=45056,"NotificationCenterGestureMode"={"AlwaysGenerateNotificationCenterGesture"=No,"ClientPID"=475},"UserPreferences"=Yes,"HIDDefaultParameters"=Yes,"TrackpadHorizScroll"=1,"HIDF12EjectDelay"=250,"UnifiedKeyMapping"=No,"HIDMouseKeysOn"=0,"TrackpadFourFingerVertSwipeGesture"=2,"TrackpadTwoFingerFromRightEdgeSwipeGesture"=3,"USBMouseStopsTrackpad"=0,"AppKitActuateWithID"=15,"HIDMouseScrollAcceleration"=20480,"TrackpadThreeFingerTapGesture"=0,"MouseOneFingerDoubleTapGesture"=0,"HIDScrollZoomModifierMask"=0,"TrackpadScroll"=Yes,"Trackpad Jitter Milliseconds"=192,"HIDPointerButtonMode"=2,"HIDKeyboardModifierMappingPairs"=(),"LogLevel"=6,"TrackpadFourFingerHorizSwipeGesture"=2,"HIDTrackpadAcceleration"=45056,"DragLock"=0,"DeviceOpenedByEventSystem"=Yes,"ReportInterval"=0,"TwofingerNoAction"=1,"HIDClickTime"=500000000,"MouseTwoFingerDoubleTapGesture"=3,"HIDMouseAcceleration"=45056,"SecondClickThreshold"=1,"HIDStickyKeysShiftToggles"=0,"TrackpadHandResting"=Yes,"TrackpadFourFingerPinchGesture"=2,"MouseMomentumScroll"=Yes,"TrackpadFiveFingerPinchGesture"=2,"PalmNoAction Permanent"=1,"MouseButtonMode"="OneButton","TrackpadThreeFingerVertSwipeGesture"=2,"OutsidezoneNoAction When Typing"=1,"ForceSuppressed"=No,"TrackpadCornerSecondaryClick"=0,"MouseButtonDivision"=55,"HIDStickyKeysOn"=0,"HIDStickyKeysDisabled"=0,"EjectDelay"=0,"TrackpadPinch"=1,"FirstClickThreshold"=1,"HIDClickSpace"=(0,0),"PalmNoAction When Typing"=1,"MouseVerticalScroll"=Yes,"version"=1,"HIDSubinterfaceID"=3,"MouseHorizontalScroll"=Yes,"BatchInterval"=1,"TrackpadMomentumScroll"=Yes,"MouseTwoFingerHorizSwipeGesture"=2,"MouseKeysStopsTrackpad"=0,"HIDKeyRepeat"=83333333,"HIDScrollReset"=Yes,"TrackpadTwoFingerDoubleTapGesture"=1,"Clicking"=0,"TrackpadRightClick"=Yes,"TrackpadRotate"=1,"Use Panther Settings for W"=0,"HIDScrollAcceleration"=20480,"HIDFKeyMode"=0}
+    | |   |   "CFBundleIdentifier" = "com.apple.driver.AppleACPIButtons"
+    | |   |   "IOCFPlugInTypes" = {"7DDEECA8-A7B4-11DA-8A0E-0014519758EF"="IOHIDFamily.kext/Contents/PlugIns/IOHIDLib.plugin","FA12FA38-6F1A-11D4-BA0C-0005028F18D5"="IOHIDFamily.kext/Contents/PlugIns/IOHIDLib.plugin"}
+    | |   |   "IOProviderClass" = "IOACPIPlatformDevice"
+    | |   |   "IONameMatch" = ("PNP0C0C","power-button","PNP0C0E","sleep-button")
+    | |   |   "LocationID" = 0
+    | |   |   "IOClass" = "AppleACPIButton"
+    | |   |   "CFBundleIdentifierKernel" = "com.apple.driver.AppleACPIButtons"
+    | |   |   "PrimaryUsage" = 1
+    | |   |   "HIDDefaultBehavior" = ""
+    | |   |   "CountryCode" = 0
+    | |   |   "HIDServiceSupport" = Yes
+    | |   |   "IOMatchCategory" = "IODefaultMatchCategory"
+    | |   |   "IOGeneralInterest" = "IOCommand is not serializable"
+    | |   |   "IOProbeScore" = 0
+    | |   | }
+    | |   | 
+    | |   +-o IOHIDEventServiceUserClient  <class IOHIDEventServiceUserClient, id 0x10000084d, !registered, !matched, active, busy 0, retain 6>
+    | |       {
+    | |         "DebugState" = {"EventQueue"={"NoFullMsg"=0,"tail"=128,"NotificationForce"=0,"NotificationCount"=2,"head"=128},"EnqueueEventCount"=2,"LastEventType"=3,"LastEventTime"=93035472183428}
+    | |         "IOUserClientCreator" = "pid 167, WindowServer"
+    | |         "IOUserClientDefaultLocking" = Yes
+    | |       }
+    | |       
+    | +-o PNLF@0  <class IOACPIPlatformDevice, id 0x10000028e, registered, matched, active, busy 0 (74 ms), retain 12>
+    | | | {
+    | | |   "acpi-path" = "IOACPIPlane:/_SB/PNLF@0"
+    | | |   "compatible" = <"backlight">
+    | | |   "AppleNumBrightLevels" = 1024
+    | | |   "_UID" = "15"
+    | | |   "device-properties" = {"acpi-device"="IOACPIPlatformDevice is not serializable","acpi-path"="IOACPIPlane:/_SB/PNLF@0"}
+    | | |   "AppleRestoreBacklight" = No
+    | | |   "ApplePanelRawBrightness" = 698
+    | | |   "_STA" = 11
+    | | |   "acpi-device" = "IOACPIPlatformDevice is not serializable"
+    | | |   "_ADR" = 0
+    | | |   "name" = <"APP0002">
+    | | | }
+    | | | 
+    | | +-o AGDCBacklightControl  <class AGDCBacklightControl, id 0x100000702, registered, matched, active, busy 0 (3 ms), retain 5>
+    | | |   {
+    | | |     "IOClass" = "AGDCBacklightControl"
+    | | |     "CFBundleIdentifier" = "com.apple.driver.AGDCBacklightControl"
+    | | |     "IOProviderClass" = "IOACPIPlatformDevice"
+    | | |     "IODisplayParameters" = {"brightness"={"max"=255,"min"=40},"commit"={"reg"=0}}
+    | | |     "ApplePanel" = {"F15Ia040"=<0011000002950351043d05b707bf0a540e33135d1a5f2426319d443d5e6e8322b6c7ff72>}
+    | | |     "IOProbeScore" = 5000
+    | | |     "IONameMatch" = "backlight"
+    | | |     "IOMatchCategory" = "IODisplayParameters"
+    | | |     "ConfigMap" = {"Mac-1E7E29AD0135F9BC"=Yes,"Mac-564FBA6031E5946A"=Yes,"Mac-A61BADE1FDAD7B05"=Yes,"Mac-B1C85D9DF5D23D37"=Yes,"Mac-937A206F2EE63C01"=Yes,"Mac-E1008331FDC96864"=Yes}
+    | | |     "IONameMatched" = "backlight"
+    | | |     "IOPersonalityPublisher" = "com.apple.driver.AGDCBacklightControl"
+    | | |     "CFBundleIdentifierKernel" = "com.apple.driver.AGDCBacklightControl"
+    | | |     "AppleScaledLE" = <00000a000d00100016001e00290038004d0069009000c600100179010c02da02fc03>
+    | | |   }
+    | | |   
+    | | +-o AGDCBacklightControlNub  <class AGDCBacklightControlNub, id 0x100000859, !registered, !matched, active, busy 0, retain 9>
+    | | |   {
+    | | |     "IOMatchCategory" = "IODisplayParameters"
+    | | |     "IONameMatched" = "backlight"
+    | | |     "IONameMatch" = "backlight"
+    | | |     "IOProviderClass" = "IOACPIPlatformDevice"
+    | | |     "CFBundleIdentifier" = "com.apple.driver.AGDCBacklightControl"
+    | | |     "IODisplayParameters" = {"brightness"={"max"=255,"min"=40},"commit"={"reg"=0}}
+    | | |   }
+    | | |   
+    | | +-o AGDCBacklightControlNub  <class AGDCBacklightControlNub, id 0x10000085a, !registered, !matched, active, busy 0, retain 9>
+    | |     {
+    | |       "IOMatchCategory" = "IODisplayParameters"
+    | |       "IONameMatched" = "backlight"
+    | |       "IONameMatch" = "backlight"
+    | |       "IOProviderClass" = "IOACPIPlatformDevice"
+    | |       "CFBundleIdentifier" = "com.apple.driver.AGDCBacklightControl"
+    | |       "IODisplayParameters" = {"brightness"={"max"=255,"min"=40},"commit"={"reg"=0}}
+    | |     }
+    | |     
+    | +-o SLPB  <class IOACPIPlatformDevice, id 0x10000028f, registered, matched, active, busy 0 (4 ms), retain 9>
+    | | | {
+    | | |   "name" = <"PNP0C0E">
+    | | |   "_STA" = 11
+    | | |   "DeviceOpenedByEventSystem" = Yes
+    | | | }
+    | | | 
+    | | +-o AppleACPIButton  <class AppleACPIButton, id 0x1000003b3, registered, matched, active, busy 0 (0 ms), retain 7>
+    | |   | {
+    | |   |   "IOClass" = "AppleACPIButton"
+    | |   |   "PressCountDoublePressTimeout" = 300000
+    | |   |   "IOMatchedAtBoot" = Yes
+    | |   |   "HIDDefaultBehavior" = ""
+    | |   |   "HIDServiceGlobalModifiersUsage" = 2
+    | |   |   "IOProviderClass" = "IOACPIPlatformDevice"
+    | |   |   "Built-In" = Yes
+    | |   |   "PressCountTrackingEnabled" = Yes
+    | |   |   "power-button-usage" = 48
+    | |   |   "DeviceUsagePairs" = ({"DeviceUsagePage"=12,"DeviceUsage"=1})
+    | |   |   "IOProbeScore" = 0
+    | |   |   "PressCountTriplePressTimeout" = 600000
+    | |   |   "VendorIDSource" = 0
+    | |   |   "power-button-usagepage" = 12
+    | |   |   "IOUserClientClass" = "IOHIDEventServiceUserClient"
+    | |   |   "HIDServiceSupport" = Yes
+    | |   |   "IONameMatch" = ("PNP0C0C","power-button","PNP0C0E","sleep-button")
+    | |   |   "IOCFPlugInTypes" = {"7DDEECA8-A7B4-11DA-8A0E-0014519758EF"="IOHIDFamily.kext/Contents/PlugIns/IOHIDLib.plugin","FA12FA38-6F1A-11D4-BA0C-0005028F18D5"="IOHIDFamily.kext/Contents/PlugIns/IOHIDLib.plugin"}
+    | |   |   "CountryCode" = 0
+    | |   |   "VendorID" = 0
+    | |   |   "CFBundleIdentifierKernel" = "com.apple.driver.AppleACPIButtons"
+    | |   |   "VersionNumber" = 0
+    | |   |   "IOMatchCategory" = "IODefaultMatchCategory"
+    | |   |   "CFBundleIdentifier" = "com.apple.driver.AppleACPIButtons"
+    | |   |   "HIDEventServiceProperties" = {"HIDMouseKeysOptionToggles"=0,"JitterNoClick"=1,"ActuateDetents"=1,"Dragging"=0,"HIDSlowKeysDelay"=0,"JitterNoMove"=1,"FlipLeftAndRightEdgeGestures"=No,"HIDTrackpadScrollAcceleration"=20480,"HIDInitialKeyRepeat"=500000000,"TrackpadThreeFingerHorizSwipeGesture"=2,"TrackpadThreeFingerDrag"=No,"MTGestureConfiguration"={"Version"=1,"Behaviors"=()},"MTGestureConfigurationOverride"={"Version"=1,"Behaviors"=({"BehaviorID"=1})},"HIDPointerAcceleration"=45056,"NotificationCenterGestureMode"={"AlwaysGenerateNotificationCenterGesture"=No,"ClientPID"=475},"UserPreferences"=Yes,"HIDDefaultParameters"=Yes,"TrackpadHorizScroll"=1,"HIDF12EjectDelay"=250,"UnifiedKeyMapping"=No,"HIDMouseKeysOn"=0,"TrackpadFourFingerVertSwipeGesture"=2,"TrackpadTwoFingerFromRightEdgeSwipeGesture"=3,"USBMouseStopsTrackpad"=0,"AppKitActuateWithID"=15,"HIDMouseScrollAcceleration"=20480,"TrackpadThreeFingerTapGesture"=0,"MouseOneFingerDoubleTapGesture"=0,"HIDScrollZoomModifierMask"=0,"TrackpadScroll"=Yes,"Trackpad Jitter Milliseconds"=192,"HIDPointerButtonMode"=2,"HIDKeyboardModifierMappingPairs"=(),"LogLevel"=6,"TrackpadFourFingerHorizSwipeGesture"=2,"HIDTrackpadAcceleration"=45056,"DragLock"=0,"DeviceOpenedByEventSystem"=Yes,"ReportInterval"=0,"TwofingerNoAction"=1,"HIDClickTime"=500000000,"MouseTwoFingerDoubleTapGesture"=3,"HIDMouseAcceleration"=45056,"SecondClickThreshold"=1,"HIDStickyKeysShiftToggles"=0,"TrackpadHandResting"=Yes,"TrackpadFourFingerPinchGesture"=2,"MouseMomentumScroll"=Yes,"TrackpadFiveFingerPinchGesture"=2,"PalmNoAction Permanent"=1,"MouseButtonMode"="OneButton","TrackpadThreeFingerVertSwipeGesture"=2,"OutsidezoneNoAction When Typing"=1,"ForceSuppressed"=No,"TrackpadCornerSecondaryClick"=0,"MouseButtonDivision"=55,"HIDStickyKeysOn"=0,"HIDStickyKeysDisabled"=0,"EjectDelay"=0,"TrackpadPinch"=1,"FirstClickThreshold"=1,"HIDClickSpace"=(0,0),"PalmNoAction When Typing"=1,"MouseVerticalScroll"=Yes,"version"=1,"HIDSubinterfaceID"=3,"MouseHorizontalScroll"=Yes,"BatchInterval"=1,"TrackpadMomentumScroll"=Yes,"MouseTwoFingerHorizSwipeGesture"=2,"MouseKeysStopsTrackpad"=0,"HIDKeyRepeat"=83333333,"HIDScrollReset"=Yes,"TrackpadTwoFingerDoubleTapGesture"=1,"Clicking"=0,"TrackpadRightClick"=Yes,"TrackpadRotate"=1,"Use Panther Settings for W"=0,"HIDScrollAcceleration"=20480,"HIDFKeyMode"=0}
+    | |   |   "IONameMatched" = "PNP0C0E"
+    | |   |   "LocationID" = 0
+    | |   |   "ProductID" = 0
+    | |   |   "PrimaryUsage" = 1
+    | |   |   "IOGeneralInterest" = "IOCommand is not serializable"
+    | |   |   "ReportInterval" = 18000
+    | |   |   "PrimaryUsagePage" = 12
+    | |   | }
+    | |   | 
+    | |   +-o IOHIDEventServiceUserClient  <class IOHIDEventServiceUserClient, id 0x10000084c, !registered, !matched, active, busy 0, retain 6>
+    | |       {
+    | |         "DebugState" = {"EventQueue"={"NoFullMsg"=0,"tail"=0,"NotificationForce"=0,"NotificationCount"=0,"head"=0}}
+    | |         "IOUserClientCreator" = "pid 167, WindowServer"
+    | |         "IOUserClientDefaultLocking" = Yes
+    | |       }
+    | |       
+    | +-o AppleEFIRuntime  <class AppleEFIRuntime, id 0x1000003a8, registered, matched, active, busy 0 (323 ms), retain 8>
+    |   | {
+    |   |   "IOProbeScore" = 0
+    |   |   "CFBundleIdentifier" = "com.apple.driver.AppleEFIRuntime"
+    |   |   "IOMatchCategory" = "AppleEFIRuntime"
+    |   |   "IOClass" = "AppleEFIRuntime"
+    |   |   "IOPersonalityPublisher" = "com.apple.driver.AppleEFIRuntime"
+    |   |   "IOProviderClass" = "AppleACPIPlatformExpert"
+    |   |   "CFBundleIdentifierKernel" = "com.apple.driver.AppleEFIRuntime"
+    |   |   "IOMatchedAtBoot" = Yes
+    |   |   "IOResourceMatch" = "ACPI"
+    |   | }
+    |   | 
+    |   +-o AppleEFINVRAM  <class AppleEFINVRAM, id 0x1000003ae, registered, matched, active, busy 0 (0 ms), retain 1586>
+    |       {
+    |         "efi-backup-boot-device-data" = <02010c00d041030a0000000001010600001b0101060000000316100001000000000000000000000004012a0002000000062c010000000000fa1566050000000082acf88ac41ec44aa3edb6a8714e2dc8020204032400f7fc74be7c0bf349914701f4042e68425444888115a46d47b26fbc9f462d1a6404049a005c00350039003700340044004400310038002d0036004200430042002d0034003800300033002d0038004100370039002d003900320030003100340034003300410037003400410039005c00530079007300740065006d005c004c006900620072006100720079005c0043006f0072006500530065007200760069006300650073005c0062006f006f0074002e0065006600690000007fff0400>
+    |         "PSFU_STATUS" = <"NOUPD">
+    |         "current-network" = <0000000000000000040005000a000000426c616e636861726473000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000050000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000091d6a35613e29c9d2d36ebaf5b2d35b9bf7e3566149fa716c9bbff52f6b86317>
+    |         "fm-spstatus" = <"YES">
+    |         "efi-boot-device" = <"<array><dict><key>IOMatch</key><dict><key>IOProviderClass</key><string>IOMedia</string><key>IOPropertyMatch</key><dict><key>UUID</key><string>81884454-A415-476D-B26F-BC9F462D1A64</string></dict></dict><key>BLLastBSDName</key><string>disk1s3</string></dict><dict><key>IOEFIDevicePathType</key><string>MediaFilePath</string><key>Path</key><string>\5974DD18-6BCB-4803-8A79-9201443A74A9\System\Library\CoreServices\boot.efi</string></dict></array>">
+    |         "efi-apple-payload2-data" = <02010c00d041030a0000000001010600001b0101060000000316100001000000000000000000000004012a00010000000600000000000000002c0100000000009e712b585867d54a9fa2cb3c4dc69d3e02020404a2005c004500460049005c004100500050004c0045005c00550050004400410054004500520053005c004d0055004c005400490055005000440041005400450052005c004d00610063002d0039003300370041003200300036004600320045004500360033004300300031002d00430030005f00310036002e0032002e0030002d00430031005f00340037002e0035002d00520030002e00620069006e0000007fff0400>
+    |         "prev-lang-diags:kbd" = <"en">
+    |         "gpu-policy" = <01>
+    |         "bodega-product-id" = <5f00000000000000>
+    |         "efi-apple-payload0-data" = <02010c00d041030a0000000001010600001b0101060000000316100001000000000000000000000004012a00010000000600000000000000002c0100000000009e712b585867d54a9fa2cb3c4dc69d3e0202040460005c004500460049005c004100500050004c0045005c00550050004400410054004500520053005c004d0055004c005400490055005000440041005400450052005c00540068006f0072005500740069006c002e0065006600690000007fff0400>
+    |         "LocationServicesEnabled" = <01>
+    |         "auto-boot" = <"true">
+    |         "IONVRAM-SYNCNOW-PROPERTY" = <"IONVRAM-SYNCNOW-PROPERTY">
+    |         "efi-backup-boot-device" = <"<array><dict><key>IOMatch</key><dict><key>IOProviderClass</key><string>IOMedia</string><key>IOPropertyMatch</key><dict><key>UUID</key><string>81884454-A415-476D-B26F-BC9F462D1A64</string></dict></dict><key>BLLastBSDName</key><string>disk1s3</string></dict><dict><key>IOEFIDevicePathType</key><string>MediaFilePath</string><key>Path</key><string>\5974DD18-6BCB-4803-8A79-9201443A74A9\System\Library\CoreServices\boot.efi</string></dict></array>">
+    |         "multiupdater-retry-limits" = <0100000000000000>
+    |         "efi-apple-payload0" = <"<array><dict><key>IOMatch</key><dict><key>IOProviderClass</key><string>IOMedia</string><key>IOPropertyMatch</key><dict><key>UUID</key><string>582B719E-6758-4AD5-9FA2-CB3C4DC69D3E</string></dict></dict><key>BLLastBSDName</key><string>disk0s1</string></dict><dict><key>IOEFIDevicePathType</key><string>MediaFilePath</string><key>Path</key><string>\EFI\APPLE\UPDATERS\MULTIUPDATER\ThorUtil.efi</string></dict></array>">
+    |         "bluetoothActiveControllerInfo" = <7b00ac05010000000000a483e7c0bec3>
+    |         "fmm-computer-name" = <"Dan's MacBook Pro">
+    |         "preferred-count" = 10
+    |         "ThorUpdateResult" = <0000000000000000>
+    |         "ota-controllerVersion" = <"SUMacController-2.4">
+    |         "ota-updateType" = <"incremental">
+    |         "bluetoothInternalControllerInfo" = <7b00ac0500000000a483e7c0bec3>
+    |         "fmm-mobileme-token-FMM" = <62706c6973743030de0102030405060708090a0b0c0d0e0f1017180f191a0f1b1c1d1e1f215f100f756e726567697374657253746174655875736572496e666f5961757468546f6b656e5e726f6f74566f6c756d65555549445e64697361626c65436f6e74657874567573657269645f10106c6173744964656e7469747954696d655d656e61626c65436f6e7465787458757365726e616d6558706572736f6e49445761646454696d6554677569645f1012656e61626c656444617461636c61737365735f101364617461636c61737350726f706572746965731000d31112131415165f1015496e5573654f776e6572446973706c61794e616d655f1013496e5573654f776e657246697273744e616d655f1012496e5573654f776e65724c6173744e616d655f101044616e69656c20426c616e63686172645644616e69656c59426c616e63686172645f10cc4541415941414141424c77494141414141474b5065735552446d647a4c6d6c6a624739315a4335686458526f765143717a4b6237367664454f65475338536b33416943574a595f66547858583378484f6f6279554f4b4e6d4d746b7757484f61546e45436c6b4b31413463346b515a433573386d387879794d79446276455779426f6e384a73314637474733794169676d556f756473446967466b317448544669794e726b696a4e6b50344d42334276634a496d53546b32566d31424b32736a4246544c485f7a4643677e7e5f102434433146363144422d413530452d344632322d384534302d3734324544373533374636391101f52341d8a3dec1cd3a085f101764616e2e626c616e636861726440676d61696c2e636f6d5835313733323834362341d8a3debfd4354c5f102443363234323543452d344336332d344635342d423545452d303136394644424537423641a1205f1021636f6d2e6170706c652e44617461636c6173732e4465766963654c6f6361746f72d12022d5232425262728292a2b2c56617073456e7658686f73746e616d655d6964734964656e74696669657256736368656d655d617574684d656368616e69736d5a70726f64756374696f6e5f10137035372d666d69702e69636c6f75642e636f6d5f102434303545303746442d324441362d344145302d393939362d32434530373142304145344355687474707355746f6b656e0008002500370040004a00590068006f00820090009900a200aa00af00c400da00dc00e300fb0111012601390140014a021902400243024c0266026f0278029f02a102c502c802d302da02e302f102f8030603110327034e03540000000000000201000000000000002d0000000000000000000000000000035a>

_service:tar_scm:chardet-4.0.0.tar.gz/tests/README.txt -> _service:tar_scm:chardet-5.1.0.tar.gz/tests/README.txt Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/UTF-16BE Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/UTF-16BE/nobom-utf16be.txt Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/UTF-16BE/plane1-utf-16be.html Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/UTF-16LE Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/UTF-16LE/nobom-utf16le.txt Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/UTF-16LE/plane1-utf-16le.html Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/UTF-32BE Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/UTF-32BE/nobom-utf32be.txt Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/UTF-32BE/plane1-utf-32be.html Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/UTF-32LE Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/UTF-32LE/nobom-utf32le.txt Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/UTF-32LE/plane1-utf-32le.html Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-2-croatian Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-2-croatian/_ude_1.txt Added

@@ -0,0 +1,17 @@
+Eto, ni genocid nije ¹to je nekad bio. Iznenada su poo¹trili kriterije, nepo¹teno podigli letvicu, i mi se nismo kvalificirali. Haa¹ka presuda kako Srbi nisu napravili genocid nad Hrvatima, a ni obrnuto, ogorèila je mnoge. Stvar je zavr¹ila kao i inaèe kad se negdje moramo meðunarodno dokazivati, jednako kao nogometno prvenstvo u Brazilu ili rukometno u Kataru. U jednu rijeè, suci su nas pokrali.
+Vijesti
+Ovo je gotovo jednodu¹an zakljuèak. Politièari se redom, i u vlasti i u opoziciji, trse koji æe djelovati zgranutije, ojaðenije, kao kazali¹ni amateri potreseno se hvataju za srce, a novine se natjeèu u gnjevu za Haag koji je ponovno ravnodu¹an za na¹u agoniju devedesetih, no sve je to zapravo jedna kukavna i prijetvorna predstava jer drugaèiji ishod nismo ni oèekivali. Svakome je iole upuæenijem otpoèetka bilo jasno kako je ta optu¾nica samo lakomisleno, neuko politièko razmetanje za domaæu upotrebu, proziran blef, mahanje praznom pu¹kom. Desetke milijuna potro¹ili smo na jednu uzaludnu sudsku bitku koja je samo trebala nahraniti na¹e nacionalistièke iluzije.
+Za ljubav istine
+Napokon, poèetkom tjedna obistinilo se ono na ¹to su trezveniji odvjetnici godinama upozoravali. U Haagu su presudili kako genocida ovdje nije bilo. I zaista, po du¹i govoreæi, nije. Genocid, koji bi se na na¹ jezik najbli¾e mogao prevesti kao istrebljenje, je velika i te¹ka rijeè koju ne valja olako izgovarati. Genocid je zloèin iznad svih zloèina. Razaranje Vukovara bila je krvava, ljudski sramotna i vojnièki neèasna opsada jednog grada do njegovog potpunog iscrpljivanja, i dogodilo se tu koje¹ta protiv zakona i boljih ratnih obièaja, masovno i neselektivno bombardiranje kuæa, ubijanje civila, hladnokrvno strijeljanje ranjenika na Ovèari, ali sve je to, veæ i samo po sebi dovoljno u¾asno, opet te¹ko ugurati u definiciju da se ?u cijelosti ili djelomièno uni¹ti jedna nacionalna, etnièka, rasna ili vjerska skupina?.
+Koridorom izmeðu blatnjavih rovova i tenkova tu¾na je kolona prognanika naposljetku iza¹la iz Vukovara, a veæina zarobljenika su se nakon nekoliko mjeseci premlaæeni vratili iz srbijanskih logora. Da je stvarno bio genocid, nitko se ne bi vratio ¾iv. Ili bi ¾ivih barem bilo daleko manje.
+A jednako je bilo i sa Srbima u Oluji.
+?U Oluji je poèinjen genocid?, reèe mi nedavno jedan prijatelj, inaèe povjesnièar.
+?Gle, nije?, odgovorio sam mu ne¹to i i¾ivciran megalomanskim posezanjem za genocidom. ?U Oluji se dogodilo etnièko èi¹æenje.?
+Nije to, naravno, ne¹to na ¹to sam kao Hrvat specijalno ponosan, ali za ljubav istine mora se kazati kako je u opse¾noj akciji zauzimanja Knina, Drni¹a, Benkovca, Graèaca i stotina kraj¹kih i lièkih sela bilo razmjerno malo ¾rtava. Hrvatski vojni i politièki vrh razumno se odluèio za protjerivanje umjesto istrebljivanja neprijatelja i za to je, da se ne la¾emo i ne pravimo glupljim nego ¹to jesmo, imao pre¹utnu suglasnost srpskog politièkog i vojnog vrha koji je, podjednako razumno, shvaæao da vi¹e nema snage dr¾ati taj teritorij.
+Dva bliska pojma
+Mo¾e vam se èiniti da cjepidlaèim, reæi æete mo¾da da sam cinièan u inzistiranju na odvajanju dva bliska, srodna pojma kao ¹to su etnièko èi¹æenje i genocid, protjerivanje i istrebljivanje, no izmeðu njih zbilja stoji jedna va¾na, zapravo najva¾nija, egzistencijalna razlika, ona izmeðu biolo¹ki ¾ive i biolo¹ki mrtve èinjenice. Veæina je ipak pre¾ivjela, a meni je draga svaka od onih tisuæa i tisuæa glava na ramenima koje su odnijeli nesretni Srbi na traktorima, jednako kao ¹to me neopisivom sreæom ispunjava svaki onaj siroti Vukovarac koji se u masnicama vratio iz srbijanskog zarobljenièkog logora.
+Takvim su se opreznim, osjetljivim pravnim distinkcijama vodili oni suci koji su odluèili da genocida, bjesomuènog klanja ?do istrage va¹e ili na¹e?, kako je jedan ovdje prije mnogo godina precizno definirao genocid prije nego smo i znali za taj pojam, nije bilo. Ni srpskog nad Hrvatima ni hrvatskog nad Srbima. Pravo im budi, mudro i pravièno su odluèili. Poruèili su kako smo, istina, èinili stra¹ne i neèovjeène zloèine, kako smo nesumnjivo bili stoka prema svojim susjedima, ali ipak nismo onakva stoka kakvom se meðusobno do¾ivljavamo. I to je, ako mene pitate, odlièna vijest. Divna vijest. Slo¾it æu se bez ostatka s Antom Nobilom koji je prije nekoliko dana zakljuèio: ?Osobno sam izrazito zadovoljan ¹to moj narod nije stradao u genocidu, isto tako sam zadovoljan da moj narod nije poèinio genocid.?
+Svatko bi zdrave pameti, mislim, trebao tako razmi¹ljati. Genocid? Ma, bje¾i u vra¾ju mater, kome to treba?!
+Ali, na¾alost, treba. I treba mnogima. Narodu su drage neumjerene, stra¹ne rijeèi poput genocida, zloèina iznad svih zloèina, jer im je perverzna naslada druge prikazivati gorima nego ¹to jesu. Nacionalizam to stalno èini, njemu nijedna svinjarija nije dovoljno velika. Mora je napuhati do pucanja. Nacionalizam ima vampirsku prirodu, ovisan je o krvi i nikad mu je nije dosta. Nedavno odbaèena haa¹ka optu¾nica za genocid u tome je smislu vrlo nalik trudu onih srpskih pseudopovjesnièara, ¹ovinistièkih imbecila koji nisu bili zadovoljni èudovi¹no velikim brojem od sedamdeset tisuæa ubijenih u Jasenovcu, veæ su je morali pomno¾iti cijelih deset puta, na sedamsto tisuæa.
+Bez iluzija
+Ni o Hrvatima ni o Srbima, da se razumijemo, ja nemam mnogo iluzija, ali ih dr¾im mnogo moralnijima, ljudskijima nego ¹to oni jedni za druge dr¾e. Sretna je okonlost, osim toga, da oni, sve kad bi i htjeli, genocid ne bi znali izvesti. To je, brate, ne¹to za organiziranu, predanu i marljivu èeljad poput Japanaca ili Nijemaca, koji imaju tehnologiju, menad¾ment, koji æe znati rije¹iti trasportne, smje¹tajne, komunalne i druge bezbrojne nevolje s kojima æe se suoèiti svatko tko poku¹a takvo zama¹no, silno klanje. Genocid je Mercedes meðu zloèinima, a gdje bi Srbi to znali? Srbi su i Yugo Floridu zajebali. Jednaka je stvar i s Hrvatima. Koji put, vidite, nije ni lo¹e biti pripadnik takvog jednog traljavog, nemarnog, lijenog i zaostalog naroda.

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-2-czech Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-2-czech/_ude_1.txt Added

@@ -0,0 +1,7 @@
+U¾ týdny cloumá èeskou veøejností skandál s dìtmi, odebranými èeské matce norskou sociálkou. Péèe o dìti ale není jedinou oblastí, kde se zákony v této skandinávské zemi vymkly z kloubù zdravého rozumu. Nyní Norové zaèali potírat i ¾ebráky.
+Norsko, stejnì jako øadu dal¹ích zemí západní Evropy, trápí nájezdy gangù organizovaných ¾ebrákù, pøedev¹ím Romù z Rumunska a Bulharska. V devadesátých letech byly tyto gangy k vidìní u i nás - takové ty ¾enièky v ¹átku a s podvy¾iveným oblièejem, které mìly v uzlu na hrudi ?spící? (ve skuteènosti èasto nadopované prá¹ky) nemluvnì.
+Norský systém my¹lení, zøejmì zformovaný nedostatkem sluneèního svitu této severské zemì, vyprodukoval skuteènì revoluèní zpùsob boje proti ¾ebrákùm. Místo toho, aby policie tyto gangy øe¹ila jako skupiny organizovaného zloèinu, bude trestat lidi, kteøí nìco dají ¾ebrákovi.
+Chudí lidé úplnì na okraji jsou v ka¾dé spoleènosti. V Norsku, pøes vyspìlý sociální systém, ¾ebrají nejen gangy z Rumunska, ale také nor¹tí ztroskotanci, alkoholici, asociální schizofrenici a hranièáøi. Je celkem jedno, jestli jde o etnické Nory, nebo dlouhodobì usazené pøistìhovalce.
+Pøesto Norsko zavádí zákonnou úpravu, kdy bude nejen zakázáno ¾ebrat. Obdarovat ¾ebráka - a» u¾ penìzi nebo jídlem, a» u¾ rumunského organizovaného, nebo norského neorganizovaného - bude také trestným èinem, Dostat mù¾ete pokutu i pùlroèní vìzení. Podrobnì to u nás napøíklad popsal web iDnes.cz
+Pomáhat lidem v nouzi se tak stane zloèinem, charitativní jednání bude trestné, je proto otázkou, jak v kontextu toho zákona budou smìt pùsobit norské humanitární organizace zamìøené na bezdomovce. Chudoba se stane zloèinem, a je¹tì vìt¹ím zloèinem bude pomoc chudým. Suma sumárum: Norsko chce trestnì postihnout projevy soucitu, køes»anského milosrdenství a lidské vzájemnosti. Tedy projevy principù, na kterých stojí na¹e civilizace a jejich¾ dodr¾ování dostalo euroatlantický civilizaèní okruh k takové míøe humanity, svobody a blahobytu, jaká není nikde jinde na planetì. V Norsku z toho teï udìlají trestný èin.
+Myslím si, ¾e je to - podobnì jako politika norské sociálky - cesta k urychlenému konci na¹í civilizace.
\ No newline at end of file

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-2-czech/_ude_2.txt Added

@@ -0,0 +1,23 @@
+?Tìch policisté nakonec zajistili asi 30 kusù,? øekl iDNES.cz policejní mluvèí Tomá¹ Hulan. Dodal, ¾e v drtivé vìt¹inì ¹lo o padìlané karty amerických spoleèností, evropská mìla být jen jedna.
+
+?Tito mladíci nakupovali v centru Prahy v tìch nejluxusnìj¹ích obchodech. A to pøedev¹ím v Paøí¾ské ulici a blízkém okolí.
+
+Do Prahy pøitom pøiletìli minulou nedìli a prakticky ihned se vydali na nákupy,? popisoval mluvèí.
+
+Policisté se o jejich podezøelé poèínání dozvìdìli hned druhý den. Upozornila je na to firma, její¾ znaèku karet skupina padìlala.
+
+Spoleènosti se toti¾ nezdálo velké mno¾ství zamítnutých transakcí, které ale zaznamenala nejen v Praze, ale u¾ i pøed tím v Dubaji èi Øecku - tedy místech, kde skupinka na cestì do Èeska pøestupovala.
+
+Tuto ¹kodu ve stádiu pokusu kriminalisté následnì pøedbì¾nì vyèíslili na 5 a¾ 7 milionù.
+Byli jsme jen bílí konì, hájili se Asiaté
+
+Kriminalisté skupinku proto monitorovali, finální zátah pak pøi¹el o ètyøi dny pozdìji.
+
+?Dva z mu¾ù pomohli zadr¾et pracovníci bezpeènostní agentury v jednom z obchodù v Paøí¾ské ulici, tøetího pak v hotelu,? vysvìtlil mluvèí.
+
+Kromì padìlaných karet kriminalisté u mu¾ù zajistili hodinky za milion a dal¹í zbo¾í za statisíce vèetnì mobilních telefonù a jejich pøíslu¹enství.
+
+Mladíci se k trestné èinnosti pøiznali. ?Uvedli, ¾e k takovému jednání je vedla jejich tí¾ivá ¾ivotní situace. Pøistoupili na to, ¾e budou takzvaní bílí konì, fale¹nými platebními kartami je vybavil nìkdo jiný,? dodal Hulan.
+
+Zadr¾ení jsou nyní v policejní cele. Za trestné èiny neoprávnìné opatøení, padìlání a pozmìnìní platebního prostøedku a podvod jim hrozí a¾ osmileté vìzení.
+

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-2-hungarian/_ude_1.txt Added

@@ -0,0 +1,8 @@
+Nemrég mutattunk be egy 25 éves fiút, akinek gyakorlatilag gyógyító társaságra lenne szüksége, megfelelõ intézmény hiányában azonban az elmúlt egy évben erõs gyógyszeres terápiára kényszerült. Édesanyja szerint túl erõsre. A riportot itt olvashatja.
+A történet kapcsán pszichiátriai betegek érdekképviselõit kérdeztük a mai magyar pszichiátriai rendszerrõl. A Társaság a Szabadságjogokért fogyatékosügyi programvezetõje Verdes Tamás ( a továbbiakban: VT) és a Pszichiátriai Érdekvédelmi Fórum elnöke, Radó Iván ( a továbbiakban: RI) is úgy látja, a bajokat nem a Lipót bezárása okozza, nincs szükség egy nagy intézményre tele betegekkel, és nem is kórházi ágyban van hiány.
+Mi hiányzik?
+Kórosan alulfejlett rendszer jellemzi a pszichiátriát, a legtöbb beteg nem kap semmilyen szolgáltatást és egy nagyon intézet-centrikus ellátórendszerünk van. Vagyis ha valaki bajban van, nem kap segítséget vagy egy pszichiátriai intézetben, illetve szociális otthonban kap szinte kizárólag csak gyógyszeres terápiát. Beszélgetõs gyógyítás alig elérhetõ, a tébé minimális módon támogatja.
+Hadipszichiátriai zajlik. Nincs idõ. A magyar lakosság negyede érintett valami pszichiátriai problémával, a depresszió népbetegség, csak a szkizofrének száma egy százalék, ami nem kis szám. Irtózatos számú beteg van. A kórházi ellátásuk még csak megoldható, de a rendszeres pszichoterápiára, az egyénnek igazán megfelelõ gyógyszerbeállításokra már nem tudnak elég idõt szánni az orvosok.
+Járóbeteg ellátásban nagyjából két kérdést tesz fel az orvos a betegnek és felírja a gyógyszert, ennyire van idõ. Egyszer kiszámoltam, hogy egy betegre egy alkalommal fél óra jut, ha az orvos nem adminisztrál, nincs ebédszünete és nem megy ki vécére. Szóval nyilván nincs 20 perc sem, miközben legalább 45-60 perc lenne ideális, nyugodt körülmények között.
+Ha valaki valóban elképesztõ állapotban van és kórházban õt már jól kezelni nem hatékony, sokszor kimondatik, hogy betegotthonba kell mennie. Csakhogy a kettõ között, a kórháztól amíg elhelyezik, nincs semmi. Ilyenkor kapja a problémát a hozzátartozó vagy az utca. Vagy ilyenkor ? bár utálom, ha a média így stigmatizál ? lehet, hogy bûncselekményt követ el a beteg vagy õ maga lesz áldozat.
+Hol van országos rehabilitációs intézet? Ha leesik a lábam, az orrom, a nyakam, arra van. Pszichiátriában hol van ilyen? Pedig itt ez iszonyú fontos lenne. Nem elidegeníteni kell a betegeket, hanem integrálni a társadalomba. Nem védett munkahelyet kell adni, ahol simogatják a kis feje búbját, hanem önálló munkavégzést, olyat, ami neki jó. Arról nem beszélve, hogy vannak állami kedvezmények a megváltozott munkaképességûeknek, arra bárkit felvesznek, csak pszichiátriai beteget nem.
\ No newline at end of file

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-2-hungarian/_ude_2.txt Added

@@ -0,0 +1,12 @@
+Az önkormányzat által készíttetett szakvélemény szerint rétegvíz okozta a föld megcsúszását a pályaudvarnál.
+
+
+A végleges szakértõi jelentés szerint a rétegvíz okozta a rézsûcsúszást a Déli pályaudvari alagútnál, a helyreállítást a MÁV-nak kell elvégeznie. A megfelelõ intézkedések után néhány héten belül újraindulhat a vasúti közlekedés ? áll abban a jelentésben, melyet az újbudai önkormányzat juttatott el hétfõn az MTI-hez.
+
+A jelentést az önkormányzat felkérésére a BME út- és vasútépítési tanszékének szakértõi készítették. A jelentés szerint nem állapítható meg az újbudai önkormányzat mulasztása, annak oka az volt, hogy a Gellért-hegyre mint vízgyûjtõ területre hullott csapadékból rétegvíz keletkezett, a talajtömeg belsejében íves csúszólap alakult ki, mely felett a talaj elmozdult.
+
+Az alagút bejáratának közelében nagy számban találhatók felszíni mozgásokra utaló jelek, ezért mielõbb el kell kezdeni a helyreállítást, a késlekedés a ?suvadás? továbbterjedését okozhatja, ami a terület közmûveit és a környékbeli ingatlanok megközelítését is veszélyezteti. Magukat az épületeket azonban nem fenyegeti veszély ? tették hozzá.
+
+A szakvélemény szerint az alagút szerkezete valószínûleg nem sérült meg, a mellvédfal ideiglenes elbontásával és a föld további lecsúszásának meggátolásával a vasúti közlekedés néhány héten belül helyreállhat.
+
+A Déli pályaudvart január 31-én zárta le teljesen a MÁV.

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-2-hungarian/_ude_3.txt Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-2-polish Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-2-polish/_ude_1.txt Added

@@ -0,0 +1,14 @@
+Naukowcy zauwa¿yli, ¿e zmiana rytmu snu w dni wolne od pracy prowadzi do rozregulowania naszego wewnêtrznego zegara biologicznego. Zjawisko to nazwano "social jet lag", czyli spo³eczny jet lag.
+Badanie z udzia³em ponad 800 ochotników wykaza³o, ¿e osoby, które w dni wolne od pracy sypiaj± w innych godzinach ni¿ w dni robocze, s± bardziej nara¿one na oty³o¶ci i wynikaj±ce z niej choroby, ni¿ ci, którzy przez ca³y tydzieñ k³ad± siê spaæ i wstaj± mniej wiêcej w podobnych godzinach.
+W przeciwieñstwie do jet lagu, czyli zespo³u nag³ej zmiany strefy czasowej, polegaj±cego na desynchronizacji organizmu, która objawia siê m.in. chwilowymi zaburzeniami apetytu i problemami ¿o³±dkowo-jelitowymi, spo³eczny jet lag mo¿e przybraæ postaæ przewlek³± i towarzyszyæ nam przez ca³e ¿ycie zawodowe, dlatego jego wp³yw na metabolizm mo¿e byæ znacznie powa¿niejszy.
+- Zjawisko spo³ecznego jet lagu nie zosta³o jeszcze dok³adnie przebadane. Uwa¿amy jednak, ¿e jako czynnik wp³ywaj±cy na rozregulowanie zegara biologicznego, ma wp³yw na nasze zdrowie ? mówi autor badania, dr Michael Parsons, z wydzia³u genetyki ssaków przy Radzie ds. Badañ Medycznych, agencji rz±dowej odpowiedzialnej za koordynacjê i finansowanie badañ medycznych w Wielkiej Brytanii.
+Czas na zdrowie: Jak walczyæ z oty³o¶ci± - Czy kilka zbêdnych kilogramów to ju¿ oty³o¶æ? Kiedy rzeczywi¶cie powinni¶my zacz±æ siê martwiæ nasz± wag±? A je¿eli ju¿ mamy z ni± problem, jak skutecznie zrzuciæ kilogramy? - Onet.
+- Nasz eksperyment potwierdzi³ wnioski z wcze¶niejszego badania, które dowiod³o, ¿e istnieje zwi±zek miêdzy ostr± form± spo³ecznego jet lagu, a wzrostem BMI. Nasze badanie po raz pierwszy wskaza³o, ¿e zmiany rytmu snu zwiêkszaj± zagro¿enie chorobami zwi±zanymi z oty³o¶ci±.
+Naukowcy zmierzyli wzrost, wagê i obwód w pasie uczestników badania, przeanalizowali te¿ biomarkery stanów zapalnych i cukrzycy w krwi. Wyniki porównano z danymi z kwestionariusza na temat d³ugo¶ci snu i chronotypu, czyli typu aktywno¶ci oko³odobowej.
+Badacze stwierdzili, ¿e ju¿ ró¿nica dwóch godzin w rytmie snu w dni wolne od pracy mo¿e podwy¿szaæ zagro¿enie wzrostem BMI i zwiêkszaæ obecno¶æ biomarkerów stanów zapalnych i cukrzycy.
+Od nadwagi siê zaczyna, a gdy nadwaga przekszta³ca siê w oty³o¶æ, nie tylko jest ci ciê¿ej ¿yæ, dopada ciê tak¿e wiele powa¿nych chorób i dolegliwo¶ci, w tym. m.in. problemy z uk³adem kra¿enia, cukrzyca, zwyrodnienie stawów. Do jakich jeszcze chorób prowadzi oty³o¶æ?
+Przyczyny tego wzrostu na razie s± nieznane, ale wiele wskazuje, ¿e spo³eczny jet lag zaburza zdrowe nawyki, takie jak dieta i gimnastyka, na tyle, ¿e odbija siê to na zdrowiu.
+- Badanie pomo¿e nam zrozumieæ fizjologiê spo³ecznego jet lagu i zwi±zek tego zjawiska z oty³o¶ci± i chorobami wynikaj±cymi z oty³o¶ci ? t³umaczy dr Terrie Moffitt, wspó³autorka badania. - Dalsze badania mog± wp³yn±æ na politykê zapobiegania oty³o¶ci, poprzez zmiany praktyk prowadz±cych do spo³ecznego jet lagu, takich jak harmonogram pracy czy zmiana czasu na letni.
+Profesor David Lomas z Rady ds. Badañ Naukowych zauwa¿a: - To badanie przynosi kolejne dowody na to, ¿e funkcjonowanie niezgodne z zegarem biologicznym, nawet na bardzo ma³± skalê, mo¿e po czê¶ci przyczyniaæ siê do wzrostu oty³o¶ci i zwi±zanych z ni± chorób.
+- Nale¿a³oby przyjrzeæ siê w jaki sposób zmiany w rozk³adzie dnia i diecie mog± zmniejszaæ ten szkodliwy trend i wp³ywaæ na poprawê zdrowia publicznego - dodaje.
+

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-2-slovak Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-2-slovak/_ude_1.txt Added

@@ -0,0 +1,9 @@
+Tisíce firiem na Slovensku neplatia ¾iadnu daò z príjmu. Vyplýva to zo zoznamu daòových subjektov s vý¹kou ich splatnej dane za rok 2013, ktorý na základe údajov z registra úètových závierok prvýkrát zverejnil rezort financií.
+Zoznam má a¾ 1 408 strán. Kým viac ako 10 mil. eur na dani z príjmov pritom do rozpoètu za rok 2013 odviedlo podµa zoznamu 16 spoloèností, tisíce firiem neodviedli niè alebo len symbolických pár eur.
+Zmeni» by sa to malo a¾ za rok 2014. V òom toti¾ u¾ platili daòové licencie, ktoré sa vlani zaviedli. Vláda stanovila, ¾e firmy musia plati» za licenciu 480 a¾ 2 880 eur aj v prípade, ak nevyká¾u zisk. V prípade kladnej daòovej povinnosti si licenciu z nej mô¾u odpoèíta». Zavedenie licencií vláda vysvetµovala tým, ¾e podnikatelia si pri kreatívnom úètovníctve vedia zni¾ova» základ dane tak, aby daò neplatili. Daòová kontrola na to prís» mô¾e, ale pri niekoµkých státisícoch firiem kontrolóri nestíhajú preveri» v¹etky.
+Zoznam firiem a odvedených daní.
+Najvy¹¹í objem dane z príjmov za rok 2013 do rozpoètu odviedli eustream, Kia a telekomunikaèné spoloènosti Slovak Telekom a Orange Slovensko. Eustream odviedol 97,6 milióna eur a Kia Motors Slovakia 83,4 milióna eur. V prípade Slovak Telekomu to bolo takmer 46 mil. eur a pri Orangeu Slovensko 42,4 mil. eur. V zozname chýbajú niektoré veµké firmy, ktoré by mali patri» medzi najväè¹ích platcov dane. Ministerstvo u¾ raz svoj zoznam opravilo, v prvej verzii v òom chýbali eustram aj Kia.
+Medzi firmami, ktoré odviedli viac ako 10 mil. eur na dani z príjmov, nechýbali napríklad Slovenská sporiteµòa, Tatra banka, Nafta, Continental Matador, Allianz ? Slovenská pois»ovòa, ale ani energetické distribuèné spoloènosti èi firma Eset, ktorá odviedla na dani z príjmov takmer 17,7 mil. eur. Len týchto prvých 16 firiem zo zoznamu platiteµov dane z príjmov pritom do rozpoètu prispelo celkovou sumou 417,5 mil. eur. Na druhej strane rebríèka, ktorý obsahuje vy¹e 73-tisíc záznamov, sa v¹ak nachádza takmer 32-tisíc spoloèností, ktorých daòová povinnos» na dani z príjmov za rok 2013 nedosiahla ani 500 eur.
+"Na¹ím cieµom je takýmto zoznamom zvý¹i» informovanos» verejnosti o tom, ako firmy na Slovensku platia dane, a prispie» k väè¹ej daòovej transparentnosti podnikateµského prostredia,? povedal na margo zverejnených údajov minister financií Peter Ka¾imír. Tento zoznam ministerstvo zverejòuje na základe novely zákona o správe daní platnej od 1. januára 2015. Zoznam pozostáva z verejne dostupných údajov registra úètovných závierok za rok 2013 a uvádza sa v òom obchodné meno alebo názov právnickej osoby, jej sídlo, identifikaèné èíslo a vý¹ka splatnej dane. Firmy sú zoradené podµa vý¹ky splatnej dane od najvy¹¹ej po najni¾¹iu.
+Ministerstvo financií pritom tvrdí, ¾e zoznamy o tom, ako firmy platia dane, bude aktualizova» z dostupných údajov na ¹tvr»roènej báze. Údaje budú zohµadòova» posledné dostupné úètovné obdobie.
+Daò z príjmov je podstatnou daòou na Slovensku. Z celkových vlaòaj¹ích príjmov ¹tátneho rozpoètu vo vý¹ke 12,5 mld. eur tvorili dane z príjmov fyzických a právnických osôb sumu 2,1 mld. eur. Pritom z dane z pridanej hodnoty ¹tát zinkasoval 4,9 miliardy eur a zo spotrebných daní 2 miliardy eur.
\ No newline at end of file

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-2-slovak/_ude_2.txt Added

@@ -0,0 +1,7 @@
+Ako informuje talianska televízna stanica TGcom24 na svojej internetovej stránke, do pondelka sa podarilo zachráni» 2 164 imigrantov na dvanástich plavidlách. Niektorých zachránila talianska vojenská loï, ïal¹ích lode pobre¾nej hliadky a talianskej colnej polície ako aj iné komerèné lode v oblasti.
+
+Talianske ministerstvo dopravy informovalo, ¾e pobre¾nú strá¾ ohrozovali zbraòami ¹tyria ozbrojení mu¾i, ktorí pri¹li na motorovom èlne od líbyjského pobre¾ia. Prinútili ich vráti» im loï, z ktorej vystúpili uteèenci.
+
+Minulý rok si nebezpeèná plavba z afrického pobre¾ia do Európy vy¾iadala najmenej 3200 ¾ivotov. Len minulý tý¾deò zahynulo v Stredozemnom mori viac ne¾ 300 imigrantov, ktorí sa v rozbúrenom mori plavili v nafukovacích èlnoch. Talianska pobre¾ná strá¾ minulý tý¾deò zachránila z mora asi 600 uteèencov.
+
+Tieto poèty ukazujú na nutnos» posilni» európsku misiu Triton, ktorá v novembri nahradila taliansku námornú operáciu Mare Nostrum zameranú na záchranu uteèencov. Talianska ministerka obrany Roberta Pinottiová nevylúèila, ¾e s imigrantmi sa na talianske pobre¾ie, a tým aj do Európy, mô¾u dosta» islamskí radikáli.
\ No newline at end of file

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-2-slovene Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-2-slovene/_ude_1.txt Added

@@ -0,0 +1,14 @@
+Zavod RS za zaposlovanje je na svojih spletnih straneh vzpostavil novo rubriko Skupaj do zaposlitve, v okviru katere bodo uporabnikom na voljo primeri dobrih praks oz. uspe¹nih zgodb brezposelnih oseb, iskalcev zaposlitve in delodajalcev iz vse Slovenije.
+Kot ekonomski tehnik ni na¹la slu¾be, bilo pa je povpra¹anje po varilcih in ....
+LJUBLJANA ? Zavod RS za zaposlovanje je na svojih spletnih straneh vzpostavil novo rubriko Skupaj do zaposlitve, v okviru katere bodo uporabnikom na voljo primeri dobrih praks oz. uspe¹nih zgodb brezposelnih oseb, iskalcev zaposlitve in delodajalcev iz vse Slovenije. Trenutno so v rubriki objavljene tri zgodbe. Svoje izku¹nje so javnosti zaupali Branko Ileniè, Darja Avgu¹tinèiè in Mojca Rupert.
+Slu¾ba v Esolu
+Ileniè pojasnjuje, da je leta 2012 pristal na cesti po 30 letih dela. ?Star sem bil 47 let, z zdravstvenimi omejitvami in na trgu dela za mnoge delodajalce nezanimiv za zaposlitev. Poleg tega je bila brezposelna tudi zakonska partnerka. Dru¾ina z dvema ¹olajoèima otrokoma je ostala brez sredstev za pre¾ivetje,? se spominja. Branko je bil aktiven iskalec zaposlitve, zavod pa ga je v zaèetku lanskega leta napotil na razgovor v podjetje Esol v Èrnomlju, ki je ravno takrat potrebovalo delavce za delo v proizvodnji. S subvencijo, ki jo je delodajalec prek zavoda dobil za njegovo zaposlitev, je konèno dobil delo.
+Esol je lani prek razliènih razpisov zavoda omogoèil delo veè delavcem: prek Delovnega preizkusa za mlade petim, prek Zaposli me ¹estim in prek Zaposli me A ¹tirim.
+Slu¾ba pri Akrapovièu
+Avgu¹tinèièeva medtem opisuje svoje spoznanje, da je varjenje tudi poklic za ¾enske. Kot pravi, je po poklicu ekonomski tehnik, a v èasu, ko je iskala delo, za ta poklic ni bilo povpra¹evanja, zato je zaèela razmi¹ljati o drugih zaposlitvenih mo¾nostih. ?Ker je trenutno na trgu dela veliko povpra¹evanje po varilcih, sva se skupaj s svetovalko zaposlitve odloèili, da grem na teèaj TIG varjenja. Vsi, ki sem jim omenila, v kaj se spu¹èam, so me zaèudeno gledali in govorili, da je to mo¹ki poklic, in me spra¹evali, èemu mi bo ta teèaj koristil. Ker postopka TIG varjenja prej nisem poznala, sem o svoji odloèitvi zaèela dvomiti ¹e sama, a sta radovednost in velika ¾elja po zaposlitvi premagali vse moje dvome,? navaja.
+Po opravljenem teèaju je bila povabljena na razgovor pri podjetju Akrapoviè, kmalu nato pa je dobila klic, da je izbrana.
+Slu¾ba na univerzi
+?V svoji kratki 'karieri' brezposelnosti sem se med kopico kraj¹ih in dalj¹ih brezplaènih in plaèljivih izobra¾evanj med drugim udele¾ila tudi BTC Campusa 2014. Vabilo s strani zavoda za zaposlovanje je zvenelo nekako tako: 'Udele¾ite se dodatnega podjetni¹kega izobra¾evanja za iskalce zaposlitve.' Poziv je veliko obetal, realizacija pa je bila ¹e bolj¹a,? pa svojo izku¹njo predstavlja Rupertova, diplomirana politologinja, sedaj zaposlena na Centru za univerzitetni ¹port na Univerzi v Ljubljani. Mlad èlovek se mora pri iskanju dela po njenih besedah poslu¾evati predvsem podjetni¹kih naèel. ?Pri tem se je treba vpra¹ati, kak¹en je moj doprinos k podjetju oziroma k organizaciji, kjer se ¾elim zaposliti, in kdo ima veèji problem pri zaposlovanju, delodajalec ali iskalec zaposlitve,? svetuje.
+Trimeseèno izobra¾evanje je po njenih besedah temeljilo na razvoju ve¹èin, naèina podjetni¹kega mi¹ljenja in empatije, udele¾enci pa so pridobili tudi nekatera osnovna poslovna znanja, kot so raèunovodstvo, marketing itd. ?Vse to smo poèeli neobremenjeno, pogosto celo preko igre. Ravno tu se je izkazalo, da lahko 'nepopisan list' pogosto poi¹èe veliko bolj¹e re¹itve, kot znanja polna glava,? zakljuèuje Rupertova.
+Pozitivni pogled na brezposelnost
+Zavod se je za objavljanje pozitivnih zgodb odloèil, ker ¾eli prispevati k bolj pozitivnemu gledanju na obdobje brezposelnosti. Poleg tega ¾eli s tem spodbuditi iskalce zaposlitev in delodajalce.
\ No newline at end of file

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-5-russian Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-5-russian/_chromium_ISO-8859-5_with_no_encoding_specified.html Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-5-russian/aif.ru.health.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-5-russian/aug32.hole.ru.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-5-russian/aviaport.ru.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-5-russian/blog.mlmaster.com.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-5-russian/forum.template-toolkit.ru.1.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-5-russian/forum.template-toolkit.ru.4.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-5-russian/forum.template-toolkit.ru.6.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-5-russian/forum.template-toolkit.ru.8.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-5-russian/forum.template-toolkit.ru.9.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-5-russian/greek.ru.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-5-russian/intertat.ru.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-5-russian/janulalife.blogspot.com.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-5-russian/kapranoff.ru.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-5-russian/money.rin.ru.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-5-russian/music.peeps.ru.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-5-russian/newsru.com.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-5-russian/susu.ac.ru.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-7-greek/_ude_1.txt Added

@@ -0,0 +1,7 @@
+¸íáò äéáäéêôõáêüò ìáò ößëïò, ôïí ïðïßï èá Ý÷ïõìå õðïäåßîåé åê ôùí ðñïôÝñùí, èá Ý÷åé ôï äéêáßùìá íá åðåîåñãáóôåß ôï ðñïößë ìáò, áöïý ðåèÜíïõìå
+Ìðïñåß ôï áßôçìá ãéá êïõìðß «dislike» ðïõ áðáéôïýí ïé ÷ñÞóôåò åäþ êáé êáéñü, ôï facebook íá ìçí ôï éêáíïðïéåß, ðñï÷ùñÜ üìùò óôçí éêáíïðïßçóç åíüò Üëëïõ áéôÞìáôïò, ðïõ üðùò ëÝíå ïé õðåýèõíïé ôïõ êïéíùíéêïý äéêôýïõ, ôïõò áðáó÷üëçóå ãéá êáéñü. 
+Ôï ðñüâëçìá Þôáí ôé ãßíåôáé ìå ôïí ëïãáñéáóìü åíüò áôüìïõ, óå ðåñßðôùóç ðïõ ðåèÜíåé. Êáíåßò, ïýôå ç ïéêïãÝíåéá, äåí Ý÷åé ôï äéêáßùìá íá åðåîåñãáóôåß ôï ðñïößë, ãåãïíüò ðïõ îåóÞêùóå áíôéäñÜóåéò óå äéÜöïñåò ðåñéðôþóåéò. 
+Ðïéá åßíáé ç ëýóç; Ï êáèÝíáò áðü üóïõò äéáôçñïýí ðñïößë óôï facebook, èá Ý÷åé óýíôïìá ôç äõíáôüôçôá íá êÜíåé ìéá éäéüôõðç «çëåêôñïíéêÞ äéáèÞêç», ìå ôçí ïðïßá èá ïñßæåé Ýíáí áðü ôïõò äéáäéêôõáêïýò ôïõ ößëïõò ùò «êëçñïíüìï». Ï ößëïò áõôüò èá Ý÷åé ôï äéêáßùìá íá áëëÜîåé ôçí öùôïãñáößá ðñïößë êáé ôçí öùôïãñáößá åîùöýëëïõ, áëëÜ êáé íá áíáíåþóåé ôï óôÜôïõò ôïõ èáíüíôïò ãéá ìßá êáé ìüíï, ôåëåõôáßá öïñÜ. 
+Åðßóçò, ï «êëçñïíüìïò» èá ìðïñåß íá ðñïóèÝôåé êáéíïýñéïõò ößëïõò, ìå ôï íá áðïäÝ÷åôáé áéôÞìáôá öéëßáò ðïõ Ý÷ïõí ãßíåé ðñïò ôï ðñïößë.
+Ôçí óôéãìÞ ðïõ ôï facebook âåâáéþíåé ôï èÜíáôï êÜðïéïõ, èá ðñïóôßèåôáé óôï õðÜñ÷ïí ðñïößë ç Ýíäåéîç «åéò ìíÞìçí» (remembering óôá áããëéêÜ) êáé áõôüìáôá èá åíçìåñþíåôáé ï «êëçñïíüìïò», ðïõ ðëÝïí èá áðïêôÜ äéêáßùìá åðåîåñãáóßáò ôïõ ðñïößë.
+Çäç áðü óÞìåñá, ÷ñÞóôåò óôéò ÇÐÁ Ý÷ïõí ôç äõíáôüôçôá áõôÞ, ðïõ ðÜíôùò Ý÷åé îåóçêþóåé êáé èýåëëá áíôéäñÜóåùí. Ðïëëïß áíáñùôéïýíôáé ãéá ðáñÜäåéãìá, ôé èá ãßíåé áí «êëçñïíüìïò» åßíáé Ýíáò ðñþçí óýíôñïöïò, ðïõ åí ôù ìåôáîý Ý÷åé å÷èñéêÜ áéóèÞìáôá ðñïò ôïí áðïèáíüíôá. 
\ No newline at end of file

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-7-greek/_ude_2.txt Added

@@ -0,0 +1,7 @@
+ÄùñåÜí óôï Ïé ¶íåìïé, ôï ÌÞíõìá, êáé ï Êþäéêáò. ¹, Ðþò ï Áñßóôáñ÷ïòìïñ, ÐñÜêôïñáò ôçò ÅðáíÜóôáóçò, ÌÜãïò ôïõ ÔÜãìáôïò ôùí Ôå÷íïìáèþí, êáé ÔáîéäåõôÞò ôïõ Ðïñöõñïý Êåíïý êáé Ðïëëþí ÄéáóôÜóåùí,
+ÁíáêÜëõøå ìßá Ðëçñïöïñßá ÐëÝïí ÓçìáíôéêÞ ãéá ôçí ÅðáíÜóôáóç. Åßìáé óôçí ÁãêÝíñïâ, óôç ÓåñãÞëç, óÝíá ìðáñ. ¸÷ù ðáñáäþóåé êÜôé ìçíýìáôá óå åðáíáóôÜôåò, áíèñþðïõò ðéóôïýò óôïí ÐñßãêéðÜ ìáò.
+ÊÜèïìáé êáé ðßíù Ýíáí Öëåãüìåíï Ãñýðá. Äåí Ý÷åé ðïëý êüóìï. Öáóáñßá áñ÷ßæåé íá ãßíåôáé áðü Ýíá ðëáúíü äùìÜôéï: êÜðïéïò óêïýæåé óá íá ôïí óöÜæïõí. Ï ôýðïò ðßóù áð¢ôïí ðÜãêï ôïõ ìðáñ ðçãáßíåé ðñïò ôá êåé,
+ôï ßäéï êáé ìéá óåñâéôüñá. ÏñéóìÝíïé áðü ôïõò ëéãïóôïýò ðåëÜôåò óçêþíïíôáé, ðÜíå íá êïéôÜîïõí êÜðïéïé Üëëïé ìïéÜæïõí Ýôïéìïé íá ôçí êïðáíÞóïõí, ìçí ôõ÷üí êé åßíáé êßíäõíïò.
+ÐáñáîåíåìÝíïò, ðçãáßíù êé åãþ íá ñßîù ìéá ìáôéÜ. Ìåò óôï ìéêñü äùìÜôéï âëÝðù Ýíáí ôýðï îáðëùìÝíï ðÜíù ó¢Ýíá êñåâÜôé, áíÜóêåëá, íá ÷ôõðéÝôáé ï Üíôñáò áðü ôï ìðáñ ðñïóðáèåß íá ôïõ êñáôÞóåé ôá ÷Ýñéá,
+ëÝåé óôç óåñâéôüñá «ÂÜëôïõ êÜôé óôï óôüìá, ÌÜñá, èá ðíéãåß!» Ç êïðÝëá ? ôï ãáëáíü ôçò äÝñìá Ý÷åé ãßíåé ó÷åäüí Üóðñï áð¢ôçí áíçóõ÷ßá ôçò ? ðñïóðáèåß íá ôïõ âÜëåé Ýíá îýëéíï ìáñáöÝôé áíÜìåóá óôá
+óáãüíéá áëëÜ äå öáßíåôáé íá ìðïñåß.

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-7-greek/_ude_3.txt Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-9-turkish/_ude_1.txt Added

@@ -0,0 +1,8 @@
+Politik amaçlý "hack" eylemi gerçekleþtiren Anonymous, terör örgütü IÞÝD'e savaþ ilan etti. ABD baþta olmak üzere birçok Batýlý ülke, IÞÝD'in sosyal paylaþým sitelerindeki hesaplarýný engelleyemezken, Anonymous IÞÝD'e ait 100 hesabý hack'ledi.
+Suriye ve Irak'ta etkili olan terör örgütü IÞÝD propagandasýnýn önemli bir kýsmýný Twitter ve Facebook gibi sosyal paylaþým siteleri aracýlýðýyla yapýyor.
+IÞÝD'in Twitter ve Facebook üzerindeki hesaplarýna müdahale edilmemesi tepkilere neden olmuþ, baþta ABD olmak üzere Batýlý ülkeler eleþtirilmiþti.
+IÞÝD'in þimdiye kadar engellenemeyen ve tweet'leri silinemeyen 100 hesabý, politik amaçlý "hack" eylemi gerçekleþtiren Anonymous tarafýndan saldýrýya uðradý.
+IÞÝD'in sosyal medyadaki hesaplarýný hedef aldýðýný yayýnladýðý bir videoyla duyuran Anonymous, IÞÝD'e karþý düzenlenen saldýrýnýn Müslüman, Hristiyan ve Yahudi hacker'lar tarafýndan gerçekleþtirildiðini açýkladý.
+Anonymous siber alemde IÞÝD'e karþý savaþ ilan ettiðini duyurdu.  
+Anonymous'un açýklamasýnda þöyle denildi: "IÞÝD, sizi her yerde avlayacaðýz, internet sitelerinizi, sosyal medya hesaplarýnýzý, elektronik postalarýnýzý yok edeceðiz. Ýnternet dünyasýnda sizin için güvenli bir alan olmayacak. Virüs gibi muamele göreceksiniz ve bunun ilacý biz olacaðýz."
+IÞÝD'in Twitter ve Facebook'ta saldýrýya uðrayan hesaplarýnýn listesi de, Anonymous tarafýndan yayýnlandý.
\ No newline at end of file

_service:tar_scm:chardet-5.1.0.tar.gz/tests/iso-8859-9-turkish/_ude_2.txt Added

@@ -0,0 +1,12 @@
+Taþta kan vardý. Bahçede ürkütücü bir serinlik. Cinayetin tek tanýðý dolunaydý. Hiç þaþýrmadan, ürpermeden, korkmadan bakýyordu uzun boylu kavak aðaçlarýnýn ölü
+yapraklarýnýn arasýndan. Yedi kiþiden en genç olaný vurmuþtu kapýya. En yaþlý olaný çaðýrmýþtý içeridekini. Yedi kiþinin yedisi birden saplamýþtý býçaklarýný içeriden çýkana. Taþta kan vardý, insanlarýn yüreklerinde nefret, dolunayda derin bir sükunet. Bir bebek aðlýyordu uzaklarda bir yerlerde, bir bebek kýpýrdanýyordu evlerden birinde. Genç
+bir kýz uyuyordu uzaklarda, genç bir kýzýn bedeni aðýr aðýr çürüyordu topraðýn altýnda. Yedi kiþiden en genç olaný
+saplarken býçaðý adama, kýpýrdandý mezarda çürümekte olan genç kýzýn körpe bedeni. Bir gülümseme yayýldý ölümün bile örseleyemediði yüzüne. Yedi kiþiden en genç olaný, saplarken býçaðý, bir oh çýktý genç kýzýn boðazýnda düðümlenip kalmýþ son nefesinden.
+Taþta kan vardý, yedi býçak, yedi yara açmýþtý. Yedi kýzýl fýskiye. Yedi kez sarsýlmýþtý adam, yedi kez sarsýlmýþtý
+býçaðý saplayan yedi kiþi. Ama yerin altýndaki kýzýn körpe bedeni kýpýrdamýyordu artýk. Genç kýzýn bedeni gibi yerin üstü de sessizdi þimdi. Sanki dünyanýn son vaktiymiþcesine canlý cansýz ne kadar mahlukat varsa susmuþ, kýpýrtýsýz kalmýþtý. Taþtaki kan kýpýrtýsýzdý. Taþtaki
+kanýn içinde sönmekte olan dolunay kýpýrtýsýzdý. Uzun boylu kavaklar katmerlenen kýþ gülleri, tazelenen nergisler, toprak kokulu bahçe... Canlý cansýz ne kadar mahlukat varsa hepsi susmuþ, hepsi hapsolmuþtu taþtaki kanýn içinde...
+"... bozkýrýn içinden bir þehir çýkývermiþti karþýma."
+Uçaðýn iniþe geçmesine sadece yarým saat kalmýþtý, ama bu bile gidermiyordu içimdeki
+huzursuzluðu. Çok iyi biliyordum ki, indiðim yerde de býrakmayacaktý bu karamsarlýk yakamý.
+Keþke bu iþi hiç kabul etmeseydim. Kendini yeryüzünün en iyi yöneticisi sanan Simon'un iþgüzarlýðý iþte. Yok Türkçe biliyormuþum da, yok
+Türkleri tanýyormuþum da... Dava da herkese verilmeyecek kadar önemliymiþ. Üç milyon paundluk bir poliçe söz konusuymuþ. Keþke hiç tanýmasaydým Türkleri, keþke bu kente daha önce hiç gelmeseydim. Sýkýntýyla ofladým ama oflamanýn puflamanýn hiçbir yararý yoktu, olan olmuþtu; bu da ötekiler gibi sadece bir iþti. Altý ay önce gittiðim Rio gezisinden ne farký vardý ki? Üstelik Brezilyalýlar hakkýnda hiçbir þey bilmiyordum. Ama en azýndan bu ülkenin çok da yabancýsý deðildim. Evet, artýk kendimi iþime vermeliydim. Bakýþlarýmý

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1250-croatian Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1250-croatian/_ude_1.txt Added

@@ -0,0 +1,17 @@
+Eto, ni genocid nije to je nekad bio. Iznenada su pootrili kriterije, nepoteno podigli letvicu, i mi se nismo kvalificirali. Haaka presuda kako Srbi nisu napravili genocid nad Hrvatima, a ni obrnuto, ogorèila je mnoge. Stvar je zavrila kao i inaèe kad se negdje moramo meðunarodno dokazivati, jednako kao nogometno prvenstvo u Brazilu ili rukometno u Kataru. U jednu rijeè, suci su nas pokrali.
+Vijesti
+Ovo je gotovo jednoduan zakljuèak. Politièari se redom, i u vlasti i u opoziciji, trse koji æe djelovati zgranutije, ojaðenije, kao kazalini amateri potreseno se hvataju za srce, a novine se natjeèu u gnjevu za Haag koji je ponovno ravnoduan za nau agoniju devedesetih, no sve je to zapravo jedna kukavna i prijetvorna predstava jer drugaèiji ishod nismo ni oèekivali. Svakome je iole upuæenijem otpoèetka bilo jasno kako je ta optunica samo lakomisleno, neuko politièko razmetanje za domaæu upotrebu, proziran blef, mahanje praznom pukom. Desetke milijuna potroili smo na jednu uzaludnu sudsku bitku koja je samo trebala nahraniti nae nacionalistièke iluzije.
+Za ljubav istine
+Napokon, poèetkom tjedna obistinilo se ono na to su trezveniji odvjetnici godinama upozoravali. U Haagu su presudili kako genocida ovdje nije bilo. I zaista, po dui govoreæi, nije. Genocid, koji bi se na na jezik najblie mogao prevesti kao istrebljenje, je velika i teka rijeè koju ne valja olako izgovarati. Genocid je zloèin iznad svih zloèina. Razaranje Vukovara bila je krvava, ljudski sramotna i vojnièki neèasna opsada jednog grada do njegovog potpunog iscrpljivanja, i dogodilo se tu kojeta protiv zakona i boljih ratnih obièaja, masovno i neselektivno bombardiranje kuæa, ubijanje civila, hladnokrvno strijeljanje ranjenika na Ovèari, ali sve je to, veæ i samo po sebi dovoljno uasno, opet teko ugurati u definiciju da se u cijelosti ili djelomièno uniti jedna nacionalna, etnièka, rasna ili vjerska skupina.
+Koridorom izmeðu blatnjavih rovova i tenkova tuna je kolona prognanika naposljetku izala iz Vukovara, a veæina zarobljenika su se nakon nekoliko mjeseci premlaæeni vratili iz srbijanskih logora. Da je stvarno bio genocid, nitko se ne bi vratio iv. Ili bi ivih barem bilo daleko manje.
+A jednako je bilo i sa Srbima u Oluji.
+U Oluji je poèinjen genocid, reèe mi nedavno jedan prijatelj, inaèe povjesnièar.
+Gle, nije, odgovorio sam mu neto i iivciran megalomanskim posezanjem za genocidom. U Oluji se dogodilo etnièko èiæenje.
+Nije to, naravno, neto na to sam kao Hrvat specijalno ponosan, ali za ljubav istine mora se kazati kako je u opsenoj akciji zauzimanja Knina, Drnia, Benkovca, Graèaca i stotina krajkih i lièkih sela bilo razmjerno malo rtava. Hrvatski vojni i politièki vrh razumno se odluèio za protjerivanje umjesto istrebljivanja neprijatelja i za to je, da se ne laemo i ne pravimo glupljim nego to jesmo, imao preutnu suglasnost srpskog politièkog i vojnog vrha koji je, podjednako razumno, shvaæao da vie nema snage drati taj teritorij.
+Dva bliska pojma
+Moe vam se èiniti da cjepidlaèim, reæi æete moda da sam cinièan u inzistiranju na odvajanju dva bliska, srodna pojma kao to su etnièko èiæenje i genocid, protjerivanje i istrebljivanje, no izmeðu njih zbilja stoji jedna vana, zapravo najvanija, egzistencijalna razlika, ona izmeðu bioloki ive i bioloki mrtve èinjenice. Veæina je ipak preivjela, a meni je draga svaka od onih tisuæa i tisuæa glava na ramenima koje su odnijeli nesretni Srbi na traktorima, jednako kao to me neopisivom sreæom ispunjava svaki onaj siroti Vukovarac koji se u masnicama vratio iz srbijanskog zarobljenièkog logora.
+Takvim su se opreznim, osjetljivim pravnim distinkcijama vodili oni suci koji su odluèili da genocida, bjesomuènog klanja do istrage vae ili nae, kako je jedan ovdje prije mnogo godina precizno definirao genocid prije nego smo i znali za taj pojam, nije bilo. Ni srpskog nad Hrvatima ni hrvatskog nad Srbima. Pravo im budi, mudro i pravièno su odluèili. Poruèili su kako smo, istina, èinili strane i neèovjeène zloèine, kako smo nesumnjivo bili stoka prema svojim susjedima, ali ipak nismo onakva stoka kakvom se meðusobno doivljavamo. I to je, ako mene pitate, odlièna vijest. Divna vijest. Sloit æu se bez ostatka s Antom Nobilom koji je prije nekoliko dana zakljuèio: Osobno sam izrazito zadovoljan to moj narod nije stradao u genocidu, isto tako sam zadovoljan da moj narod nije poèinio genocid.
+Svatko bi zdrave pameti, mislim, trebao tako razmiljati. Genocid? Ma, bjei u vraju mater, kome to treba?!
+Ali, naalost, treba. I treba mnogima. Narodu su drage neumjerene, strane rijeèi poput genocida, zloèina iznad svih zloèina, jer im je perverzna naslada druge prikazivati gorima nego to jesu. Nacionalizam to stalno èini, njemu nijedna svinjarija nije dovoljno velika. Mora je napuhati do pucanja. Nacionalizam ima vampirsku prirodu, ovisan je o krvi i nikad mu je nije dosta. Nedavno odbaèena haaka optunica za genocid u tome je smislu vrlo nalik trudu onih srpskih pseudopovjesnièara, ovinistièkih imbecila koji nisu bili zadovoljni èudovino velikim brojem od sedamdeset tisuæa ubijenih u Jasenovcu, veæ su je morali pomnoiti cijelih deset puta, na sedamsto tisuæa.
+Bez iluzija
+Ni o Hrvatima ni o Srbima, da se razumijemo, ja nemam mnogo iluzija, ali ih drim mnogo moralnijima, ljudskijima nego to oni jedni za druge dre. Sretna je okonlost, osim toga, da oni, sve kad bi i htjeli, genocid ne bi znali izvesti. To je, brate, neto za organiziranu, predanu i marljivu èeljad poput Japanaca ili Nijemaca, koji imaju tehnologiju, menadment, koji æe znati rijeiti trasportne, smjetajne, komunalne i druge bezbrojne nevolje s kojima æe se suoèiti svatko tko pokua takvo zamano, silno klanje. Genocid je Mercedes meðu zloèinima, a gdje bi Srbi to znali? Srbi su i Yugo Floridu zajebali. Jednaka je stvar i s Hrvatima. Koji put, vidite, nije ni loe biti pripadnik takvog jednog traljavog, nemarnog, lijenog i zaostalog naroda.

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1250-czech Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1250-czech/_ude_1.txt Added

@@ -0,0 +1,7 @@
+U týdny cloumá èeskou veøejností skandál s dìtmi, odebranými èeské matce norskou sociálkou. Péèe o dìti ale není jedinou oblastí, kde se zákony v této skandinávské zemi vymkly z kloubù zdravého rozumu. Nyní Norové zaèali potírat i ebráky.
+Norsko, stejnì jako øadu dalích zemí západní Evropy, trápí nájezdy gangù organizovaných ebrákù, pøedevím Romù z Rumunska a Bulharska. V devadesátých letech byly tyto gangy k vidìní u i nás - takové ty enièky v átku a s podvyiveným oblièejem, které mìly v uzlu na hrudi spící (ve skuteènosti èasto nadopované práky) nemluvnì.
+Norský systém mylení, zøejmì zformovaný nedostatkem sluneèního svitu této severské zemì, vyprodukoval skuteènì revoluèní zpùsob boje proti ebrákùm. Místo toho, aby policie tyto gangy øeila jako skupiny organizovaného zloèinu, bude trestat lidi, kteøí nìco dají ebrákovi.
+Chudí lidé úplnì na okraji jsou v kadé spoleènosti. V Norsku, pøes vyspìlý sociální systém, ebrají nejen gangy z Rumunska, ale také nortí ztroskotanci, alkoholici, asociální schizofrenici a hranièáøi. Je celkem jedno, jestli jde o etnické Nory, nebo dlouhodobì usazené pøistìhovalce.
+Pøesto Norsko zavádí zákonnou úpravu, kdy bude nejen zakázáno ebrat. Obdarovat ebráka - a u penìzi nebo jídlem, a u rumunského organizovaného, nebo norského neorganizovaného - bude také trestným èinem, Dostat mùete pokutu i pùlroèní vìzení. Podrobnì to u nás napøíklad popsal web iDnes.cz
+Pomáhat lidem v nouzi se tak stane zloèinem, charitativní jednání bude trestné, je proto otázkou, jak v kontextu toho zákona budou smìt pùsobit norské humanitární organizace zamìøené na bezdomovce. Chudoba se stane zloèinem, a jetì vìtím zloèinem bude pomoc chudým. Suma sumárum: Norsko chce trestnì postihnout projevy soucitu, køesanského milosrdenství a lidské vzájemnosti. Tedy projevy principù, na kterých stojí nae civilizace a jejich dodrování dostalo euroatlantický civilizaèní okruh k takové míøe humanity, svobody a blahobytu, jaká není nikde jinde na planetì. V Norsku z toho teï udìlají trestný èin.
+Myslím si, e je to - podobnì jako politika norské sociálky - cesta k urychlenému konci naí civilizace.
\ No newline at end of file

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1250-czech/_ude_2.txt Added

@@ -0,0 +1,23 @@
+Tìch policisté nakonec zajistili asi 30 kusù, øekl iDNES.cz policejní mluvèí Tomá Hulan. Dodal, e v drtivé vìtinì lo o padìlané karty amerických spoleèností, evropská mìla být jen jedna.
+
+Tito mladíci nakupovali v centru Prahy v tìch nejluxusnìjích obchodech. A to pøedevím v Paøíské ulici a blízkém okolí.
+
+Do Prahy pøitom pøiletìli minulou nedìli a prakticky ihned se vydali na nákupy, popisoval mluvèí.
+
+Policisté se o jejich podezøelé poèínání dozvìdìli hned druhý den. Upozornila je na to firma, její znaèku karet skupina padìlala.
+
+Spoleènosti se toti nezdálo velké mnoství zamítnutých transakcí, které ale zaznamenala nejen v Praze, ale u i pøed tím v Dubaji èi Øecku - tedy místech, kde skupinka na cestì do Èeska pøestupovala.
+
+Tuto kodu ve stádiu pokusu kriminalisté následnì pøedbìnì vyèíslili na 5 a 7 milionù.
+Byli jsme jen bílí konì, hájili se Asiaté
+
+Kriminalisté skupinku proto monitorovali, finální zátah pak pøiel o ètyøi dny pozdìji.
+
+Dva z muù pomohli zadret pracovníci bezpeènostní agentury v jednom z obchodù v Paøíské ulici, tøetího pak v hotelu, vysvìtlil mluvèí.
+
+Kromì padìlaných karet kriminalisté u muù zajistili hodinky za milion a dalí zboí za statisíce vèetnì mobilních telefonù a jejich pøísluenství.
+
+Mladíci se k trestné èinnosti pøiznali. Uvedli, e k takovému jednání je vedla jejich tíivá ivotní situace. Pøistoupili na to, e budou takzvaní bílí konì, falenými platebními kartami je vybavil nìkdo jiný, dodal Hulan.
+
+Zadrení jsou nyní v policejní cele. Za trestné èiny neoprávnìné opatøení, padìlání a pozmìnìní platebního prostøedku a podvod jim hrozí a osmileté vìzení.
+

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1250-hungarian/_ude_1.txt Added

@@ -0,0 +1,6 @@
+Tájékoztatják a tankerületeket az iskolákba beszivárgó szekta tevékenységérõl, amiben tudtukon kívül vesznek részt ezrek.
+A Klebelsberg Intézményfenntartó Központ naprakész információkkal látja el a tankerületeket a szcientológiai egyház és társszervezeteinek mûködésérõl és módszereirõl  közölte a Klik, miután hétfõn a Társaság a Szabadságjogokért arról írt, hogy a szcientológusok drogterápiának álcázva folytatják tevékenységüket iskolások körében.
+A TASZ jelentése szerint a Szcientológia Egyház tagjai számos olyan szervezetet hoztak létre, melyek áltudományos módszerekkel drogterápiának vagy drogprevenciónak álcázott marketingtevékenységet folytatnak az egyház befolyásának növeléséért.
+A Klik az MTI kérdésére kifejtette: a tankerületek és az intézmények elõzetesen szûrik az iskolákban megszervezendõ prevenciós eseményeket és az elõadásokat tartó szervezeteket.
+A szcientológia alapítója L. Ron Hubbard sci-fi író. Kritikusai üzleti szervezetnek tartják, mely a hívõk kizsákmányolásából tartja fenn magát. A Magyarországi Szcientológia Egyházat 1993-ban a parlament destruktív szektának minõsítette, a Nemzetbiztonsági Hivatal társadalomra veszélyes mozgalomként tesz róla említést. A TASZ jelentése szerint célja a minél nagyobb gazdasági és politikai befolyásszerzés. A mentális zavarokról, szenvedélybetegségekrõl áltudományos elméleteket állított fel, drogterápiáinak számos halálos áldozata van.
+A drogmentes maratont szervezõ Együtt egy Drogmentes Magyarországért Alapítvány is a szcientológusok érdekkörébe tartozik, mégis több tízezer iskolást, szülõt, pedagógust, sportolót és köztisztviselõt mozgatnak meg, akik gyakran nem sejtik, hogy miben vesznek részt.

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1250-hungarian/_ude_2.txt Added

@@ -0,0 +1,14 @@
+
+
+A Századvég kutatási igazgatója a Financial Times-on magyarázta a magyar külpolitikát. Mint Orbán Balázs kifejtette, Magyarország jól vizsgázott lojalitásból, amikor ellenérzései dacára megszavazta az Oroszország elleni szankciókat. A szerzõ mindenkit megnyugtatott: Putyin látogatása teljes egészében Magyarország gázbiztonságáról szól, az idén lejáró, Magyarország számára elõnytelen szerzõdést kell újratárgyalni az orosz elnökkel.
+
+Orbán Balázs, a Századvég kutatási igazgatója a Financial Times-on közzétett vendégposztban avatta be hétfõn az olvasókat az Orbán-kormány külpolitikai megfontolásaiba. A Magyarország Oroszország és az EU között próbál egyensúlyozni címû írásban kifejtette, hogy Angela Merkel német kancellár két héttel ezelõtti és Vlagyimir Putyin orosz elnök holnapi látogatása, valamint Orbán Viktor vizitje Petro Porosenko ukrán elnöknél egyaránt fontos diplomáciai események az Ukrajnával szomszédos országnak.
+
+Orbánnak nincs jó sajtója a nyugati országokban. A Századvég szerzõje szerint a magyarok acélos miniszterelnöke a 2010-es választási gyõzelme óta határozottan képviseli nemzete érdekeit, és elszántsága súlyos érdekeket sért az EU-ban és az Egyesült Államokban is.
+
+Orbán Balázs szerint a külföldrõl érkezõ hajthatatlan kritika lehetetlenné teszi, hogy reálisan értékeljék a magyar külpolitika erõfeszítéseit.
+Angela Merkel látogatása ugyanakkor megmutatta, hogy Németország osztja Magyarország geopolitikai nézeteit. Ahogy Orbán Viktor többször is megfogalmazta, Magyarország katonai, biztonsági megfontolásait a NATO-tagsága határozza meg, miközben abban is nagyon érdekeltek vagyunk, hogy Ukrajnában hallgassanak el a fegyverek.
+
+Magyarország meggyõzõdése, hogy Oroszország fegyveres beavatkozását meg kell akadályozni, de Moszkvát nem lehet erre sem fegyverrel, sem nemzetközi elszigeteléssel kényszeríteni. Az se véletlen, hogy Budapesten Orbánnal együtt Merkel is kijelentette, hogy nem szállítanak fegyvert Ukrajnába - írta a Századvég szakértõje.
+
+Bár több EU-tagállam geopolitikai helyzete merõben más, mint hazánké, Orbán Balázs szerint Szlovákia és Finnország is szoros gazdasági kötelékben van Oroszországgal. A vendégposzt magyar szerzõje ezért arra hívta fel az EU és az Egyesült Államok figyelmét, hogy Oroszországot illetõen nem Magyarország az egyetlen pragmatikus álláspontot képviselõ

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1250-hungarian/_ude_3.txt Added

@@ -0,0 +1,11 @@
+Az eurózóna pénzügyminiszterei közül többen is utaltak arra, hogy hétfõ délutáni tanácskozásukon valószínûleg nem születik megállapodás a görög államadósság kérdésének rendezésérõl.
+
+Wolfgang Schäuble német pénzügyminiszter a tanácskozást megelõzõen Brüsszelben , szkeptikusan beszélt arról, hogy születhet-e hétfõn egyezség. A nap folyamán korábban Schäuble arról beszélt, hogy sajnálja a görögöket, mert egy felelõtlenül viselkedõ kormányt választottak maguknak.
+
+Az ír Michael Noonan és Peter Kazimir szlovák pénzügyminiszter is arról beszélt, hogy nem világosak számukra a részletek, és azt várják Janisz Varufakisz görög pénzügyminisztertõl, hogy tegye teljesen világossá, mit is akar Athén nemzetközi partnereitõl. Noonan azt is elmondta, hogy bármi legyen is az alku, akár a jelenlegi mentõprogramot hosszabbítják meg, akár új programot dolgoznak ki, arról végsõ soron az ír parlamentnek is szavaznia kell
+
+Pierre Moscovici, az Európai Bizottság gazdasági és monetáris ügyekért felelõs francia tagja akárcsak a múlt héten, ismételten arról beszélt, hogy figyelembe kell venni, hogy a görög választók milyen politikai program mellett tették le voksukat, ugyanakkor Görögországnak is tiszteletben kell tartania korábbi kötelezettségvállalásait.
+
+A francia politikus szerint a felelõsség és a szolidaritás elvét kell szem elõtt tartani. Kitért arra is, hogy Athén partnerei már eddig is jelentõs szolidaritásról tettek tanúbizonyságot. Arra az újságírói kérdésre, hogy mi lesz, ha ma nem sikerül megállapodni, Moscovici azt mondta: csak egyetlen forgatókönyv létezik, a megbeszélések sikeres lezárása Görögországgal, minden más puszta spekuláció.
+
+Valdis Dombrovskis, a brüsszeli tesület euróért felelõs alelnöke rámutatott: Brüsszel konstruktívan áll a megbeszélésekhez. Úgy vélte ugyanakkor, hogy a legkézenfekvõbb a jelenlegi reformprogram meghosszabbítása lenne, legfeljebb féléves idõtávra. Az eurózóna pénzügyminisztereinek délután kezdõdõ tanácskozása a várakozások szerint éjszakába nyúlik majd.

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1250-polish Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1250-polish/_ude_1.txt Added

@@ -0,0 +1,14 @@
+Naukowcy zauwa¿yli, ¿e zmiana rytmu snu w dni wolne od pracy prowadzi do rozregulowania naszego wewnêtrznego zegara biologicznego. Zjawisko to nazwano "social jet lag", czyli spo³eczny jet lag.
+Badanie z udzia³em ponad 800 ochotników wykaza³o, ¿e osoby, które w dni wolne od pracy sypiaj¹ w innych godzinach ni¿ w dni robocze, s¹ bardziej nara¿one na oty³oci i wynikaj¹ce z niej choroby, ni¿ ci, którzy przez ca³y tydzieñ k³ad¹ siê spaæ i wstaj¹ mniej wiêcej w podobnych godzinach.
+W przeciwieñstwie do jet lagu, czyli zespo³u nag³ej zmiany strefy czasowej, polegaj¹cego na desynchronizacji organizmu, która objawia siê m.in. chwilowymi zaburzeniami apetytu i problemami ¿o³¹dkowo-jelitowymi, spo³eczny jet lag mo¿e przybraæ postaæ przewlek³¹ i towarzyszyæ nam przez ca³e ¿ycie zawodowe, dlatego jego wp³yw na metabolizm mo¿e byæ znacznie powa¿niejszy.
+- Zjawisko spo³ecznego jet lagu nie zosta³o jeszcze dok³adnie przebadane. Uwa¿amy jednak, ¿e jako czynnik wp³ywaj¹cy na rozregulowanie zegara biologicznego, ma wp³yw na nasze zdrowie  mówi autor badania, dr Michael Parsons, z wydzia³u genetyki ssaków przy Radzie ds. Badañ Medycznych, agencji rz¹dowej odpowiedzialnej za koordynacjê i finansowanie badañ medycznych w Wielkiej Brytanii.
+Czas na zdrowie: Jak walczyæ z oty³oci¹ - Czy kilka zbêdnych kilogramów to ju¿ oty³oæ? Kiedy rzeczywicie powinnimy zacz¹æ siê martwiæ nasz¹ wag¹? A je¿eli ju¿ mamy z ni¹ problem, jak skutecznie zrzuciæ kilogramy? - Onet.
+- Nasz eksperyment potwierdzi³ wnioski z wczeniejszego badania, które dowiod³o, ¿e istnieje zwi¹zek miêdzy ostr¹ form¹ spo³ecznego jet lagu, a wzrostem BMI. Nasze badanie po raz pierwszy wskaza³o, ¿e zmiany rytmu snu zwiêkszaj¹ zagro¿enie chorobami zwi¹zanymi z oty³oci¹.
+Naukowcy zmierzyli wzrost, wagê i obwód w pasie uczestników badania, przeanalizowali te¿ biomarkery stanów zapalnych i cukrzycy w krwi. Wyniki porównano z danymi z kwestionariusza na temat d³ugoci snu i chronotypu, czyli typu aktywnoci oko³odobowej.
+Badacze stwierdzili, ¿e ju¿ ró¿nica dwóch godzin w rytmie snu w dni wolne od pracy mo¿e podwy¿szaæ zagro¿enie wzrostem BMI i zwiêkszaæ obecnoæ biomarkerów stanów zapalnych i cukrzycy.
+Od nadwagi siê zaczyna, a gdy nadwaga przekszta³ca siê w oty³oæ, nie tylko jest ci ciê¿ej ¿yæ, dopada ciê tak¿e wiele powa¿nych chorób i dolegliwoci, w tym. m.in. problemy z uk³adem kra¿enia, cukrzyca, zwyrodnienie stawów. Do jakich jeszcze chorób prowadzi oty³oæ?
+Przyczyny tego wzrostu na razie s¹ nieznane, ale wiele wskazuje, ¿e spo³eczny jet lag zaburza zdrowe nawyki, takie jak dieta i gimnastyka, na tyle, ¿e odbija siê to na zdrowiu.
+- Badanie pomo¿e nam zrozumieæ fizjologiê spo³ecznego jet lagu i zwi¹zek tego zjawiska z oty³oci¹ i chorobami wynikaj¹cymi z oty³oci  t³umaczy dr Terrie Moffitt, wspó³autorka badania. - Dalsze badania mog¹ wp³yn¹æ na politykê zapobiegania oty³oci, poprzez zmiany praktyk prowadz¹cych do spo³ecznego jet lagu, takich jak harmonogram pracy czy zmiana czasu na letni.
+Profesor David Lomas z Rady ds. Badañ Naukowych zauwa¿a: - To badanie przynosi kolejne dowody na to, ¿e funkcjonowanie niezgodne z zegarem biologicznym, nawet na bardzo ma³¹ skalê, mo¿e po czêci przyczyniaæ siê do wzrostu oty³oci i zwi¹zanych z ni¹ chorób.
+- Nale¿a³oby przyjrzeæ siê w jaki sposób zmiany w rozk³adzie dnia i diecie mog¹ zmniejszaæ ten szkodliwy trend i wp³ywaæ na poprawê zdrowia publicznego - dodaje.
+

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1250-romanian Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1250-romanian/_ude_1.txt Added

@@ -0,0 +1,27 @@
+Preºedintele României, Klaus Iohannis, a acordat, în premierã de la preluarea mandatului, un interviu unei televiziuni strãine. ªeful statului a vorbit la TV5 MONDE, în cadrul principalei emisiuni de politicã externã a postului francez de televiziune. În timpul interviului, preºedintele a declarat cã eradicarea corupþiei a fost tema principalã a campaniei sale electorale ºi cã România trebuie sã respecte trei puncte esenþiale în politica externã: NATO, UE si americanii.
+Cele mai importante declaraþii ale preºedintelui României, Klaus Iohannis:
+TV5: Nimeni nu se astepta la victoria dv. Cum analizati victoria dv?
+Klaus Iohannis: Toatã lumea credea cã o sã pierd, dar s-au înselat. Dupã primul tur pãrerile alegãtorilor s-au schimbat si românii s-au schimbat. Mai multi oameni au venit la vot si foarte multi din diasporã. Tot complexul a dus la un rezultat care m-a fãcut presedinte. Un val care continuã si astãzi. Un val de încredere si simpatie. Asta îmi dã multã energie.
+TV5: Ati promis sã eradicati coruptie. Aveti mijloace?
+K.I.: A fost tema principalã a campaniei mele. Coruptia este prezenta în sferele înalre ale politice si avem rezulatte notabile. Este un fapt, lupta existã si este important sã arãtãm asta poporului. Sunt hotãrât sã sprijin institutiile care se ocupã de asta. Eu am transmis semnale foarte clare si lupta va continua.
+Citeste si:
++Declaraþia ªOCANTÃ a lui Iohannis despre Ceauºescu
++ Declaratia DURÃ a lui Klaus IOHANNIS care îl va ÎNFURIA la culme pe Ion ILIESCU
++REZULTATE ALEGERI PREZIDENÞIALE 2014. BEC: Iohannis - 54,5, Ponta - 45,49. Ce PRIORITÃÞI are presedintele ales
+TV5: Alianta cu americanii va continua?
+K.I.: Cu siguranta nu vor fi schimbari esentiale in abordare. Ei sunt partenerii nostri strategici. Avem trei puncte ale politicii externe, NATO, UE si americanii. Este si loc de relatii bilaterale, cum ar fi relatia bilaterala cu Franta. Romania trebuie sa fie un partener serios si de cursã lungã.
+TV5: Rusia?
+K.I.: Din pãcate existã Ucraina si am fi preferat ca asta sã nu se întâmple. Pentru noi europenii acestea sunt niste abuzuri. Rusia a cãlcat linia rosie. România considerã cã nu sunt alte solutii.
+TV5: Cum vedeti conflictul, va ajunge in România?
+K.I.: Nu este un scenariu posibil dar vom fi foarte prudenti.
+TV5: Veti veni la Paris, ce relatii veti avea cu Franta?
+K.I.: Avem un parteneriat traditional care este secular. Multi investitori francezi vor veni în România. Limba francezã este foarte rãspânditã la noi, si cred cã vor creste optiunea pentru francezã ca limbã strãinã. Si eu imi voi imbunatati cunostintele de francezã.
+Cele mai comentate:
++Dezvãluiri. Cum s-a îmbogãtit Iohannis folosindu-se de falsi mostenitori si de acte fabricate
++Marea gresealã a lui Traian Bãsescu. 10 ani de mandat sau cum se ajunge de la Presedinte-jucãtor la Presedinte-vinovat. Editorial de Dan Andronic
++PRIMA dezbatere FATÃ ÎN FATÃ. Ponta: Nu suntem slugile dumneavoastrã. Când aþi fost la mine în birou eraþi foarte umil. Iohannis: Unde vã credeþi, la biroul dumneavoastrã? Biroul este þara!
+TV5: Se reproseazã probleme de rasism. Este o prioritate?
+K.I.: Da, daca nu reglam problema se va agrava. Suntem foarte preocupati de romi. Trebuie sã vedem unde s-au fãcut erori la noi si in UE. Dar asta nu este doar o problema a Romaniei. Cunosc sensibilitatile in Franta si peste tot despre romi. Toti ne dorim sa rezolvam
+"64' le monde en français", emisiunea în care a fost prezentat interviul cu preºedintele României, Klaus Iohannis, este principala emisiune de analizã a politicii externe a postului TV5MONDE, emisiune care prezintã afacerile internaþionale din perspectiva þãrilor francofone.
+Rãmâneþi pe EVZ.ro pentru a afla, în timp real, cele mai importante declaraþii ale preºedintelui României, Klaus Iohannis.
+Tag-uri: klaus iohannis, iohannis interviu, interviu iohannis, declaratii iohannis, klaus iohannis declaratii, interviu klaus iohannis, tv5, interviu iohannis tv5, iohannis interviu tv5

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1250-slovak Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1250-slovak/_ude_1.txt Added

@@ -0,0 +1,14 @@
+Zaèo som bol rodièom naozaj vïaèný, bolo e naetrili dos peòazí, aby som mohol ís tudova.
+Otec ma neposlal na tátnu, ale dokonca na jednu z najprestínejích súkromných univerzít v
+Kanade. Pä rokov som tudoval Ekonomickú fakultu v Toronte a konèil som s vyznamenaním a
+èerveným diplomom. U v poslednom roèníku som nemal o prácu núdzu a mnohé firmy sa
+predchádzali, aby ma zlanárili pracova u nich. Ja som sa vak nikdy nenechal upísa natrvalo a
+roèný kontrakt som vdy predloval, len o ïalí rok. Moje ciele boli vyie, ako iba zarába na
+druhých. Túil som pracova sám pre seba a tak sa mi o tri roky neskôr podarilo zaregistrova
+spoloènos s názvom SVET. Èinnosou podnikania bol obchod s energiami. Zjednoduene
+povedané, odpredaj zásob zemného plynu od spoloènosti, kde ho majú nadbytok a po èase, alebo
+pod¾a potreby, predaj spoloènosti, alebo krajine, kde je ho naopak nedostatok. Keïe ceny energii sú
+ve¾mi kolísavé, so základným kapitálom sa dá na týchto pohyboch slune zarobi. Poèiatoèný kapitál
+som získal od Torontskej tátnej banky, keïe som bol najlepím absolventom ekonómie - nemali
+problém investova do mòa. U po prvom roku som mal prenajaté celé dvadsiate tretie poschodie
+Lemieuxovho mrakodrapu vo východnom Yorku a o desa rokov neskôr, patril celý mne.
\ No newline at end of file

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1250-slovak/_ude_2.txt Added

@@ -0,0 +1,7 @@
+Ako informuje talianska televízna stanica TGcom24 na svojej internetovej stránke, do pondelka sa podarilo zachráni 2 164 imigrantov na dvanástich plavidlách. Niektorých zachránila talianska vojenská loï, ïalích lode pobrenej hliadky a talianskej colnej polície ako aj iné komerèné lode v oblasti.
+
+Talianske ministerstvo dopravy informovalo, e pobrenú strá ohrozovali zbraòami tyria ozbrojení mui, ktorí prili na motorovom èlne od líbyjského pobreia. Prinútili ich vráti im loï, z ktorej vystúpili uteèenci.
+
+Minulý rok si nebezpeèná plavba z afrického pobreia do Európy vyiadala najmenej 3200 ivotov. Len minulý týdeò zahynulo v Stredozemnom mori viac ne 300 imigrantov, ktorí sa v rozbúrenom mori plavili v nafukovacích èlnoch. Talianska pobrená strá minulý týdeò zachránila z mora asi 600 uteèencov.
+
+Tieto poèty ukazujú na nutnos posilni európsku misiu Triton, ktorá v novembri nahradila taliansku námornú operáciu Mare Nostrum zameranú na záchranu uteèencov. Talianska ministerka obrany Roberta Pinottiová nevylúèila, e s imigrantmi sa na talianske pobreie, a tým aj do Európy, môu dosta islamskí radikáli.
\ No newline at end of file

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1250-slovak/_ude_3.txt Added

@@ -0,0 +1,9 @@
+Tisíce firiem na Slovensku neplatia iadnu daò z príjmu. Vyplýva to zo zoznamu daòových subjektov s výkou ich splatnej dane za rok 2013, ktorý na základe údajov z registra úètových závierok prvýkrát zverejnil rezort financií.
+Zoznam má a 1 408 strán. Kým viac ako 10 mil. eur na dani z príjmov pritom do rozpoètu za rok 2013 odviedlo pod¾a zoznamu 16 spoloèností, tisíce firiem neodviedli niè alebo len symbolických pár eur.
+Zmeni by sa to malo a za rok 2014. V òom toti u platili daòové licencie, ktoré sa vlani zaviedli. Vláda stanovila, e firmy musia plati za licenciu 480 a 2 880 eur aj v prípade, ak nevykáu zisk. V prípade kladnej daòovej povinnosti si licenciu z nej môu odpoèíta. Zavedenie licencií vláda vysvet¾ovala tým, e podnikatelia si pri kreatívnom úètovníctve vedia zniova základ dane tak, aby daò neplatili. Daòová kontrola na to prís môe, ale pri nieko¾kých státisícoch firiem kontrolóri nestíhajú preveri vetky.
+Zoznam firiem a odvedených daní.
+Najvyí objem dane z príjmov za rok 2013 do rozpoètu odviedli eustream, Kia a telekomunikaèné spoloènosti Slovak Telekom a Orange Slovensko. Eustream odviedol 97,6 milióna eur a Kia Motors Slovakia 83,4 milióna eur. V prípade Slovak Telekomu to bolo takmer 46 mil. eur a pri Orangeu Slovensko 42,4 mil. eur. V zozname chýbajú niektoré ve¾ké firmy, ktoré by mali patri medzi najväèích platcov dane. Ministerstvo u raz svoj zoznam opravilo, v prvej verzii v òom chýbali eustram aj Kia.
+Medzi firmami, ktoré odviedli viac ako 10 mil. eur na dani z príjmov, nechýbali napríklad Slovenská sporite¾òa, Tatra banka, Nafta, Continental Matador, Allianz  Slovenská poisovòa, ale ani energetické distribuèné spoloènosti èi firma Eset, ktorá odviedla na dani z príjmov takmer 17,7 mil. eur. Len týchto prvých 16 firiem zo zoznamu platite¾ov dane z príjmov pritom do rozpoètu prispelo celkovou sumou 417,5 mil. eur. Na druhej strane rebríèka, ktorý obsahuje vye 73-tisíc záznamov, sa vak nachádza takmer 32-tisíc spoloèností, ktorých daòová povinnos na dani z príjmov za rok 2013 nedosiahla ani 500 eur.
+"Naím cie¾om je takýmto zoznamom zvýi informovanos verejnosti o tom, ako firmy na Slovensku platia dane, a prispie k väèej daòovej transparentnosti podnikate¾ského prostredia, povedal na margo zverejnených údajov minister financií Peter Kaimír. Tento zoznam ministerstvo zverejòuje na základe novely zákona o správe daní platnej od 1. januára 2015. Zoznam pozostáva z verejne dostupných údajov registra úètovných závierok za rok 2013 a uvádza sa v òom obchodné meno alebo názov právnickej osoby, jej sídlo, identifikaèné èíslo a výka splatnej dane. Firmy sú zoradené pod¾a výky splatnej dane od najvyej po najniiu.
+Ministerstvo financií pritom tvrdí, e zoznamy o tom, ako firmy platia dane, bude aktualizova z dostupných údajov na tvrroènej báze. Údaje budú zoh¾adòova posledné dostupné úètovné obdobie.
+Daò z príjmov je podstatnou daòou na Slovensku. Z celkových vlaòajích príjmov tátneho rozpoètu vo výke 12,5 mld. eur tvorili dane z príjmov fyzických a právnických osôb sumu 2,1 mld. eur. Pritom z dane z pridanej hodnoty tát zinkasoval 4,9 miliardy eur a zo spotrebných daní 2 miliardy eur.
\ No newline at end of file

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1250-slovene Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1250-slovene/_ude_1.txt Added

@@ -0,0 +1,18 @@
+Kako tudi brez poviice do zadovoljstva na delovnem mestu
+Avtor: Ana Rupar
+e res, da veliko ljudi pravi, da jih na delu najbolj motivira dobra plaèa in poten odnos, vendar, ali jih to tudi osreèuje?
+Monosti, da dve osebi natejeta pet istih motivacijskih dejavnikov in to v istem vrstnem redu, je komaj ena proti sto tisoè. Monosti, da dve osebi natejeta pet istih motivacijskih dejavnikov in to v istem vrstnem redu, je komaj ena proti sto tisoè.
+O delovni motivaciji je bilo e veliko napisano, opravljene so bile e tevilne raziskave, katerih skupni imenovalec je, da ljudi na delovnem mestu najbolj motivirajo: odprta komunikacija, zaupanje, moè, priznanje, lastnitvo, zanimivo delo, cilji, delovni èas in seveda plaèilo.
+Bogastvo in fleksibilen delovnik vendarle ne osreèujeta vseh
+Najnoveja raziskava kulture dela, ki je zajemala 850 tisoè ljudi, to postavlja na glavo. Njen avtor Chester Elton rezultatov prejnjih raziskav sicer ne zanika, trdi pa, da ti dejavniki motivirajo le nekatere ljudi, medtem ko drugih ne oziroma jih lahko celo demotivirajo.
+Njihova raziskava je izhajala iz obsenih raziskav amerikega psihologa Stevena Reissa, ki je ugotovil, da smo si ljudje veliko bolj razlièni, kot mnogi mislijo  medtem ko nekatere spodbujajo tekmovanje, zmage in bogastvo, druge zadovoljujeta obèutek samostojnosti in socializacija.
+Izhajajoè iz tega so doloèili 23 razliènih motivacijskih dejavnikov  od izzivov, odlikovanj, timskega dela, pomoèi, soèutja, zabave, kreativnosti, uèenja do denarja in prestia. Ko so ljudi spraevali, kaj od natetega jih najbolj spodbuja, so ugotovili, da je monosti, da dve osebi natejeta pet istih dejavnikov v istem vrstnem redu, komaj ena proti sto tisoè.
+Motivacija ni enoznaèna, ima globlji pomen
+Dejstvo je namreè, da motivacijskih dejavnikov ni mogoèe kar tako preprosto posploiti na vse. Zato je delovno motivacijo treba iskati prav pri vsakem posamezniku posebej.
+Do odgovora lahko pridemo s tremi preprostimi vpraanji:
+1. Se dela veseli?
+2. Te delo napolnjuje z energijo?
+3. Ali o delu pripoveduje z arom, navduenjem?
+Ljudje, ki lahko na vsa tri vpraanja odgovorijo pritrdilno, so v svojem delu zagotovo nali globlji pomen, poèutijo se bolj polni energije in zadovoljni, svoje delo pa razumejo kot enega najveèjih virov zadovoljstva.
+Vsako delovno mesto neprijetne zadolitve in èe vidimo le te, bomo povsod nezadovoljni. Vsako delovno mesto neprijetne zadolitve in èe vidimo le te, bomo povsod nezadovoljni.
+Zadovoljstva ni treba ravno iskati v novi slubi

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian/_chromium_windows-1251_with_no_encoding_specified.html Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian/_ude_1.txt Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian/aif.ru.health.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian/anthropology.ru.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian/aug32.hole.ru.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian/aviaport.ru.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian/blog.mlmaster.com.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian/forum.template-toolkit.ru.1.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian/forum.template-toolkit.ru.4.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian/forum.template-toolkit.ru.6.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian/forum.template-toolkit.ru.8.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian/forum.template-toolkit.ru.9.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian/greek.ru.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian/intertat.ru.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian/janulalife.blogspot.com.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian/kapranoff.ru.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian/money.rin.ru.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian/music.peeps.ru.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1251-russian/newsru.com.xml Changed

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1252/_ude_1.txt Added

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1252/_ude_2.txt Added

@@ -0,0 +1,9 @@
+Stipan Arkadiewitsch was tegenover zich zelf tamelijk oprecht. Hij kon niet liegen en zich zelf verzekeren, dat zijn handelwijze hem berouwde. Hij kon er geen berouw over gevoelen, dat hij, een vijf-en-dertigjarig man met licht ontvlambaar temperament, niet op zijn eigen vrouw, moeder van vijf levende en twee gestorven kinderen, verliefd was, te meer daar zij maar een jaar jonger was dan hij. Hij betreurde het slechts, dat hij het niet beter geheim had kunnen houden. "Als men niet gepakt wordt, is men geen dief," dacht hij. Maar wel gevoelde hij al het netelige van zijn toestand en medelijden met zijn vrouw en kinderen.
+Misschien zou hij zorgvuldiger zijn zonden voor zijn vrouw hebben weten te verbergen, indien hij had kunnen vermoeden, dat de kennis daarvan zoo op haar zou werken.
+Nooit nog had hij helder over de geheele zaak doorgedacht, slechts had hij een nevelachtige voorstelling, alsof zijn vrouw er iets van giste en ze door de vingers zag.
+Het kwam hem zelfs voor, dat zij, een zwakke, vroeg verouderde en niet eens meer knappe vrouw, een door niets uitmuntende, eenvoudige en slechts goede huismoeder, volgens recht toegevend zijn moest. En nu was hem het tegenovergestelde gebleken!
+"O vreeseljjk, vreeselijk!" klaagde Stipan Arkadiewitsch weer en vond geen uitweg.
+"En hoe goed hadden wij het tot hiertoe! Hoe goed hebben wij met elkander geleefd! Zij was tevreden en gelukkig met de kinderen, ik legde haar niets in den weg en liet haar met de kinderen en de huishouding haar eigen gang gaan. Dit is waar: het was niet goed, dat Jeanne bij ons gouvernante was, dat deugde niets! Een minnarij met een gouvernante is iets triviaals! Maar welk een gouvernante was zij ook!"
+Daarbij herinnerde hij zich weer levendig de zwarte, guitachtige oogen en het betooverend glimlachje van mademoiselle Roland.
+"En zoo lang ze bij ons in huis was, heb ik mij toch ook niets veroorloofd! Het ergste is echter, dat zij al. Ach, het moest alles zoo komen! Ach, ach, wat nu gedaan, wat nu toch gedaan?"
+Op die vraag vond hij slechts het eene, algemeene antwoord, dat het leven op de ingewikkeldste en onoplosbaarste vraagstukken geeft: Leef zoo, dat gij elken dag zijn recht geeft, tracht u zelf te vergeten en te bedwelmen in den droom des levens.

_service:tar_scm:chardet-5.1.0.tar.gz/tests/windows-1254-turkish/_ude_1.txt Added

@@ -0,0 +1,8 @@
+Terör örgütünün bankasý olur mu?
+Bank Asyada imtiyazlý ortaklarýn yönetim hakkýný TMSF devraldý. Gerekçe, 185 ortaðýn bazýlarýnýn banka sahibi olma vasfýný kaybetme ihtimali. Bu yüzden, her birinden teferruatlý bilgiler talep ediliyor. Devir iþlemi, belgelerin, BDDKya ulaþmasý geciktiði için gerçekleþti.
+Ýyi niyetli davranýlsa, 185 ortaktan belge talep etmek yerine, banka sahibi olma özelliðini kaybettiði düþünülen kiþilerden bilgiler istenirdi. Oysa BDDK bu isimleri vermiyor. Belli ki asýl mesele baþka. Bunu da Nagehan Alçýdan duyduk. AHaberdeki programda PKKnýn bir bankasý olamayacaðý gibi, Cemaatin de bankasý olamaz demiþ. Biri tescilli terör örgütü diðeri sadece Tayyip Erdoðan ve yandaþlarýnýn kafasýnda terör örgütü.
+Bu zihniyetle hareket edilirse, hiçbir sermaye grubunun ya da hiçbir þahsýn güvencesi kalmaz. Henüz iddianame bile yazýlmadan terör örgütü diye damgalýyorsunuz. Ondan sonra da bunun yanlýþ olduðunu söyleyenleri ya da o gruba yakýn duranlarý terör örgütü liderliði veya üyeliðiyle suçluyorsunuz. Cadý avý yapýlan bir dönemde gazeteciye ateþe odun taþýmak yakýþmaz.
+Durum, bankalara el konulan 2001 yýlýndan farklý. O zaman, müesseselerin mali yapýlarý bozulmuþtu; iflas halindeydiler. Sermaye yeterlilik oraný çok aþaðýlara inmiþti. Bankalar BDDKya geçti. Bank Asyada ise sadece talep edilen belgeler zamanýnda teslim edilmediðinden yönetim hakkýna el konuldu; bundan yararlanarak, bütün yönetim kurulu üyeleri deðiþtirildi. Tek baþýna bu uygulama dahi, banka yapýsýnýn saðlam olduðunu gösteriyor. Zira Bankacýlýk Kanununun 70 ve 71inci maddesine göre iþlem yapýlmadý; 18inci madde devreye sokuldu. Aslýnda iyi niyetli davranýlsa, ortaklarýn %85i belgeleri teslim ettiði için %51 çoðunluk var diyerek, yönetim hemen onlara devredilirdi. Kaldý ki 18inci madde, bu gecikmeden dolayý sadece para cezasýný öngörüyor.
+Ama dedim ya niyet baþka. PKKnýn bankasý olur muymuþ!!! Cemaat ile PKKnýn ne farký var öyle deðil mi!!!
+Zaten Erdoðan da Kolombiyaya giderken bu konuyu açtý ve Paralel yapý PKKdan beter dedi.
+Ýmam ve cemaat meselesi
\ No newline at end of file