Refactor code structure for improved readability and maintainability
This commit is contained in:
parent
389d72a136
commit
aa4c067ea8
1685 changed files with 393439 additions and 71932 deletions
|
|
@ -0,0 +1 @@
|
|||
pip
|
||||
|
|
@ -0,0 +1,102 @@
|
|||
Metadata-Version: 2.4
|
||||
Name: pytokens
|
||||
Version: 0.4.1
|
||||
Summary: A Fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.
|
||||
Author-email: Tushar Sadhwani <tushar.sadhwani000@gmail.com>
|
||||
License: MIT License
|
||||
|
||||
Copyright (c) 2024 Tushar Sadhwani
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
||||
Project-URL: Homepage, https://github.com/tusharsadhwani/pytokens
|
||||
Classifier: License :: OSI Approved :: MIT License
|
||||
Classifier: Operating System :: OS Independent
|
||||
Classifier: Programming Language :: Python :: 3
|
||||
Classifier: Programming Language :: Python :: 3 :: Only
|
||||
Classifier: Programming Language :: Python :: 3.8
|
||||
Classifier: Programming Language :: Python :: 3.9
|
||||
Classifier: Programming Language :: Python :: 3.10
|
||||
Classifier: Programming Language :: Python :: 3.11
|
||||
Classifier: Programming Language :: Python :: 3.12
|
||||
Classifier: Programming Language :: Python :: 3.13
|
||||
Classifier: Programming Language :: Python :: 3.14
|
||||
Classifier: Programming Language :: Python :: Implementation :: CPython
|
||||
Classifier: Typing :: Typed
|
||||
Requires-Python: >=3.8
|
||||
Description-Content-Type: text/markdown
|
||||
License-File: LICENSE
|
||||
Provides-Extra: dev
|
||||
Requires-Dist: black; extra == "dev"
|
||||
Requires-Dist: build; extra == "dev"
|
||||
Requires-Dist: mypy; extra == "dev"
|
||||
Requires-Dist: pytest; extra == "dev"
|
||||
Requires-Dist: pytest-cov; extra == "dev"
|
||||
Requires-Dist: setuptools; extra == "dev"
|
||||
Requires-Dist: tox; extra == "dev"
|
||||
Requires-Dist: twine; extra == "dev"
|
||||
Requires-Dist: wheel; extra == "dev"
|
||||
Dynamic: license-file
|
||||
|
||||
# pytokens
|
||||
|
||||
A Fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install pytokens
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
python -m pytokens path/to/file.py
|
||||
```
|
||||
|
||||
## Local Development / Testing
|
||||
|
||||
- Create and activate a virtual environment
|
||||
- Run `PYTOKENS_USE_MYPYC=0 pip install -e '.[dev]'` to do an editable install
|
||||
- Run `pytest` to run tests
|
||||
|
||||
## Type Checking
|
||||
|
||||
Run `mypy .`
|
||||
|
||||
## Compilation with mypyc
|
||||
|
||||
By default, we compile with mypyc.
|
||||
Use the `PYTOKENS_USE_MYPYC` environment variable to control this.
|
||||
|
||||
To check if you are using a compiled version, see whether the output of this is `.py` or `.so`:
|
||||
```bash
|
||||
python -c "import pytokens; print(pytokens.__file__)"
|
||||
```
|
||||
|
||||
## Create and upload a package to PyPI
|
||||
|
||||
1. Make sure to bump the version in `pyproject.toml`.
|
||||
2. Push to Github so CI can build the wheels and sdist.
|
||||
3. Download the artifacts from the CI run.
|
||||
- Find the "Build wheels" job, click "Summary", scroll down to the bottom to see "Artifacts"
|
||||
and download the `cibw-wheels` artifact.
|
||||
4. Unzip the artifact.
|
||||
5. Upload the contents of the artifact to PyPI
|
||||
- e.g. via `twine upload cibw-wheels/*`
|
||||
|
|
@ -0,0 +1,18 @@
|
|||
fd7dcdb10166ebd4db98__mypyc.cp310-win_amd64.pyd,sha256=pdQYlfZzl6N-odqNggiUSWE_wPAuM7pdmuvraEL61ZY,197120
|
||||
pytokens-0.4.1.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
|
||||
pytokens-0.4.1.dist-info/METADATA,sha256=j5eZ3EzTcKw0a-1RILf4buzpDzK-vwSVL-6yd2sQTgo,3890
|
||||
pytokens-0.4.1.dist-info/RECORD,,
|
||||
pytokens-0.4.1.dist-info/WHEEL,sha256=lVtJYX4SZwMxwg8oP4kB_UdF4VQRXLlqu7hUy_2nnAE,102
|
||||
pytokens-0.4.1.dist-info/licenses/LICENSE,sha256=sKlbDAGSQGR4vJmsfidK8BxVvg2y5kE3z-lxzXu358M,1093
|
||||
pytokens-0.4.1.dist-info/top_level.txt,sha256=j1LLhvKvGSi8gfQIfeRcUm1zQgR8mgcMjkpPt0BucnI,37
|
||||
pytokens/__init__.cp310-win_amd64.pyd,sha256=535-xWxQ1o5QaClRRWQE9JwThwdhhyafN75ncvrL_xQ,10752
|
||||
pytokens/__init__.py,sha256=-SP6H8-_w1ezuergzsQhWT-zPTAQ2D9ecizJ1obI2tQ,41044
|
||||
pytokens/__main__.py,sha256=Q4vMs5rh9g2a3QSqgSI6H-aDOIbv_gSpqJwoFDcpC1o,191
|
||||
pytokens/__pycache__/__init__.cpython-310.pyc,,
|
||||
pytokens/__pycache__/__main__.cpython-310.pyc,,
|
||||
pytokens/__pycache__/_mypyc_dummy.cpython-310.pyc,,
|
||||
pytokens/__pycache__/cli.cpython-310.pyc,,
|
||||
pytokens/_mypyc_dummy.cp310-win_amd64.pyd,sha256=gI6qx59rdQ_BAaWoybgth--xGPqd_O3Rqs2UOE13jig,10752
|
||||
pytokens/_mypyc_dummy.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
|
||||
pytokens/cli.py,sha256=WOvm-WJ_niq7m6YQcHNzehBNEcH3JTO4lyO8-kZ-1gg,8236
|
||||
pytokens/py.typed,sha256=-iPtxftU8G1qwXw8jhb2RRoV38oTBriahST6CMXv_qY,60
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
Wheel-Version: 1.0
|
||||
Generator: setuptools (80.10.2)
|
||||
Root-Is-Purelib: false
|
||||
Tag: cp310-cp310-win_amd64
|
||||
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
MIT License
|
||||
|
||||
Copyright (c) 2024 Tushar Sadhwani
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
fd7dcdb10166ebd4db98__mypyc
|
||||
pytokens
|
||||
Loading…
Add table
Add a link
Reference in a new issue