102 lines
3.7 KiB
Text
102 lines
3.7 KiB
Text
Metadata-Version: 2.4
|
|
Name: pytokens
|
|
Version: 0.4.1
|
|
Summary: A Fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.
|
|
Author-email: Tushar Sadhwani <tushar.sadhwani000@gmail.com>
|
|
License: MIT License
|
|
|
|
Copyright (c) 2024 Tushar Sadhwani
|
|
|
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
of this software and associated documentation files (the "Software"), to deal
|
|
in the Software without restriction, including without limitation the rights
|
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
copies of the Software, and to permit persons to whom the Software is
|
|
furnished to do so, subject to the following conditions:
|
|
|
|
The above copyright notice and this permission notice shall be included in all
|
|
copies or substantial portions of the Software.
|
|
|
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
SOFTWARE.
|
|
|
|
Project-URL: Homepage, https://github.com/tusharsadhwani/pytokens
|
|
Classifier: License :: OSI Approved :: MIT License
|
|
Classifier: Operating System :: OS Independent
|
|
Classifier: Programming Language :: Python :: 3
|
|
Classifier: Programming Language :: Python :: 3 :: Only
|
|
Classifier: Programming Language :: Python :: 3.8
|
|
Classifier: Programming Language :: Python :: 3.9
|
|
Classifier: Programming Language :: Python :: 3.10
|
|
Classifier: Programming Language :: Python :: 3.11
|
|
Classifier: Programming Language :: Python :: 3.12
|
|
Classifier: Programming Language :: Python :: 3.13
|
|
Classifier: Programming Language :: Python :: 3.14
|
|
Classifier: Programming Language :: Python :: Implementation :: CPython
|
|
Classifier: Typing :: Typed
|
|
Requires-Python: >=3.8
|
|
Description-Content-Type: text/markdown
|
|
License-File: LICENSE
|
|
Provides-Extra: dev
|
|
Requires-Dist: black; extra == "dev"
|
|
Requires-Dist: build; extra == "dev"
|
|
Requires-Dist: mypy; extra == "dev"
|
|
Requires-Dist: pytest; extra == "dev"
|
|
Requires-Dist: pytest-cov; extra == "dev"
|
|
Requires-Dist: setuptools; extra == "dev"
|
|
Requires-Dist: tox; extra == "dev"
|
|
Requires-Dist: twine; extra == "dev"
|
|
Requires-Dist: wheel; extra == "dev"
|
|
Dynamic: license-file
|
|
|
|
# pytokens
|
|
|
|
A Fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
pip install pytokens
|
|
```
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
python -m pytokens path/to/file.py
|
|
```
|
|
|
|
## Local Development / Testing
|
|
|
|
- Create and activate a virtual environment
|
|
- Run `PYTOKENS_USE_MYPYC=0 pip install -e '.[dev]'` to do an editable install
|
|
- Run `pytest` to run tests
|
|
|
|
## Type Checking
|
|
|
|
Run `mypy .`
|
|
|
|
## Compilation with mypyc
|
|
|
|
By default, we compile with mypyc.
|
|
Use the `PYTOKENS_USE_MYPYC` environment variable to control this.
|
|
|
|
To check if you are using a compiled version, see whether the output of this is `.py` or `.so`:
|
|
```bash
|
|
python -c "import pytokens; print(pytokens.__file__)"
|
|
```
|
|
|
|
## Create and upload a package to PyPI
|
|
|
|
1. Make sure to bump the version in `pyproject.toml`.
|
|
2. Push to Github so CI can build the wheels and sdist.
|
|
3. Download the artifacts from the CI run.
|
|
- Find the "Build wheels" job, click "Summary", scroll down to the bottom to see "Artifacts"
|
|
and download the `cibw-wheels` artifact.
|
|
4. Unzip the artifact.
|
|
5. Upload the contents of the artifact to PyPI
|
|
- e.g. via `twine upload cibw-wheels/*`
|