Package software for any distribution with upt

Transcript

1 Package software for any distribution with upt Cyril Roelandt Package software for any distribution with upt 1/28

2 Introduction Cyril Roelandt OpenStack (Red Hat) GNU Guix (Do not miss Ludovic’s talk at 4:30PM!) Package software for any distribution with upt 2/28

3 Outline 1 Packaging 2 Automation 3 The Universal Packaging Tool 4 fpm 5 Conclusion Package software for any distribution with upt 3/28

4 Outline 1 Packaging 2 Automation 3 The Universal Packaging Tool 4 fpm 5 Conclusion Package software for any distribution with upt 4/28

5 Debugging Send/maintain patches What is packaging? Turn an “upstream” package into a “downstream” package Package software for any distribution with upt 5/28

6 Send/maintain patches What is packaging? Turn an “upstream” package into a “downstream” package Debugging Package software for any distribution with upt 5/28

7 What is packaging? Turn an “upstream” package into a “downstream” package Debugging Send/maintain patches Package software for any distribution with upt 5/28

8 Dependency hell Debugging Package layout Metadata Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated: Some tasks can be automated: Is packaging hard? Many sources (PyPI, RubyGems. . . ) Package software for any distribution with upt 6/28

9 Dependency hell Debugging Package layout Metadata Cannot be fully automated: Some tasks can be automated: Is packaging hard? Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . ) Package software for any distribution with upt 6/28

10 Package layout Metadata Dependency hell Debugging Some tasks can be automated: Is packaging hard? Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated: Package software for any distribution with upt 6/28

11 Package layout Metadata Debugging Some tasks can be automated: Is packaging hard? Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated: Dependency hell Package software for any distribution with upt 6/28

12 Package layout Metadata Some tasks can be automated: Is packaging hard? Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated: Dependency hell Debugging Package software for any distribution with upt 6/28

13 Package layout Metadata Is packaging hard? Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated: Dependency hell Debugging Some tasks can be automated: Package software for any distribution with upt 6/28

14 Metadata Is packaging hard? Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated: Dependency hell Debugging Some tasks can be automated: Package layout Package software for any distribution with upt 6/28

15 Is packaging hard? Many sources (PyPI, RubyGems. . . ) Many targets (Debian, Fedora, Guix. . . ) Cannot be fully automated: Dependency hell Debugging Some tasks can be automated: Package layout Metadata Package software for any distribution with upt 6/28

16 Outline 1 Packaging 2 Automation 3 The Universal Packaging Tool 4 fpm 5 Conclusion Package software for any distribution with upt 7/28

17 Existing tools Fedora Guix FreeBSD OpenBSD Debian CPAN dh-make-perl cpan2rpm guix import ? PortGen NPM npm2deb npm2rpm N/A ? ? PyPI pypi2deb pyp2rpm guix import pytoport PortGen Ruby gem2deb gem2rpm guix import ? PortGen Package software for any distribution with upt 8/28

18 Different behaviours Code duplication (read upstream archive) Code duplication (write downstream package) Issues One CLI per tool Package software for any distribution with upt 9/28

19 Code duplication (read upstream archive) Code duplication (write downstream package) Issues One CLI per tool Different behaviours Package software for any distribution with upt 9/28

20 Code duplication (write downstream package) Issues One CLI per tool Different behaviours Code duplication (read upstream archive) Package software for any distribution with upt 9/28

21 Issues One CLI per tool Different behaviours Code duplication (read upstream archive) Code duplication (write downstream package) Package software for any distribution with upt 9/28

22 License :: OSI Approved :: BSD License BSD2 or BSD3 or . . . ? https://github.com/pypa/warehouse/issues/2996 For values not available in the standard classifiers ‘BSD2’, ‘BSD 2’, ‘BSD-2’, ‘BSD 2 Clause’. . . Classifiers: “license” field (setup.{cfg,py}) Example: licenses (PyPI) SPDX identifiers? Package software for any distribution with upt 10/28

23 For values not available in the standard classifiers ‘BSD2’, ‘BSD 2’, ‘BSD-2’, ‘BSD 2 Clause’. . . License :: OSI Approved :: BSD License BSD2 or BSD3 or . . . ? https://github.com/pypa/warehouse/issues/2996 “license” field (setup.{cfg,py}) Example: licenses (PyPI) SPDX identifiers? Classifiers: Package software for any distribution with upt 10/28

24 For values not available in the standard classifiers ‘BSD2’, ‘BSD 2’, ‘BSD-2’, ‘BSD 2 Clause’. . . BSD2 or BSD3 or . . . ? https://github.com/pypa/warehouse/issues/2996 “license” field (setup.{cfg,py}) Example: licenses (PyPI) SPDX identifiers? Classifiers: License :: OSI Approved :: BSD License Package software for any distribution with upt 10/28

25 For values not available in the standard classifiers ‘BSD2’, ‘BSD 2’, ‘BSD-2’, ‘BSD 2 Clause’. . . https://github.com/pypa/warehouse/issues/2996 “license” field (setup.{cfg,py}) Example: licenses (PyPI) SPDX identifiers? Classifiers: License :: OSI Approved :: BSD License BSD2 or BSD3 or . . . ? Package software for any distribution with upt 10/28

26 For values not available in the standard classifiers ‘BSD2’, ‘BSD 2’, ‘BSD-2’, ‘BSD 2 Clause’. . . “license” field (setup.{cfg,py}) Example: licenses (PyPI) SPDX identifiers? Classifiers: License :: OSI Approved :: BSD License BSD2 or BSD3 or . . . ? https://github.com/pypa/warehouse/issues/2996 Package software for any distribution with upt 10/28

27 For values not available in the standard classifiers ‘BSD2’, ‘BSD 2’, ‘BSD-2’, ‘BSD 2 Clause’. . . Example: licenses (PyPI) SPDX identifiers? Classifiers: License :: OSI Approved :: BSD License BSD2 or BSD3 or . . . ? https://github.com/pypa/warehouse/issues/2996 “license” field (setup.{cfg,py}) Package software for any distribution with upt 10/28

28 ‘BSD2’, ‘BSD 2’, ‘BSD-2’, ‘BSD 2 Clause’. . . Example: licenses (PyPI) SPDX identifiers? Classifiers: License :: OSI Approved :: BSD License BSD2 or BSD3 or . . . ? https://github.com/pypa/warehouse/issues/2996 “license” field (setup.{cfg,py}) For values not available in the standard classifiers Package software for any distribution with upt 10/28

29 Example: licenses (PyPI) SPDX identifiers? Classifiers: License :: OSI Approved :: BSD License BSD2 or BSD3 or . . . ? https://github.com/pypa/warehouse/issues/2996 “license” field (setup.{cfg,py}) For values not available in the standard classifiers ‘BSD2’, ‘BSD 2’, ‘BSD-2’, ‘BSD 2 Clause’. . . Package software for any distribution with upt 10/28

30 Example: dependencies (PyPI) pipenv $ curl -s https://pypi.org/pypi/pipenv/json \ | jq .info.requires_dist [ "ordereddict; python_version < \"2.7\"", "requests[security]; python_version < \"2.7\"", "virtualenv", "virtualenv-clone (>=0.2.5)", "setuptools (>=36.2.1)", "certifi", "pip (>=9.0.1)" ] Package software for any distribution with upt 11/28

31 $ wget https://.../botocore-1.12.16-py2.py3-none-any.whl $ unzip botocore-1.12.16-py2.py3-none-any.whl $ cd botocore-1.12.16.dist-info/ $ jq .run_requires metadata.json [ { "requires": [ "jmespath>=0.7.1,<1.0.0", "docutils>=0.10", "urllib3>=1.20,<1.24" ] }, ... ] Example: dependencies (PyPI) botocore $ curl -s https://pypi.org/pypi/botocore/json \ jq .info.requires_dist | null Package software for any distribution with upt 12/28

32 Example: dependencies (PyPI) botocore $ curl -s https://pypi.org/pypi/botocore/json \ | jq .info.requires_dist null $ wget https://.../botocore-1.12.16-py2.py3-none-any.whl $ unzip botocore-1.12.16-py2.py3-none-any.whl $ cd botocore-1.12.16.dist-info/ $ jq .run_requires metadata.json [ { "requires": [ "jmespath>=0.7.1,<1.0.0", "docutils>=0.10", "urllib3>=1.20,<1.24" ] }, ... Package software for any distribution with upt 12/28 ]

33 $ pip download --only-binary :all: MarkupSafe Collecting MarkupSafe Could not find a version that satisfies the \ requirement MarkupSafe (from versions: ) No matching distribution found for MarkupSafe $ cat setup.py ... extras_require= { ' dev ' : ... docs ' ' : ... } Example: dependencies (PyPI) MarkupSafe $ curl -s https://pypi.org/pypi/MarkupSafe/json \ jq .info.requires_dist | null Package software for any distribution with upt 13/28

34 $ cat setup.py ... extras_require= { ' dev ' : ... docs ' ' : ... } Example: dependencies (PyPI) MarkupSafe $ curl -s https://pypi.org/pypi/MarkupSafe/json \ | jq .info.requires_dist null $ pip download --only-binary :all: MarkupSafe Collecting MarkupSafe Could not find a version that satisfies the \ requirement MarkupSafe (from versions: ) No matching distribution found for MarkupSafe Package software for any distribution with upt 13/28

35 Example: dependencies (PyPI) MarkupSafe $ curl -s https://pypi.org/pypi/MarkupSafe/json \ | jq .info.requires_dist null $ pip download --only-binary :all: MarkupSafe Collecting MarkupSafe Could not find a version that satisfies the \ requirement MarkupSafe (from versions: ) No matching distribution found for MarkupSafe $ cat setup.py ... extras_require= { ' ' : ... dev ' docs ' : ... } Package software for any distribution with upt 13/28

36 Example: dependencies (PyPI) Read the JSON file provided by PyPI Read the wheel Read the source code Install the package, run “pip list” What about test dependencies? There might be even more ideas! Package software for any distribution with upt 14/28

37 Unified behaviour Modular Simple What we really need Unified CLI Package software for any distribution with upt 15/28

38 Modular Simple What we really need Unified CLI Unified behaviour Package software for any distribution with upt 15/28

39 Simple What we really need Unified CLI Unified behaviour Modular Package software for any distribution with upt 15/28

40 What we really need Unified CLI Unified behaviour Modular Simple Package software for any distribution with upt 15/28

41 Outline 1 Packaging 2 Automation 3 The Universal Packaging Tool 4 fpm 5 Conclusion Package software for any distribution with upt 16/28

42 Architecture Upstream | | upt | upt | Downstream upt | | Packages Repos | Frontends | Backends ----------------------------------------------------------------------- +------+ | +----------+ | +-----+ | +-------------+ | +----------+ | | | | | | | | | | | | | | | PyPI |-|->| upt-pypi |-|->| |-|->| upt-guix |-|->| Guix pkg | | | | | | | | | | | | | | | +------+ | +----------+ | | | | +-------------+ | +----------+ | | | upt | | | | | +-------------+ | +----------+ +------+ | +----------+ | | | | | | | | | | | | | | | | | CPAN |-|->| upt-cpan |-|->| |-|->| upt-openbsd |-|->| Makefile | | | | | | | | | | | | | | | +------+ | +----------+ | +-----+ | +-------------+ | +----------+ | | | | Package software for any distribution with upt 17/28

43 Available frontends $ upt list-frontends cpan pypi rubygems Available backends $ upt list-backends fedora freebsd guix nix openbsd Usage Install (with modules) $ pip install upt[frontends,backends] Package software for any distribution with upt 18/28

44 Available backends $ upt list-backends fedora freebsd guix nix openbsd Usage Install (with modules) $ pip install upt[frontends,backends] Available frontends $ upt list-frontends cpan pypi rubygems Package software for any distribution with upt 18/28

45 Usage Install (with modules) $ pip install upt[frontends,backends] Available frontends $ upt list-frontends cpan pypi rubygems Available backends $ upt list-backends fedora freebsd guix nix openbsd Package software for any distribution with upt 18/28

46 Usage Package upt for OpenBSD $ upt package -f pypi -b openbsd -o /tmp/py-upt upt [INFO ] [Backend] Creating the directory \ structure in /tmp/py-upt [INFO ] [Backend] Creating the Makefile [INFO ] [Backend] Creating pkg/DESCR [INFO ] [Backend] You still need to create \ /tmp/py-upt/distinfo [INFO ] [Backend] You still need to create \ /tmp/py-upt/pkg/PLIST Package software for any distribution with upt 19/28

47 Usage $ less /tmp/py-upt/Makefile # $OpenBSD$ COMMENT= package software from any package manager to any distribution MODPY_EGG_VERSION= 0.4.1 DISTNAME= upt-${MODPY_EGG_VERSION} PKGNAME= py-upt-${MODPY_EGG_VERSION} CATEGORIES= XXX HOMEPAGE= https://framagit.org/upt/upt MAINTAINER= XXX # BSD-3-Clause PERMIT_PACKAGE_CDROM = Yes MODULES= lang/python MODPY_SETUPTOOLS= Yes MODPY_PI= Yes RUN_DEPENDS= xxx/py-spdx-lookup${MODPY_FLAVOR} .include Package software for any distribution with upt 20/28

48 Usage $ upt package -f pypi -b nix upt stdenv, buildPythonPackage, fetchPypi { } : , spdx-lookup buildPythonPackage rec { name = "${pname}-${version}" ; version = "0.4.1" ; pname = "upt" ; src = fetchPypi { inherit pname version ; sha256 = "TODO" ; }; propagatedBuildInputs = [ spdx-lookup ] ; meta = with stdenv.lib ; { description = ... homepage = https://framagit.org/upt/upt ; license = [ licenses.bsd3 ] ; } Package software for any distribution with upt 21/28 }

49 Easier for “small” projects More independence Modularity One module (backend or frontend) -> one Git repo Package software for any distribution with upt 22/28

50 More independence Modularity One module (backend or frontend) -> one Git repo Easier for “small” projects Package software for any distribution with upt 22/28

51 Modularity One module (backend or frontend) -> one Git repo Easier for “small” projects More independence Package software for any distribution with upt 22/28

52 Add a backend or frontend $ cookiecutter https://framagit.org/upt/cookiecutter-upt project_name []: upt-archlinux project_slug [upt_archlinux]: author []: John Doe email []: [email protected] short_description []: Arch Linux backend for upt. url [https://framagit.org/upt/upt-archlinux]: Select license: 1 - BSD-3 2 - other Choose from 1, 2 (1, 2) [1]: 1 Select kind: 1 - backend 2 - frontend Choose from 1, 2 (1, 2) [1]: 1 Initialized empty Git repository in /tmp/upt-archlinux/.git/ Package software for any distribution with upt 23/28

53 Outline 1 Packaging 2 Automation 3 The Universal Packaging Tool 4 fpm 5 Conclusion Package software for any distribution with upt 24/28

54 Modules are distributed along with the core Generates a .rpm instead of a .spec (or .deb instead of a debian/ folder) Useful for users who need to use their distro’s package manager Not the best tool for package maintainers? A different approcah Ruby Package software for any distribution with upt 25/28

55 Generates a .rpm instead of a .spec (or .deb instead of a debian/ folder) Useful for users who need to use their distro’s package manager Not the best tool for package maintainers? A different approcah Ruby Modules are distributed along with the core Package software for any distribution with upt 25/28

56 Useful for users who need to use their distro’s package manager Not the best tool for package maintainers? A different approcah Ruby Modules are distributed along with the core Generates a .rpm instead of a .spec (or .deb instead of a debian/ folder) Package software for any distribution with upt 25/28

57 Not the best tool for package maintainers? A different approcah Ruby Modules are distributed along with the core Generates a .rpm instead of a .spec (or .deb instead of a debian/ folder) Useful for users who need to use their distro’s package manager Package software for any distribution with upt 25/28

58 A different approcah Ruby Modules are distributed along with the core Generates a .rpm instead of a .spec (or .deb instead of a debian/ folder) Useful for users who need to use their distro’s package manager Not the best tool for package maintainers? Package software for any distribution with upt 25/28

59 Outline 1 Packaging 2 Automation 3 The Universal Packaging Tool 4 fpm 5 Conclusion Package software for any distribution with upt 26/28

60 Type hints Need a standard format for all upstream package archives Future work More modules (written by you?) Package software for any distribution with upt 27/28

61 Need a standard format for all upstream package archives Future work More modules (written by you?) Type hints Package software for any distribution with upt 27/28

62 Future work More modules (written by you?) Type hints Need a standard format for all upstream package archives Package software for any distribution with upt 27/28

63 Join me! Code: https://framagit.org/upt Emails: [email protected], [email protected] Freenode: #upt-packaging, Steap Questions? Package software for any distribution with upt 28/28

Related documents

DoD7045.7H

DoD7045.7H

DoD 7045.7-H EPARTMENT OF D EFENSE D F UTURE Y EARS D EFENSE P ROGRAM (FYDP) S TRUCTURE Codes and Definitions for All DoD Components Office of the Director, Program Analysis and Evaluation A pril 2004

More info »
Description of STM32F1 HAL and Low layer drivers

Description of STM32F1 HAL and Low layer drivers

UM1850 User manual Description of STM32F1 HAL and Low - layer drivers Introduction TM is an STMicroelectronics original initiative to make developers' lives easier by reducing STMCube development effo...

More info »
JO 7210.3Y   Facility Operation and Administration

JO 7210.3Y Facility Operation and Administration

ORDER Y JO 7210.3 Air Traffic Organization Policy Effective Date: April 3, 2014 Facility Operation and Administration SUBJ: This order provides direction and guidance for the day- to-day operation of ...

More info »
Same Visit Contraceptive Services Coding Examples

Same Visit Contraceptive Services Coding Examples

Same-Visit Contraceptive Services Coding Examples CPT = Current Procedural Terminology LARC Device Codes E/M = Evaluation & Management CPT Code ICD-10 = Diagnostic code Kyleena IUD HCPCS code J7296 HC...

More info »
System Automation for z/OS: End to End Automation

System Automation for z/OS: End to End Automation

S IBM s t e m A u t o m a ti o n f o r z/O S y n d A o- E n E u t o m a ti o n d-t l e s i o n 4 R e r e a s e 1 V SC34-2750-02

More info »
System Automation for z/OS: End to End Automation

System Automation for z/OS: End to End Automation

S IBM s t e m A u t o m a ti o n f o r z/O S y n d A o- E n E u t o m a ti o n d-t l e s i o n 4 R e r e a s e 1 V SC34-2750-02

More info »
small claims manual

small claims manual

Claims Sma ll Manual (201 4) Ind iana Judicia l Center 30 South ridian Street, Suite 900 Me ianapolis , Ind iana 46204- 3564 Ind

More info »
six books

six books

SIX BOOKS OF THE COMMONWEALTH by ODIN JEAN B M. J. TOOLEY Abridged and translated by L BL ACKWEL L OXF ORD BASI PRINT ED IN GREAT BR ITAI N IN THE CI TY OF OX FORD AT T HE AL DEN PRESS BOUND BY ALL BI...

More info »
STM32CubeMX for STM32 configuration and initialization C code generation   User manual

STM32CubeMX for STM32 configuration and initialization C code generation User manual

UM1718 User manual M32 configuration STM32CubeMX for ST and initialization C code generation Introduction STM32CubeMX is a graphical tool for STM32 microcontrollers. It is part of the STM32Cube™ initi...

More info »
hx 2013 2017 11 hours of work

hx 2013 2017 11 hours of work

ARTICLE 11 HOURS OF WORK EXEMPT EMPLOYEES A. 1. The normal workweek for a full - time exempt employee is considered to be time exempt employees the proportion of 40 hours 40 hours, and for part - emph...

More info »
HANDBOOK of METAL ETCHANTS

HANDBOOK of METAL ETCHANTS

HANDBOOK of METAL ETCHANTS Editors Perrin Walker William H. Tarn CRC Press Boca Raton Boston London New York Washington, D.C. © 1991 by CRC Press LLC

More info »
Developmentally Regulated Genome Editing in Terminally Differentiated N2 Fixing Heterocysts of Anabaena cylindrica ATCC 29414

Developmentally Regulated Genome Editing in Terminally Differentiated N2 Fixing Heterocysts of Anabaena cylindrica ATCC 29414

bioRxiv preprint first posted online May. 7, 2019; doi: http://dx.doi.org/10.1101/629832 . The copyright holder for this preprint (which was not peer-reviewed) is the author/funder, who has granted bi...

More info »
The Digital Person: Technology and Privacy in the Information Age

The Digital Person: Technology and Privacy in the Information Age

GW Law Faculty Publications & Othe r Works Faculty Scholarship 2004 y in the g it al P e rson: T echnolog y a nd P r i v The D c i a I m a t ion A ge nfor a niel J . S olo ve D Ge orge W ashington U n...

More info »
ayout 1

ayout 1

0465039146-FM:FM 12/5/06 12:25 AM Page i C O D E

More info »
Microsoft Word   ElSalvador Report.doc

Microsoft Word ElSalvador Report.doc

- year war in El Salvador: From Madness to Hope: the 12 1 Report of the Commission on the Truth for El Salvador The Commission on the Truth for El Salvador Belisario Betancur Chairman Reinaldo Figuere...

More info »
doj final opinion

doj final opinion

UNITED STAT ES DIS TRICT COURT IC F OR THE D ISTR T OF CO LU M BIA UNITED STAT F AMERICA, : ES O : : la in t if f, P 99 No. on cti l A vi Ci : 96 (GK) -24 : and : TOBACCO-F UND, : REE KIDS ACTION F : ...

More info »
When Main Street is a State Highway

When Main Street is a State Highway

MAIN STREET WHEN ST ATE HIGHW AY IS A BLENDING FUNCTION, BEAUT Y AND IDENTIT Y A HANDBOOK FOR C OMMUNITIES AND DESIGNERS

More info »
main

main

What’s in Main Tobias Nipkow August 15, 2018 Abstract This document lists the main types, functions and syntax provided by theory Main . It is meant as a quick overview of what is available. For infix...

More info »