Closes #23472: Add a plugin API for registering export formats (#23519)

* Closes #23472: Add a plugin API for registering export formats

Introduce a registry of exporters for object list views. Plugins register
Exporter or TableExporter subclasses with register_exporter(), and the
export_format query parameter selects one. NetBox's CSV and YAML exports
become registered exporters, with a single CSV code path for buffered and
streaming responses, so tablib is no longer needed.

* Move YAML export prefetching to the models and drop get_export_queryset()

YAMLExporter now applies the prefetching named by a model's yaml_prefetch
attribute, along with the FETCH_PEERS fetch mode, so ObjectListView no
longer needs an export queryset hook.

* Move SLUG_RE to utilities.constants for reuse by event rule actions and exporters

* Address review feedback on exporter responses

- Stream an exporter's iterable only if STREAMING_EXPORTS is enabled
- Return bytearray and memoryview data as regular content
- Raise a clear error when export() is called on a table exporter

* Tweak docs wording

* Restore backward compatibility for export_yaml() methods on views

* Fetch records in chunks only when streaming

* Don't require a request when calling the deprecated export_yaml() method

* Document YAML export support and deprecated ObjectListView export methods

* Remove oudated comment

* Merge TableExporter into Exporter

Tabular exporters now set tabular = True and receive the table's headers and rows as optional arguments to export(), replacing export_rows().

* Retain utilities.export and tablib as deprecated until NetBox v5.0

* Consolidate CSV delimiter lookup and allow CSV export without a request

* Test export ordering and document the ordering of the queryset passed to exporters

* Validate the yaml_prefetch attribute of models

* Test explicit YAML export of device types and module types

* Remove the export method deprecation warning from the plugin views docs (handled under #23539)

* Prefetch all related objects read by to_yaml() on device and module types

* Return a 400 response when export_format is combined with an export template

base_requirements.txt

@@ -176,6 +176,7 @@ svgwrite
 # Tabular dataset library (for table-based exports)
 # Current: https://github.com/jazzband/tablib/releases
 # Previous: https://github.com/jazzband/tablib/blob/master/HISTORY.md
+# TODO: Remove in NetBox v5.0 along with utilities.export
 tablib
 
 # Timezone data (required by django-timezone-field on Python 3.9+)

docs/configuration/miscellaneous.md

@@ -127,7 +127,7 @@ The maximum size (in bytes) of an incoming HTTP request (i.e. `GET` or `POST` da
 
 Default: `False`
 
-When set to `True`, CSV bulk exports are returned as a streaming HTTP response, emitting rows to the client as they are rendered rather than buffering the entire dataset in memory first. This can significantly reduce memory usage and time-to-first-byte for very large exports.
+When set to `True`, CSV bulk exports (and exports from any plugin exporters which return an iterable) are returned as a streaming HTTP response, emitting rows to the client as they are rendered rather than buffering the entire dataset in memory first. This can significantly reduce memory usage and time-to-first-byte for very large exports.
 
 Because streaming responses do not have a `Content-Length` header and defer errors until after the response has begun, this behavior is opt-in.
 

docs/plugins/development/exporters.md

@@ -0,0 +1,105 @@
+# Exporters
+
+!!! info "This feature was introduced in NetBox v4.8."
+
+The "Export" button on object list views offers a choice of export formats. NetBox provides CSV and (for certain object types) YAML exporters. Plugins can register their own exporters to add formats, such as XLSX or JSON, by subclassing NetBox's `Exporter` class and implementing its `export()` method.
+
+An exporter with `tabular = True` exports the rows of the list view's table. NetBox resolves the table's columns and passes `export()` a list of column headers (`headers`) and an iterator of cell values for each row (`rows`). Tabular exporters are offered under both "Current View" (the user's selected columns) and "All Data" (all columns).
+
+```python title="exporters.py"
+import io
+
+from openpyxl import Workbook
+
+from netbox.exporters import Exporter, register_exporter
+
+@register_exporter
+class XLSXExporter(Exporter):
+    slug = 'my_plugin.xlsx'
+    label = 'Excel (XLSX)'
+    content_type = 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
+    file_extension = 'xlsx'
+    tabular = True
+
+    def export(self, *, request, queryset, headers=None, rows=None):
+        workbook = Workbook(write_only=True)
+        sheet = workbook.create_sheet()
+        sheet.append(headers)
+        for row in rows:
+            sheet.append(row)
+        buffer = io.BytesIO()
+        workbook.save(buffer)
+        return buffer.getvalue()
+```
+
+A non-tabular exporter works on the queryset directly, and is offered under "All Data" only. Override `supports_model()` to limit any exporter to specific models.
+
+```python title="exporters.py"
+from netbox.exporters import Exporter, register_exporter
+
+@register_exporter
+class VendorDeviceTypeExporter(Exporter):
+    slug = 'my_plugin.vendor_device_types'
+    label = 'Vendor Format'
+    content_type = 'application/json'
+    file_extension = 'json'
+
+    def supports_model(self, model):
+        return model._meta.label == 'dcim.DeviceType'
+
+    def export(self, *, request, queryset, headers=None, rows=None):
+        ...
+```
+
+The queryset passed to an exporter has the user's filters applied, and is restricted to objects the user has permission to view. It isn't sorted to match the list view's table; only the rows passed to a tabular exporter follow the table's sort order. `export()` returns the exported data as a string or bytes, or as an iterable of chunks. NetBox streams an iterable to the client if [`STREAMING_EXPORTS`](../../configuration/miscellaneous.md#streaming_exports) is enabled; otherwise, it collects the chunks before responding.
+
+The `register_exporter()` decorator registers an exporter with NetBox. Exporters can be defined anywhere within a plugin, but must be imported during plugin initialization, under the PluginConfig's `ready()` method:
+
+```python
+def ready(self):
+    super().ready()
+    from my_plugin import exporters
+```
+
+`slug` must begin with a lowercase letter, and may contain only letters, digits, underscores, and dot-separated segments. A dotted namespace prefix (e.g. `my_plugin.xlsx`) is strongly recommended to avoid collisions with other plugins or with exporters added to NetBox in the future.
+
+!!! warning "Exporters must be stateless"
+    Registration instantiates the class once, and that single instance serves every request. Don't store per-request data on `self`.
+
+## YAML Export
+
+NetBox's YAML exporter supports any model which defines a `to_yaml()` method, including plugin models. The method returns a YAML document representing the object, and the exporter joins the documents for all exported objects.
+
+To avoid a database query per object, set `yaml_prefetch` on the model to the related objects which `to_yaml()` accesses through reverse or many-to-many relations. The exporter passes these to the queryset's `prefetch_related()`. Related objects accessed through foreign keys are fetched in batches automatically. This doesn't extend to objects loaded by `yaml_prefetch`, so if `to_yaml()` reads a foreign key on them, include it in the lookup (e.g. `parts__supplier`).
+
+```python title="models.py"
+import yaml
+
+from netbox.models import NetBoxModel
+
+class Widget(NetBoxModel):
+    ...
+    yaml_prefetch = ('parts',)
+
+    def to_yaml(self):
+        data = {
+            'name': self.name,
+            'parts': [part.name for part in self.parts.all()],
+        }
+        return yaml.dump(data, sort_keys=False)
+```
+
+## Export Formats
+
+The `export_format` query parameter selects an exporter by its slug:
+
+| URL                                  | Result                                                 |
+|--------------------------------------|--------------------------------------------------------|
+| `?export=table&export_format=<slug>` | Current view, in the given format                      |
+| `?export&export_format=<slug>`       | All data, in the given format                          |
+| `?export=table`                      | Current view, as CSV                                   |
+| `?export`                            | All data, as [YAML](#yaml-export) if the model supports it, or as CSV |
+
+An unknown slug, or an exporter that doesn't support the model (or the current view), returns a 404 response. Combining `export_format` with an export template (`?export=<name>`) returns a 400 response. Exporters are available only in the web UI. The REST API supports [export templates](../../customization/export-templates.md) only.
+
+::: netbox.exporters.Exporter

docs/plugins/development/views.md

@@ -154,7 +154,6 @@ Below are the class definitions for NetBox's multi-object views. These views han
     options:
       members:
         - get_table
-        - export_table
         - export_template
 
 !!! warning "Deprecation Warning"

mkdocs.yml

@@ -156,6 +156,7 @@ nav:
             - Search: 'plugins/development/search.md'
             - Event Types: 'plugins/development/event-types.md'
             - Event Rule Actions: 'plugins/development/event-rule-actions.md'
+            - Exporters: 'plugins/development/exporters.md'
             - Permissions: 'plugins/development/permissions.md'
             - Data Backends: 'plugins/development/data-backends.md'
             - Webhooks: 'plugins/development/webhooks.md'

netbox/core/apps.py

@@ -45,6 +45,7 @@ class CoreConfig(AppConfig):
             check_redis_version,
         )
         from netbox import context_managers  # noqa: F401
+        from netbox.exporters import CSVExporter, YAMLExporter, register_exporter
         from netbox.models.features import register_models
 
         # Importing netbox.search.jobs registers its system job(s) (e.g. the
@@ -68,6 +69,10 @@ class CoreConfig(AppConfig):
         EventType(JOB_FAILED, _('Job failed'), kind=EVENT_TYPE_KIND_WARNING).register()
         EventType(JOB_ERRORED, _('Job errored'), kind=EVENT_TYPE_KIND_DANGER).register()
 
+        # Register core exporters
+        register_exporter(CSVExporter)
+        register_exporter(YAMLExporter)
+
         # Clear Redis cache on startup in development mode
         if settings.DEBUG:
             try:

netbox/dcim/models/devices.py

@@ -220,6 +220,24 @@ class DeviceType(ImageAttachmentsMixin, PrimaryModel, WeightMixin):
     prerequisite_models = (
         'dcim.Manufacturer',
     )
+    # Related objects read by to_yaml(). Prefetched objects don't inherit the export's fetch mode, so the
+    # foreign keys of component templates are listed too.
+    yaml_prefetch = (
+        'consoleporttemplates',
+        'consoleserverporttemplates',
+        'powerporttemplates',
+        'poweroutlettemplates__power_port',
+        'coolingintaketemplates',
+        'coolingoutflowtemplates__cooling_intake',
+        'interfacetemplates__parent',
+        'interfacetemplates__bridge',
+        'frontporttemplates',
+        'rearporttemplates',
+        'port_mappings__front_port',
+        'port_mappings__rear_port',
+        'modulebaytemplates__module_bay_types',
+        'devicebaytemplates',
+    )
 
     class Meta:
         ordering = ['manufacturer', 'model']

netbox/dcim/models/modules.py

@@ -227,6 +227,23 @@ class ModuleType(ImageAttachmentsMixin, PrimaryModel, WeightMixin):
     prerequisite_models = (
         'dcim.Manufacturer',
     )
+    # Related objects read by to_yaml(). Prefetched objects don't inherit the export's fetch mode, so the
+    # foreign keys of component templates are listed too.
+    yaml_prefetch = (
+        'module_bay_types',
+        'consoleporttemplates',
+        'consoleserverporttemplates',
+        'powerporttemplates',
+        'poweroutlettemplates__power_port',
+        'coolingintaketemplates',
+        'coolingoutflowtemplates__cooling_intake',
+        'interfacetemplates__parent',
+        'interfacetemplates__bridge',
+        'frontporttemplates',
+        'rearporttemplates',
+        'port_mappings__front_port',
+        'port_mappings__rear_port',
+    )
 
     class Meta:
         ordering = ('profile', 'manufacturer', 'model')

netbox/dcim/tests/test_views.py

@@ -22,7 +22,6 @@ from core.models import ObjectChange, ObjectType
 from dcim.choices import *
 from dcim.constants import *
 from dcim.models import *
-from dcim.views import DeviceTypeListView, ModuleTypeListView
 from extras.models import ConfigContext, ConfigTemplate
 from ipam.models import ASN, RIR, VLAN, VRF, IPAddress, VLANGroup
 from netbox.choices import (
@@ -32,6 +31,7 @@ from netbox.choices import (
     ImportFormatChoices,
     WeightUnitChoices,
 )
+from netbox.exporters import get_exporter
 from tenancy.models import Tenant
 from users.models import ObjectPermission, Owner, User
 from utilities.testing import ViewTestCases, create_tags, create_test_device, post_data
@@ -800,6 +800,25 @@ class ManufacturerTestCase(ViewTestCases.OrganizationalObjectViewTestCase):
         }
 
 
+def create_yaml_export_templates(**parent):
+    """
+    Create one of each component template exported by to_yaml(), with its related templates, on a device or module type.
+    """
+    ConsolePortTemplate.objects.create(name='Console Port', **parent)
+    ConsoleServerPortTemplate.objects.create(name='Console Server Port', **parent)
+    power_port = PowerPortTemplate.objects.create(name='Power Port', **parent)
+    PowerOutletTemplate.objects.create(name='Power Outlet', power_port=power_port, **parent)
+    cooling_intake = CoolingIntakeTemplate.objects.create(name='Cooling Intake', **parent)
+    CoolingOutflowTemplate.objects.create(name='Cooling Outflow', cooling_intake=cooling_intake, **parent)
+    lag = InterfaceTemplate.objects.create(name='LAG', type=InterfaceTypeChoices.TYPE_LAG, **parent)
+    InterfaceTemplate.objects.create(
+        name='Interface', type=InterfaceTypeChoices.TYPE_1GE_FIXED, parent=lag, bridge=lag, **parent
+    )
+    front_port = FrontPortTemplate.objects.create(name='Front Port', type=PortTypeChoices.TYPE_8P8C, **parent)
+    rear_port = RearPortTemplate.objects.create(name='Rear Port', type=PortTypeChoices.TYPE_8P8C, **parent)
+    PortTemplateMapping.objects.create(front_port=front_port, rear_port=rear_port)
+
+
 # TODO: Change base class to PrimaryObjectViewTestCase
 # Blocked by absence of bulk import view for DeviceTypes
 class DeviceTypeTestCase(
@@ -1270,33 +1289,38 @@ inventory-items:
         ii1 = InventoryItemTemplate.objects.first()
         self.assertEqual(ii1.name, 'Inventory Item 1')
 
-    def test_bulk_yaml_export_module_bay_types_query_count_is_constant(self):
-        """Query count shouldn't scale with bay count -- module_bay_types is prefetched."""
+    def test_bulk_yaml_export_query_count_is_constant(self):
+        """Query count shouldn't scale with the number of device types or module bays."""
         manufacturer = Manufacturer.objects.create(name='Export Query Manufacturer', slug='export-query-mfr')
         bay_type = ModuleBayType.objects.create(name='Export Query SFP28', slug='export-query-sfp28')
 
-        def make_device_type(model_name, bay_count):
-            device_type = DeviceType.objects.create(
-                manufacturer=manufacturer, model=model_name, slug=model_name.lower().replace(' ', '-'),
-            )
-            for i in range(bay_count):
-                bay = ModuleBayTemplate.objects.create(device_type=device_type, name=f'Bay {i}')
-                bay.module_bay_types.set([bay_type])
-            return device_type
-
-        one_bay_device_type = make_device_type('Export Query DT One Bay', 1)
-        five_bay_device_type = make_device_type('Export Query DT Five Bays', 5)
-
-        view = DeviceTypeListView()
-        view.queryset = DeviceType.objects.filter(pk=one_bay_device_type.pk)
-        with CaptureQueriesContext(connection) as one_bay_queries:
-            view.export_yaml()
-
-        view.queryset = DeviceType.objects.filter(pk=five_bay_device_type.pk)
-        with CaptureQueriesContext(connection) as five_bay_queries:
-            view.export_yaml()
-
-        self.assertEqual(len(one_bay_queries), len(five_bay_queries))
+        def make_device_types(count):
+            pks = []
+            for i in range(count):
+                device_type = DeviceType.objects.create(
+                    manufacturer=manufacturer,
+                    model=f'Export Query DT {count}-{i}',
+                    slug=f'export-query-dt-{count}-{i}',
+                )
+                create_yaml_export_templates(device_type=device_type)
+                DeviceBayTemplate.objects.create(device_type=device_type, name='Device Bay')
+                for j in range(count):
+                    bay = ModuleBayTemplate.objects.create(device_type=device_type, name=f'Bay {j}')
+                    bay.module_bay_types.set([bay_type])
+                pks.append(device_type.pk)
+            return DeviceType.objects.filter(pk__in=pks)
+
+        one_device_type = make_device_types(1)
+        five_device_types = make_device_types(5)
+
+        exporter = get_exporter('yaml')
+        with CaptureQueriesContext(connection) as one_queries:
+            exporter.export(request=None, queryset=one_device_type)
+
+        with CaptureQueriesContext(connection) as five_queries:
+            exporter.export(request=None, queryset=five_device_types)
+
+        self.assertEqual(len(one_queries), len(five_queries))
 
     def test_import_channelized_interfaces(self):
         """
@@ -1503,13 +1527,16 @@ console-ports:
         url = reverse('dcim:devicetype_list')
         self.add_permissions('dcim.view_devicetype')
 
-        # Test default YAML export
-        response = self.client.get(f'{url}?export')
-        self.assertEqual(response.status_code, 200)
-        data = list(yaml.load_all(response.content, Loader=yaml.SafeLoader))
-        self.assertEqual(len(data), 3)
-        self.assertEqual(data[0]['manufacturer'], 'Manufacturer 1')
-        self.assertEqual(data[0]['model'], 'Device Type 1')
+        # Test YAML export (the default, or requested explicitly)
+        for params in ('export', 'export&export_format=yaml'):
+            with self.subTest(params=params):
+                response = self.client.get(f'{url}?{params}')
+                self.assertEqual(response.status_code, 200)
+                self.assertEqual(response.get('Content-Type'), 'text/yaml')
+                data = list(yaml.load_all(response.content, Loader=yaml.SafeLoader))
+                self.assertEqual(len(data), 3)
+                self.assertEqual(data[0]['manufacturer'], 'Manufacturer 1')
+                self.assertEqual(data[0]['model'], 'Device Type 1')
 
         # Test table-based export (streams row-by-row)
         response = self.client.get(f'{url}?export=table')
@@ -1521,6 +1548,14 @@ console-ports:
         self.assertGreater(len(rows), 1)
         self.assertEqual(len(rows) - 1, DeviceType.objects.count())
 
+        # Test CSV export of all data
+        response = self.client.get(f'{url}?export&export_format=csv')
+        self.assertHttpStatus(response, 200)
+        self.assertEqual(response.get('Content-Type'), 'text/csv; charset=utf-8')
+        content = b''.join(response.streaming_content).decode('utf-8')
+        rows = list(csv.reader(StringIO(content)))
+        self.assertEqual(len(rows) - 1, DeviceType.objects.count())
+
 
 class ModuleTypeTestCase(ViewTestCases.PrimaryObjectViewTestCase):
     model = ModuleType
@@ -2176,45 +2211,47 @@ interfaces:
             self.assertEqual(channel.channel_id, i)
             self.assertEqual(channel.parent, imported_parent)
 
-    def test_bulk_yaml_export_prefetches_module_bay_types_on_the_module_type_itself(self):
-        """Compares an unprefetched to_yaml() call per instance against export_yaml() (which
-        prefetches and issues no queries of its own beyond that), rather than a row-count
-        comparison, which other per-instance relations that legitimately scale with it would
-        swamp."""
+    def test_bulk_yaml_export_query_count_is_constant(self):
+        """Query count shouldn't scale with the number of module types."""
         manufacturer = Manufacturer.objects.create(name='Export Query MT Manufacturer', slug='export-query-mt-mfr')
         bay_type = ModuleBayType.objects.create(name='Export Query MT SFP28', slug='export-query-mt-sfp28')
 
-        module_types = []
-        for i in range(5):
-            module_type = ModuleType.objects.create(manufacturer=manufacturer, model=f'Export Query MT {i}')
-            module_type.module_bay_types.set([bay_type])
-            module_types.append(module_type)
-        pks = [mt.pk for mt in module_types]
+        def make_module_types(count):
+            pks = []
+            for i in range(count):
+                module_type = ModuleType.objects.create(manufacturer=manufacturer, model=f'Export Query MT {count}-{i}')
+                module_type.module_bay_types.set([bay_type])
+                create_yaml_export_templates(module_type=module_type)
+                pks.append(module_type.pk)
+            return ModuleType.objects.filter(pk__in=pks)
+
+        one_module_type = make_module_types(1)
+        five_module_types = make_module_types(5)
 
-        with CaptureQueriesContext(connection) as unprefetched:
-            [obj.to_yaml() for obj in ModuleType.objects.filter(pk__in=pks)]
+        exporter = get_exporter('yaml')
+        with CaptureQueriesContext(connection) as one_queries:
+            exporter.export(request=None, queryset=one_module_type)
 
-        view = ModuleTypeListView()
-        view.queryset = ModuleType.objects.filter(pk__in=pks)
-        with CaptureQueriesContext(connection) as prefetched:
-            view.export_yaml()
+        with CaptureQueriesContext(connection) as five_queries:
+            exporter.export(request=None, queryset=five_module_types)
 
-        # Without the prefetch, each of the 5 module types issues its own module_bay_types
-        # query; with it, exactly one query serves all 5.
-        self.assertEqual(len(unprefetched) - len(prefetched), 4)
+        self.assertEqual(len(one_queries), len(five_queries))
 
     @override_settings(STREAMING_EXPORTS=True)
     def test_export_objects(self):
         url = reverse('dcim:moduletype_list')
         self.add_permissions('dcim.view_moduletype')
 
-        # Test default YAML export
-        response = self.client.get(f'{url}?export')
-        self.assertEqual(response.status_code, 200)
-        data = list(yaml.load_all(response.content, Loader=yaml.SafeLoader))
-        self.assertEqual(len(data), 3)
-        self.assertEqual(data[0]['manufacturer'], 'Manufacturer 1')
-        self.assertEqual(data[0]['model'], 'Module Type 1')
+        # Test YAML export (the default, or requested explicitly)
+        for params in ('export', 'export&export_format=yaml'):
+            with self.subTest(params=params):
+                response = self.client.get(f'{url}?{params}')
+                self.assertEqual(response.status_code, 200)
+                self.assertEqual(response.get('Content-Type'), 'text/yaml')
+                data = list(yaml.load_all(response.content, Loader=yaml.SafeLoader))
+                self.assertEqual(len(data), 3)
+                self.assertEqual(data[0]['manufacturer'], 'Manufacturer 1')
+                self.assertEqual(data[0]['model'], 'Module Type 1')
 
         # Test table-based export (streams row-by-row)
         response = self.client.get(f'{url}?export=table')
@@ -2226,6 +2263,14 @@ interfaces:
         self.assertGreater(len(rows), 1)
         self.assertEqual(len(rows) - 1, ModuleType.objects.count())
 
+        # Test CSV export of all data
+        response = self.client.get(f'{url}?export&export_format=csv')
+        self.assertHttpStatus(response, 200)
+        self.assertEqual(response.get('Content-Type'), 'text/csv; charset=utf-8')
+        content = b''.join(response.streaming_content).decode('utf-8')
+        rows = list(csv.reader(StringIO(content)))
+        self.assertEqual(len(rows) - 1, ModuleType.objects.count())
+
     @tag('regression')  # Issue #22961
     def test_bulk_edit_module_bay_types(self):
         """

netbox/dcim/views.py

@@ -1449,11 +1449,6 @@ class DeviceTypeListView(generic.ObjectListView):
     filterset_form = forms.DeviceTypeFilterForm
     table = tables.DeviceTypeTable
 
-    def export_yaml(self):
-        # Avoid one module_bay_types query per module bay template across the export.
-        self.queryset = self.queryset.prefetch_related('modulebaytemplates__module_bay_types')
-        return super().export_yaml()
-
 
 @register_model_view(DeviceType)
 class DeviceTypeView(GetRelatedModelsMixin, generic.ObjectView):
@@ -1914,13 +1909,6 @@ class ModuleTypeListView(generic.ObjectListView):
     filterset_form = forms.ModuleTypeFilterForm
     table = tables.ModuleTypeTable
 
-    def export_yaml(self):
-        # Avoid one module_bay_types query per module type across the export. (Unlike
-        # DeviceType.to_yaml(), ModuleType.to_yaml() doesn't export module bay templates at
-        # all, so there's nothing to prefetch alongside it.)
-        self.queryset = self.queryset.prefetch_related('module_bay_types')
-        return super().export_yaml()
-
 
 @register_model_view(ModuleType)
 class ModuleTypeView(GetRelatedModelsMixin, generic.ObjectView):

netbox/netbox/event_rules.py

@@ -1,11 +1,10 @@
-import re
-
 from django.core.exceptions import ImproperlyConfigured, ValidationError
 from django.utils.text import capfirst
 from django.utils.translation import gettext_lazy as _
 
 from netbox.registry import registry
 from utilities.choices import Choice
+from utilities.constants import SLUG_RE
 from utilities.string import enum_key
 
 __all__ = (
@@ -16,11 +15,6 @@ __all__ = (
     'register_event_rule_action',
 )
 
-# A slug must sanitize (via enum_key()) into a valid GraphQL enum member name, which rules out a
-# leading digit or underscore. Hyphens are rejected outright rather than sanitized away, so that a
-# slug always reads as it was written.
-SLUG_RE = re.compile(r'^[a-z][a-z0-9_]*(\.[a-z0-9_]+)*$')
-
 
 # This module must not import any concrete Django models: it's imported by netbox.plugins (itself
 # imported by netbox.settings, before the app registry is populated), so only registry-level

netbox/netbox/exporters.py

@@ -0,0 +1,157 @@
+import csv
+from collections.abc import Iterable
+from itertools import chain
+
+from django.core.exceptions import ImproperlyConfigured
+from django.db.models import FETCH_PEERS
+from django.utils.translation import gettext as _
+
+from netbox.registry import registry
+from utilities.constants import CSV_DELIMITERS, SLUG_RE
+
+__all__ = (
+    'CSVExporter',
+    'Exporter',
+    'YAMLExporter',
+    'get_exporter',
+    'get_exporters',
+    'register_exporter',
+)
+
+
+class Exporter:
+    """
+    Base class for an export format offered on object list views.
+
+    Registration instantiates the class once, and that instance serves every request, so implementations must be
+    stateless.
+
+    Attributes:
+        slug: A unique identifier for this exporter (e.g. "csv"). Plugins should use a dotted namespace prefix
+            (e.g. "my_plugin.xlsx").
+        label: The human-friendly name shown in the UI
+        content_type: The Content-Type of the HTTP response (e.g. "text/csv; charset=utf-8")
+        file_extension: The extension of the downloaded file, without a leading period (e.g. "csv")
+        tabular: If True, export() receives the rows of the list view's table, and the exporter is listed under both
+            "Current View" and "All Data". Otherwise, it is listed under "All Data" only.
+    """
+    slug = None
+    label = None
+    content_type = None
+    file_extension = None
+    tabular = False
+
+    def __repr__(self):
+        return f'<{self.__class__.__name__}: {self.slug}>'
+
+    def supports_model(self, model):
+        """
+        Return True if this exporter can export objects of the given model.
+        """
+        return True
+
+    def export(self, *, request, queryset, headers=None, rows=None):
+        """
+        Return the exported data as str or bytes, or as an iterable of chunks (streamed if STREAMING_EXPORTS is
+        enabled).
+
+        Args:
+            request: The current request
+            queryset: The filtered queryset of objects to export, restricted to those the user can view
+            headers: A list of the table's column headers (tabular exporters only)
+            rows: An iterator yielding a list of cell values for each table row (tabular exporters only)
+        """
+        raise NotImplementedError(f"{self.__class__.__name__} must implement export()")
+
+
+class _EchoBuffer:
+    """
+    File-like object whose write() returns the value written, so that csv.writer output can be captured line by line.
+    """
+    def write(self, value):
+        return value
+
+
+class CSVExporter(Exporter):
+    slug = 'csv'
+    label = 'CSV'
+    content_type = 'text/csv; charset=utf-8'
+    file_extension = 'csv'
+    tabular = True
+
+    def export(self, *, request, queryset, headers=None, rows=None):
+        return self.write_rows(headers, rows, delimiter=self.get_delimiter(request))
+
+    @staticmethod
+    def get_delimiter(request):
+        """
+        Return the name of the requesting user's preferred CSV delimiter, if any.
+        """
+        user = getattr(request, 'user', None)
+        return user.config.get('csv_delimiter') if user is not None and user.is_authenticated else None
+
+    def write_rows(self, headers, rows, delimiter=None):
+        """
+        Return an iterator of CSV lines for the given headers and rows.
+
+        Args:
+            headers: A list of column headers
+            rows: An iterator yielding a list of cell values for each row
+            delimiter: Name of a delimiter in utilities.constants.CSV_DELIMITERS (defaults to 'comma')
+        """
+        if delimiter and delimiter not in CSV_DELIMITERS:
+            raise ValueError(_("Invalid delimiter name: {name}").format(name=delimiter))
+        writer = csv.writer(_EchoBuffer(), delimiter=CSV_DELIMITERS[delimiter or 'comma'])
+        return (writer.writerow(row) for row in chain([headers], rows))
+
+
+class YAMLExporter(Exporter):
+    slug = 'yaml'
+    label = 'YAML'
+    content_type = 'text/yaml'
+    file_extension = 'yaml'
+
+    def supports_model(self, model):
+        return hasattr(model, 'to_yaml')
+
+    def export(self, *, request, queryset, headers=None, rows=None):
+        # Prefetch the relations named by the model, and batch any other foreign key lookups across all objects
+        prefetch = getattr(queryset.model, 'yaml_prefetch', ())
+        if isinstance(prefetch, str) or not isinstance(prefetch, Iterable):
+            raise ValueError(
+                f"{queryset.model.__name__}.yaml_prefetch must be an iterable of lookups, not {type(prefetch).__name__}"
+            )
+        queryset = queryset.prefetch_related(*prefetch).fetch_mode(FETCH_PEERS)
+        return '---\n'.join(obj.to_yaml() for obj in queryset)
+
+
+def register_exporter(cls):
+    """
+    Register an Exporter subclass. Can be used as a decorator.
+    """
+    if not issubclass(cls, Exporter):
+        raise ImproperlyConfigured(f"{cls.__name__} must be a subclass of netbox.exporters.Exporter.")
+    instance = cls()
+    for attr in ('slug', 'label', 'content_type', 'file_extension'):
+        if not getattr(instance, attr):
+            raise ImproperlyConfigured(f"{cls.__name__} must define a '{attr}' attribute.")
+    if not SLUG_RE.fullmatch(instance.slug):
+        raise ImproperlyConfigured(
+            f"Invalid exporter slug {instance.slug!r}: must be lowercase, start with a letter, and use only letters, "
+            f"digits, underscores, and dot-separated segments."
+        )
+    if instance.slug in registry['exporters']:
+        raise ImproperlyConfigured(f"An exporter named {instance.slug} has already been registered!")
+    registry['exporters'][instance.slug] = instance
+    return cls
+
+
+def get_exporter(slug):
+    return registry['exporters'].get(slug)
+
+
+def get_exporters(model):
+    """
+    Return all registered exporters which support the given model.
+    """
+    return [exporter for exporter in registry['exporters'].values() if exporter.supports_model(model)]

netbox/netbox/object_actions.py

@@ -5,6 +5,7 @@ from django.utils.translation import gettext_lazy as _
 
 from core.models import ObjectType
 from extras.models import ExportTemplate
+from netbox.exporters import get_exporters
 from utilities.querydict import prepare_cloned_fields
 from utilities.views import get_action_url
 
@@ -151,9 +152,7 @@ class BulkExport(ObjectAction):
     def get_context(cls, context, model):
         object_type = ObjectType.objects.get_for_model(model)
         user = context['request'].user
-
-        # Determine if the "all data" export returns CSV or YAML
-        data_format = 'YAML' if hasattr(object_type.model_class(), 'to_yaml') else 'CSV'
+        exporters = get_exporters(model)
 
         # Retrieve all export templates for this model
         export_templates = ExportTemplate.objects.restrict(user, 'view').filter(object_types=object_type)
@@ -161,8 +160,9 @@ class BulkExport(ObjectAction):
         return {
             'object_type': object_type,
             'url_params': context['request'].GET.urlencode() if context['request'].GET else '',
+            'table_exporters': [exporter for exporter in exporters if exporter.tabular],
+            'exporters': exporters,
             'export_templates': export_templates,
-            'data_format': data_format,
         }
 
 

netbox/netbox/registry.py

@@ -27,6 +27,7 @@ registry = Registry({
     'data_backends': dict(),
     'event_rule_actions': dict(),
     'event_types': dict(),
+    'exporters': dict(),
     'filtersets': dict(),
     'model_actions': collections.defaultdict(set),
     'model_features': dict(),

netbox/netbox/tests/dummy_plugin/__init__.py

@@ -27,7 +27,7 @@ class DummyPluginConfig(PluginConfig):
     def ready(self):
         super().ready()
 
-        from . import jobs, webhook_callbacks, widgets  # noqa: F401
+        from . import exporters, jobs, webhook_callbacks, widgets  # noqa: F401
 
 
 config = DummyPluginConfig

netbox/netbox/tests/dummy_plugin/exporters.py

@@ -0,0 +1,36 @@
+from netbox.exporters import Exporter, register_exporter
+
+__all__ = (
+    'DummySiteExporter',
+    'DummyTableExporter',
+)
+
+
+@register_exporter
+class DummyTableExporter(Exporter):
+    """Exports table rows as tab-separated values, encoded as bytes."""
+    slug = 'dummy_plugin.tsv'
+    label = 'Dummy TSV'
+    content_type = 'text/tab-separated-values; charset=utf-8'
+    file_extension = 'tsv'
+    tabular = True
+
+    def export(self, *, request, queryset, headers=None, rows=None):
+        lines = ['\t'.join(headers), *('\t'.join('' if value is None else str(value) for value in row) for row in rows)]
+        return '\n'.join(lines).encode()
+
+
+@register_exporter
+class DummySiteExporter(Exporter):
+    """Streams the names of sites, one per line."""
+    slug = 'dummy_plugin.site_names'
+    label = 'Dummy Site Names'
+    content_type = 'text/plain; charset=utf-8'
+    file_extension = 'txt'
+
+    def supports_model(self, model):
+        return model._meta.label_lower == 'dcim.site'
+
+    def export(self, *, request, queryset, headers=None, rows=None):
+        for site in queryset:
+            yield f'{site.name}\n'

netbox/netbox/tests/test_exporters.py

@@ -0,0 +1,356 @@
+import csv
+from collections.abc import Iterator
+from io import StringIO
+from unittest import skipIf
+from unittest.mock import patch
+
+import yaml
+from django.conf import settings
+from django.core.exceptions import ImproperlyConfigured
+from django.db import connection
+from django.db.models import QuerySet
+from django.http import StreamingHttpResponse
+from django.test import RequestFactory, SimpleTestCase, override_settings
+from django.test.utils import CaptureQueriesContext
+from django.urls import reverse
+
+from core.models import ObjectType
+from dcim.models import DeviceType, Manufacturer, Site
+from dcim.views import DeviceTypeListView, SiteListView
+from extras.models import ExportTemplate
+from netbox.exporters import (
+    CSVExporter,
+    Exporter,
+    YAMLExporter,
+    get_exporter,
+    get_exporters,
+    register_exporter,
+)
+from netbox.registry import registry
+from utilities.testing import TestCase
+
+
+class ExporterRegistrationTestCase(SimpleTestCase):
+
+    def _register(self, cls):
+        self.addCleanup(registry['exporters'].pop, cls.slug, None)
+        return register_exporter(cls)
+
+    def test_core_exporters(self):
+        self.assertIsInstance(get_exporter('csv'), CSVExporter)
+        self.assertIsInstance(get_exporter('yaml'), YAMLExporter)
+
+    def test_get_exporters(self):
+        self.assertIn(get_exporter('csv'), get_exporters(Site))
+        self.assertNotIn(get_exporter('yaml'), get_exporters(Site))
+        self.assertIn(get_exporter('csv'), get_exporters(DeviceType))
+        self.assertIn(get_exporter('yaml'), get_exporters(DeviceType))
+
+    def test_register_exporter(self):
+        class TestExporter(Exporter):
+            slug = 'test.exporter'
+            label = 'Test'
+            content_type = 'text/plain'
+            file_extension = 'txt'
+
+        self.assertIs(self._register(TestExporter), TestExporter)
+        self.assertIsInstance(get_exporter('test.exporter'), TestExporter)
+
+    def test_register_exporter_not_subclass(self):
+        class TestExporter:
+            slug = 'test.exporter'
+            label = 'Test'
+            content_type = 'text/plain'
+            file_extension = 'txt'
+
+        with self.assertRaises(ImproperlyConfigured):
+            self._register(TestExporter)
+
+    def test_register_exporter_missing_attrs(self):
+        for attr in ('slug', 'label', 'content_type', 'file_extension'):
+            attrs = {
+                'slug': 'test.exporter',
+                'label': 'Test',
+                'content_type': 'text/plain',
+                'file_extension': 'txt',
+                attr: None,
+            }
+            with self.subTest(attr=attr), self.assertRaises(ImproperlyConfigured):
+                self._register(type('TestExporter', (Exporter,), attrs))
+
+    def test_register_exporter_invalid_slug(self):
+        for slug in ('Test', 'my-plugin.xlsx', '1test', 'test.', '.test'):
+            attrs = {'slug': slug, 'label': 'Test', 'content_type': 'text/plain', 'file_extension': 'txt'}
+            with self.subTest(slug=slug), self.assertRaises(ImproperlyConfigured):
+                self._register(type('TestExporter', (Exporter,), attrs))
+
+    def test_register_exporter_duplicate_slug(self):
+        class TestExporter(Exporter):
+            slug = 'csv'
+            label = 'Test'
+            content_type = 'text/plain'
+            file_extension = 'txt'
+
+        with self.assertRaises(ImproperlyConfigured):
+            register_exporter(TestExporter)
+        self.assertIsInstance(get_exporter('csv'), CSVExporter)
+
+
+class CSVExporterTestCase(SimpleTestCase):
+    headers = ['Name', 'Count']
+    rows = [['Site 1', 1], ['Site, 2', None]]
+
+    def test_write_rows(self):
+        data = get_exporter('csv').write_rows(self.headers, iter(self.rows))
+        self.assertEqual(list(data), ['Name,Count\r\n', 'Site 1,1\r\n', '"Site, 2",\r\n'])
+
+    def test_write_rows_delimiter(self):
+        data = get_exporter('csv').write_rows(self.headers, iter(self.rows), delimiter='semicolon')
+        self.assertEqual(''.join(data), 'Name;Count\r\nSite 1;1\r\nSite, 2;\r\n')
+
+    def test_write_rows_invalid_delimiter(self):
+        with self.assertRaises(ValueError):
+            get_exporter('csv').write_rows(self.headers, iter(self.rows), delimiter='invalid')
+
+    def test_export_without_request(self):
+        data = get_exporter('csv').export(request=None, queryset=None, headers=self.headers, rows=iter(self.rows))
+        self.assertEqual(''.join(data), 'Name,Count\r\nSite 1,1\r\n"Site, 2",\r\n')
+
+
+class YAMLExporterTestCase(TestCase):
+
+    def test_export_fetches_peers(self):
+        """
+        Related objects are fetched for all exported objects at once, rather than one by one.
+        """
+        for i in range(1, 4):
+            manufacturer = Manufacturer.objects.create(name=f'Manufacturer {i}', slug=f'manufacturer-{i}')
+            DeviceType.objects.create(manufacturer=manufacturer, model=f'Device Type {i}', slug=f'device-type-{i}')
+
+        with CaptureQueriesContext(connection) as queries:
+            get_exporter('yaml').export(request=None, queryset=DeviceType.objects.all())
+        manufacturer_queries = [q for q in queries.captured_queries if 'FROM "dcim_manufacturer"' in q['sql']]
+        self.assertEqual(len(manufacturer_queries), 1)
+
+
+class ExportViewTestCase(TestCase):
+
+    @classmethod
+    def setUpTestData(cls):
+        Site.objects.bulk_create([Site(name=f'Site {i}', slug=f'site-{i}') for i in range(1, 4)])
+
+    def setUp(self):
+        super().setUp()
+        self.add_permissions('dcim.view_site')
+        self.url = reverse('dcim:site_list')
+        self.user.config.set('tables.SiteTable.columns', ['name', 'status'], commit=True)
+
+    def get_csv_rows(self, response):
+        return list(csv.reader(StringIO(response.content.decode())))
+
+    def test_export_current_view(self):
+        for params in ('export=table', 'export=table&export_format=csv'):
+            with self.subTest(params=params):
+                response = self.client.get(f'{self.url}?{params}')
+                self.assertHttpStatus(response, 200)
+                self.assertEqual(response['Content-Type'], 'text/csv; charset=utf-8')
+                self.assertEqual(response['Content-Disposition'], 'attachment; filename="netbox_sites.csv"')
+                rows = self.get_csv_rows(response)
+                self.assertEqual(rows[0], ['Name', 'Status'])
+                self.assertEqual(rows[1], ['Site 1', 'Active'])
+                self.assertEqual(len(rows), 4)
+
+    def test_export_all_data(self):
+        for params in ('export', 'export&export_format=csv'):
+            with self.subTest(params=params):
+                response = self.client.get(f'{self.url}?{params}')
+                self.assertHttpStatus(response, 200)
+                self.assertEqual(response['Content-Type'], 'text/csv; charset=utf-8')
+                rows = self.get_csv_rows(response)
+                self.assertEqual(rows[0][:2], ['Name', 'Status'])
+                self.assertGreater(len(rows[0]), 2)
+                self.assertEqual(len(rows), 4)
+
+    def test_export_filtered(self):
+        response = self.client.get(f'{self.url}?export=table&export_format=csv&q=Site 2')
+        self.assertEqual(self.get_csv_rows(response)[1:], [['Site 2', 'Active']])
+
+    def test_export_table_ordering(self):
+        """
+        Table exports follow the table's sort order.
+        """
+        for streaming in (False, True):
+            for params in ('export=table', 'export'):
+                with self.subTest(streaming=streaming, params=params), override_settings(STREAMING_EXPORTS=streaming):
+                    response = self.client.get(f'{self.url}?{params}&sort=-name')
+                    rows = list(csv.reader(StringIO(response.getvalue().decode())))
+                    self.assertEqual([row[0] for row in rows[1:]], ['Site 3', 'Site 2', 'Site 1'])
+
+    def test_export_yaml_ordering(self):
+        """
+        YAML exports use the queryset's ordering, not the table's sort order.
+        """
+        manufacturer = Manufacturer.objects.create(name='Manufacturer 1', slug='manufacturer-1')
+        for i in range(1, 4):
+            DeviceType.objects.create(manufacturer=manufacturer, model=f'Device Type {i}', slug=f'device-type-{i}')
+        self.add_permissions('dcim.view_devicetype')
+
+        response = self.client.get(f'{reverse("dcim:devicetype_list")}?export&sort=-model')
+        models = [data['model'] for data in yaml.safe_load_all(response.content)]
+        self.assertEqual(models, ['Device Type 1', 'Device Type 2', 'Device Type 3'])
+
+    def test_export_chunked_only_if_streaming(self):
+        """
+        Table exports fetch records in chunks (using a server-side cursor) only if STREAMING_EXPORTS is enabled.
+        """
+        for streaming in (False, True):
+            with self.subTest(streaming=streaming):
+                with (
+                    override_settings(STREAMING_EXPORTS=streaming),
+                    patch.object(QuerySet, 'iterator', autospec=True, side_effect=QuerySet.iterator) as iterator,
+                ):
+                    response = self.client.get(f'{self.url}?export=table')
+                    content = response.getvalue().decode()
+                self.assertEqual(iterator.called, streaming)
+                self.assertEqual(
+                    content.splitlines(),
+                    ['Name,Status', 'Site 1,Active', 'Site 2,Active', 'Site 3,Active'],
+                )
+
+    def test_export_csv_delimiter(self):
+        self.user.config.set('csv_delimiter', 'semicolon', commit=True)
+        response = self.client.get(f'{self.url}?export=table&export_format=csv')
+        self.assertEqual(response.content.decode().splitlines()[0], 'Name;Status')
+
+    def test_export_invalid_format(self):
+        for params in (
+            'export&export_format=invalid',
+            'export=table&export_format=invalid',
+            'export&export_format=yaml',
+        ):
+            with self.subTest(params=params):
+                self.assertHttpStatus(self.client.get(f'{self.url}?{params}'), 404)
+
+    def test_export_template_with_format(self):
+        """
+        Combining export_format with an export template returns a 400 response.
+        """
+        export_template = ExportTemplate.objects.create(name='Template 1', template_code='{{ queryset|length }}')
+        export_template.object_types.set([ObjectType.objects.get_for_model(Site)])
+        self.add_permissions('extras.view_exporttemplate')
+
+        for params in ('export=Template 1', 'export=Template 1&export_format='):
+            with self.subTest(params=params):
+                self.assertHttpStatus(self.client.get(f'{self.url}?{params}'), 200)
+        for params in ('export=Template 1&export_format=csv', 'export=Template 1&export_format=invalid'):
+            with self.subTest(params=params):
+                self.assertHttpStatus(self.client.get(f'{self.url}?{params}'), 400)
+
+    @skipIf('netbox.tests.dummy_plugin' not in settings.PLUGINS, 'dummy_plugin not in settings.PLUGINS')
+    def test_export_plugin_table_exporter(self):
+        response = self.client.get(f'{self.url}?export=table&export_format=dummy_plugin.tsv')
+        self.assertHttpStatus(response, 200)
+        self.assertNotIsInstance(response, StreamingHttpResponse)
+        self.assertEqual(response['Content-Type'], 'text/tab-separated-values; charset=utf-8')
+        self.assertEqual(response['Content-Disposition'], 'attachment; filename="netbox_sites.tsv"')
+        self.assertEqual(response.content.decode().splitlines()[:2], ['Name\tStatus', 'Site 1\tActive'])
+
+    @skipIf('netbox.tests.dummy_plugin' not in settings.PLUGINS, 'dummy_plugin not in settings.PLUGINS')
+    def test_export_plugin_exporter(self):
+        response = self.client.get(f'{self.url}?export&export_format=dummy_plugin.site_names')
+        self.assertHttpStatus(response, 200)
+        self.assertNotIsInstance(response, StreamingHttpResponse)
+        self.assertEqual(response['Content-Disposition'], 'attachment; filename="netbox_sites.txt"')
+        self.assertEqual(response.content.decode(), 'Site 1\nSite 2\nSite 3\n')
+
+        # Streamed if STREAMING_EXPORTS is enabled
+        with override_settings(STREAMING_EXPORTS=True):
+            response = self.client.get(f'{self.url}?export&export_format=dummy_plugin.site_names')
+        self.assertIsInstance(response, StreamingHttpResponse)
+        self.assertEqual(response.getvalue().decode(), 'Site 1\nSite 2\nSite 3\n')
+
+        # Not available for the current view
+        response = self.client.get(f'{self.url}?export=table&export_format=dummy_plugin.site_names')
+        self.assertHttpStatus(response, 404)
+
+        # Not available for other models
+        self.add_permissions('dcim.view_devicetype')
+        response = self.client.get(f'{reverse("dcim:devicetype_list")}?export&export_format=dummy_plugin.site_names')
+        self.assertHttpStatus(response, 404)
+
+    @skipIf('netbox.tests.dummy_plugin' not in settings.PLUGINS, 'dummy_plugin not in settings.PLUGINS')
+    def test_export_dropdown(self):
+        response = self.client.get(f'{self.url}?status=active')
+        self.assertHttpStatus(response, 200)
+        self.assertContains(response, '?status=active&export=table&export_format=csv"')
+        self.assertContains(response, '?status=active&export=table&export_format=dummy_plugin.tsv"')
+        self.assertContains(response, '?status=active&export&export_format=csv"')
+        self.assertContains(response, '?status=active&export&export_format=dummy_plugin.tsv"')
+        self.assertContains(response, '?status=active&export&export_format=dummy_plugin.site_names"')
+        self.assertNotContains(response, 'export=table&export_format=dummy_plugin.site_names')
+        self.assertNotContains(response, 'export_format=yaml')
+
+    def test_export_response(self):
+        view = SiteListView()
+        view.queryset = Site.objects.all()
+        exporter = get_exporter('csv')
+
+        for streaming in (False, True):
+            for data in ('a,b', b'a,b', bytearray(b'a,b'), memoryview(b'a,b'), iter(['a,', 'b'])):
+                with self.subTest(streaming=streaming, type=type(data).__name__):
+                    with override_settings(STREAMING_EXPORTS=streaming):
+                        response = view.get_export_response(exporter, data)
+                    # Only an iterator is streamed, and only if STREAMING_EXPORTS is enabled
+                    expect_stream = streaming and isinstance(data, Iterator)
+                    self.assertEqual(isinstance(response, StreamingHttpResponse), expect_stream)
+                    self.assertEqual(response.getvalue(), b'a,b')
+
+    def test_export_table_override(self):
+        """
+        Overriding the deprecated export_table() still applies to CSV exports, using the user's delimiter.
+        """
+        with self.assertWarns(DeprecationWarning):
+            class CustomSiteListView(SiteListView):
+                def export_table(self, table, columns=None, filename=None, delimiter=None):
+                    return super().export_table(table, columns, filename='custom.csv', delimiter=delimiter)
+
+        self.user.config.set('csv_delimiter', 'semicolon', commit=True)
+        for params in ({'export': 'table'}, {'export': ''}, {'export': 'table', 'export_format': 'csv'}):
+            with self.subTest(params=params):
+                request = RequestFactory().get(self.url, params)
+                request.user = self.user
+                with self.assertWarns(DeprecationWarning):
+                    response = CustomSiteListView.as_view()(request)
+                self.assertEqual(response['Content-Disposition'], 'attachment; filename="custom.csv"')
+                self.assertTrue(response.getvalue().decode().startswith('Name;Status'))
+
+    def test_export_yaml_override(self):
+        """
+        Overriding the deprecated export_yaml() still applies to YAML exports.
+        """
+        with self.assertWarns(DeprecationWarning):
+            class CustomDeviceTypeListView(DeviceTypeListView):
+                def export_yaml(self):
+                    return 'custom'
+
+        self.add_permissions('dcim.view_devicetype')
+        url = reverse('dcim:devicetype_list')
+        for params in ({'export': ''}, {'export': '', 'export_format': 'yaml'}):
+            with self.subTest(params=params):
+                request = RequestFactory().get(url, params)
+                request.user = self.user
+                response = CustomDeviceTypeListView.as_view()(request)
+                self.assertEqual(response['Content-Type'], 'text/yaml')
+                self.assertEqual(response.content, b'custom')
+
+    def test_export_yaml(self):
+        """
+        The deprecated export_yaml() returns the YAML export of the view's queryset.
+        """
+        manufacturer = Manufacturer.objects.create(name='Manufacturer 1', slug='manufacturer-1')
+        DeviceType.objects.create(manufacturer=manufacturer, model='Device Type 1', slug='device-type-1')
+        view = DeviceTypeListView()
+        view.queryset = DeviceType.objects.all()
+        with self.assertWarns(DeprecationWarning):
+            data = view.export_yaml()
+        self.assertIn('model: Device Type 1', data)

netbox/netbox/tests/test_plugins.py

@@ -25,6 +25,7 @@ from netbox.plugins.utils import get_plugin_config
 from netbox.registry import registry
 from netbox.tests.dummy_plugin import config as dummy_config
 from netbox.tests.dummy_plugin.data_backends import DummyBackend
+from netbox.tests.dummy_plugin.exporters import DummySiteExporter, DummyTableExporter
 from netbox.tests.dummy_plugin.jobs import DummySystemJob
 from netbox.tests.dummy_plugin.models import DummyModel
 from netbox.tests.dummy_plugin.webhook_callbacks import set_context
@@ -156,6 +157,13 @@ class PluginTestCase(TestCase):
         self.assertIn('dummy', registry['data_backends'])
         self.assertIs(registry['data_backends']['dummy'], DummyBackend)
 
+    def test_exporters(self):
+        """
+        Check registered exporters.
+        """
+        self.assertIsInstance(registry['exporters']['dummy_plugin.tsv'], DummyTableExporter)
+        self.assertIsInstance(registry['exporters']['dummy_plugin.site_names'], DummySiteExporter)
+
     def test_system_jobs(self):
         """
         Check registered system jobs.

netbox/netbox/views/generic/bulk_views.py

@@ -14,11 +14,14 @@ from django.db import IntegrityError, router, transaction
 from django.db.models import ManyToManyField, ProtectedError, RestrictedError
 from django.db.models.fields.reverse_related import ManyToManyRel
 from django.forms import ModelMultipleChoiceField, MultipleHiddenInput
-from django.http import HttpResponse
+from django.http import Http404, HttpResponse, HttpResponseBadRequest, StreamingHttpResponse
 from django.shortcuts import get_object_or_404, redirect, render
+from django.utils.encoding import force_str
 from django.utils.http import content_disposition_header
 from django.utils.safestring import mark_safe
 from django.utils.translation import gettext as _
+from django_tables2.data import TableQuerysetData
+from django_tables2.rows import BoundRow
 
 from core.exceptions import JobFailed
 from core.models import ObjectType
@@ -26,12 +29,12 @@ from core.signals import clear_events
 from extras.choices import CustomFieldUIEditableChoices
 from extras.models import CustomField, ExportTemplate
 from netbox.choices import ImportMethodChoices
+from netbox.exporters import get_exporter
 from netbox.forms.bulk_rename import NetBoxModelBulkRenameForm
 from netbox.models.features import ChangeLoggingMixin
 from netbox.object_actions import AddObject, BulkDelete, BulkEdit, BulkExport, BulkImport, BulkRename
 from utilities.error_handlers import handle_protectederror
 from utilities.exceptions import AbortRequest, AbortTransaction, PermissionsViolation
-from utilities.export import TableExport, stream_table_csv_response
 from utilities.forms import BulkDeleteForm, BulkRenameForm, restrict_form_fields
 from utilities.forms.bulk_import import BulkImportForm
 from utilities.htmx import htmx_partial
@@ -58,6 +61,8 @@ __all__ = (
     'ObjectListView',
 )
 
+EXPORT_CHUNK_SIZE = 1000
+
 
 # TODO: Remove in NetBox v5.0.
 # MPTT support is retained only for plugins whose tree models still derive from the
@@ -121,6 +126,64 @@ class ObjectListView(BaseMultiObjectView, ActionsMixin, TableMixin):
     # Export methods
     #
 
+    def get_export_rows(self, table, columns=None):
+        """
+        Return the column headers and an iterator of cell values for each row to be exported from the table.
+
+        Args:
+            table: The Table instance to export
+            columns: A list of specific columns to include. If None, all columns will be exported.
+        """
+        exclude_columns = {'pk', 'actions'}
+        all_columns = [col_name for col_name, _ in table.selected_columns + table.available_columns]
+        if columns:
+            exclude_columns.update({
+                col for col in all_columns if col not in columns
+            })
+
+        # Ensure related objects are prefetched for every column that will be exported, not just
+        # those currently visible in the configured table view.
+        table._apply_prefetching(columns=[c for c in all_columns if c not in exclude_columns])
+
+        export_columns = [
+            column for column in table.columns.iterall()
+            if not (column.column.exclude_from_export or column.name in exclude_columns)
+        ]
+        headers = [force_str(column.header, strings_only=True) for column in export_columns]
+
+        def iter_rows():
+            # Fetch records in chunks only when streaming, as this requires a server-side cursor
+            if settings.STREAMING_EXPORTS and isinstance(table.data, TableQuerysetData):
+                records = table.data.data.iterator(chunk_size=EXPORT_CHUNK_SIZE)
+            else:
+                records = table.data
+            for record in records:
+                row = BoundRow(record, table=table)
+                yield [force_str(row.get_cell_value(column.name), strings_only=True) for column in export_columns]
+
+        return headers, iter_rows()
+
+    def get_export_response(self, exporter, data, filename=None):
+        """
+        Return an HTTP response delivering the exported data as a file attachment.
+
+        Args:
+            exporter: The Exporter instance which produced the data
+            data: The exported data, as str or bytes, or as an iterable of chunks (streamed if STREAMING_EXPORTS is
+                enabled)
+            filename: The name of the file attachment. If None, will be determined automatically from the queryset
+                model name.
+        """
+        filename = filename or f'netbox_{self.queryset.model._meta.verbose_name_plural}.{exporter.file_extension}'
+        if isinstance(data, (bytearray, memoryview)):
+            data = bytes(data)
+        if settings.STREAMING_EXPORTS and not isinstance(data, (str, bytes)):
+            response = StreamingHttpResponse(data, content_type=exporter.content_type)
+        else:
+            response = HttpResponse(data, content_type=exporter.content_type)
+        response['Content-Disposition'] = content_disposition_header(as_attachment=True, filename=filename)
+        return response
+
     # TODO: Remove in NetBox v5.0
     def export_yaml(self):
         """
@@ -131,9 +194,7 @@ class ObjectListView(BaseMultiObjectView, ActionsMixin, TableMixin):
             DeprecationWarning,
             stacklevel=2,
         )
-        yaml_data = [obj.to_yaml() for obj in self.queryset]
-
-        return '---\n'.join(yaml_data)
+        return get_exporter('yaml').export(request=getattr(self, 'request', None), queryset=self.queryset)
 
     # TODO: Remove in NetBox v5.0
     def export_table(self, table, columns=None, filename=None, delimiter=None):
@@ -152,34 +213,9 @@ class ObjectListView(BaseMultiObjectView, ActionsMixin, TableMixin):
             DeprecationWarning,
             stacklevel=2,
         )
-        exclude_columns = {'pk', 'actions'}
-        all_columns = [col_name for col_name, _ in table.selected_columns + table.available_columns]
-        if columns:
-            exclude_columns.update({
-                col for col in all_columns if col not in columns
-            })
-
-        # Ensure related objects are prefetched for every column that will be exported, not just
-        # those currently visible in the configured table view.
-        table._apply_prefetching(columns=[c for c in all_columns if c not in exclude_columns])
-
-        filename = filename or f'netbox_{self.queryset.model._meta.verbose_name_plural}.csv'
-
-        if settings.STREAMING_EXPORTS:
-            return stream_table_csv_response(
-                table=table,
-                exclude_columns=exclude_columns,
-                filename=filename,
-                delimiter=delimiter,
-            )
-
-        exporter = TableExport(
-            export_format=TableExport.CSV,
-            table=table,
-            exclude_columns=exclude_columns,
-            delimiter=delimiter,
-        )
-        return exporter.response(filename=filename)
+        exporter = get_exporter('csv')
+        headers, rows = self.get_export_rows(table, columns)
+        return self.get_export_response(exporter, exporter.write_rows(headers, rows, delimiter), filename)
 
     def export_template(self, template, request):
         """
@@ -234,15 +270,12 @@ class ObjectListView(BaseMultiObjectView, ActionsMixin, TableMixin):
 
         if 'export' in request.GET:
 
-            # Export the current table view
-            if request.GET['export'] == 'table':
-                table = self.get_table(self.queryset, request, has_table_actions)
-                columns = [name for name, _ in table.selected_columns]
-                delimiter = request.user.config.get('csv_delimiter') if request.user.is_authenticated else None
-                return self.export_table(table, columns, delimiter=delimiter)
-
             # Render an ExportTemplate
-            if request.GET['export']:
+            if request.GET['export'] not in ('', 'table'):
+                if request.GET.get('export_format'):
+                    return HttpResponseBadRequest(
+                        _("The export_format parameter cannot be used with an export template.")
+                    )
                 template = get_object_or_404(
                     ExportTemplate.objects.restrict(request.user, 'view'),
                     object_types=object_type,
@@ -250,17 +283,35 @@ class ObjectListView(BaseMultiObjectView, ActionsMixin, TableMixin):
                 )
                 return self.export_template(template, request)
 
-            # Check for YAML export support on the model
-            if hasattr(model, 'to_yaml'):
-                response = HttpResponse(self.export_yaml(), content_type='text/yaml')
-                filename = 'netbox_{}.yaml'.format(self.queryset.model._meta.verbose_name_plural)
-                response['Content-Disposition'] = content_disposition_header(as_attachment=True, filename=filename)
-                return response
+            # Export the current table view or all data, in the requested format (YAML or CSV by default)
+            current_view = request.GET['export'] == 'table'
+            if slug := request.GET.get('export_format'):
+                exporter = get_exporter(slug)
+                if not exporter or not exporter.supports_model(model) or (
+                    current_view and not exporter.tabular
+                ):
+                    raise Http404(_("Invalid export format: {format}").format(format=slug))
+            elif not current_view and get_exporter('yaml').supports_model(model):
+                exporter = get_exporter('yaml')
+            else:
+                exporter = get_exporter('csv')
+            if not exporter.tabular:
+                # TODO: Remove in NetBox v5.0 along with export_yaml()
+                if exporter.slug == 'yaml' and type(self).export_yaml is not ObjectListView.export_yaml:
+                    return self.get_export_response(exporter, self.export_yaml())
+                data = exporter.export(request=request, queryset=self.queryset)
+                return self.get_export_response(exporter, data)
 
-            # Fall back to default table/YAML export
             table = self.get_table(self.queryset, request, has_table_actions)
-            delimiter = request.user.config.get('csv_delimiter') if request.user.is_authenticated else None
-            return self.export_table(table, delimiter=delimiter)
+            columns = [name for name, _ in table.selected_columns] if current_view else None
+
+            # TODO: Remove in NetBox v5.0 along with export_table()
+            if exporter.slug == 'csv' and type(self).export_table is not ObjectListView.export_table:
+                return self.export_table(table, columns, delimiter=exporter.get_delimiter(request))
+
+            headers, rows = self.get_export_rows(table, columns)
+            data = exporter.export(request=request, queryset=self.queryset, headers=headers, rows=rows)
+            return self.get_export_response(exporter, data)
 
         # Render the objects table
         table = self.get_table(self.queryset, request, has_table_actions)

netbox/project-static/src/search.ts

@@ -18,32 +18,30 @@ function quickSearchEventHandler(event: Event): void {
 }
 
 /**
- * Clear the existing search parameters in the link to export Current View.
+ * Update the links to export the current view with the quick search query.
+ *
+ * @param query Quick search query
  */
-function clearLinkParams(): void {
-  const link = document.getElementById('export_current_view') as HTMLLinkElement;
-  const linkUpdated = link?.href.split('&')[0];
-  link.setAttribute('href', linkUpdated);
+function updateExportLinks(query: string): void {
+  for (const link of document.querySelectorAll<HTMLAnchorElement>('a.export-current-view')) {
+    const url = new URL(link.href);
+    if (query) {
+      url.searchParams.set('q', query);
+    } else {
+      url.searchParams.delete('q');
+    }
+    link.href = url.toString();
+  }
 }
 
 /**
- * Update the Export View link to add the Quick Search parameters.
- * @param event
+ * Update the export links to include the quick search query.
+ *
+ * @param event "change" event for the quicksearch input
  */
 function handleQuickSearchParams(event: Event): void {
-  const quickSearchParameters = event.currentTarget as HTMLInputElement;
-
-  // Clear the existing search parameters
-  clearLinkParams();
-
-  if (quickSearchParameters != null) {
-    const link = document.getElementById('export_current_view') as HTMLLinkElement;
-    const params = new URLSearchParams();
-    params.set('q', quickSearchParameters.value);
-    const search_parameter = params.toString();
-    const linkUpdated = link?.href + '&' + search_parameter;
-    link.setAttribute('href', linkUpdated);
-  }
+  const quicksearch = event.currentTarget as HTMLInputElement;
+  updateExportLinks(quicksearch.value);
 }
 
 /**
@@ -71,7 +69,7 @@ export function initQuickSearch(): void {
           quicksearch.value = '';
           await new Promise(f => setTimeout(f, 100));
           quicksearch.dispatchEvent(search);
-          clearLinkParams();
+          updateExportLinks('');
         },
         {
           passive: true,

netbox/utilities/constants.py

@@ -1,3 +1,5 @@
+import re
+
 #
 # Filter lookup expressions
 #
@@ -126,3 +128,10 @@ IMAGE_URL_SCHEMES = {'http', 'https'}
 HTTP_PROXY_SUPPORTED_SOCK_SCHEMAS = ['socks4', 'socks4a', 'socks4h', 'socks5', 'socks5a', 'socks5h']
 HTTP_PROXY_SOCK_RDNS_SCHEMAS = ['socks4h', 'socks4a', 'socks5h', 'socks5a']
 HTTP_PROXY_SUPPORTED_SCHEMAS = ['http', 'https', 'socks4', 'socks4a', 'socks4h', 'socks5', 'socks5a', 'socks5h']
+
+
+#
+# Registered component slugs (e.g. event rule actions & exporters)
+#
+
+SLUG_RE = re.compile(r'^[a-z][a-z0-9_]*(\.[a-z0-9_]+)*$')

netbox/utilities/export.py

@@ -1,4 +1,6 @@
+# TODO: Remove this module in NetBox v5.0
 import csv
+import warnings
 
 from django.http import StreamingHttpResponse
 from django.utils.encoding import force_str
@@ -15,6 +17,13 @@ __all__ = (
     'stream_table_csv_response',
 )
 
+warnings.warn(
+    "utilities.export is deprecated and will be removed in NetBox v5.0. Register an exporter using "
+    "netbox.exporters.register_exporter() instead.",
+    DeprecationWarning,
+    stacklevel=2,
+)
+
 EXPORT_CHUNK_SIZE = 1000
 
 

netbox/utilities/templates/buttons/export.html

@@ -4,8 +4,18 @@
     <i class="mdi mdi-download" aria-hidden="true"></i> {{ label }}
   </button>
   <ul class="dropdown-menu dropdown-menu-end">
-    <li><a id="export_current_view" class="dropdown-item" href="?{% if url_params %}{{ url_params }}&{% endif %}export=table">{% trans "Current View" %}</a></li>
-    <li><a class="dropdown-item" href="?{% if url_params %}{{ url_params }}&{% endif %}export">{% trans "All Data" %} ({{ data_format }})</a></li>
+    {% if table_exporters %}
+      <li><h6 class="dropdown-header">{% trans "Current View" %}</h6></li>
+      {% for exporter in table_exporters %}
+        <li><a class="dropdown-item export-current-view" href="?{% if url_params %}{{ url_params }}&{% endif %}export=table&export_format={{ exporter.slug }}">{{ exporter.label }}</a></li>
+      {% endfor %}
+    {% endif %}
+    {% if exporters %}
+      <li><h6 class="dropdown-header">{% trans "All Data" %}</h6></li>
+      {% for exporter in exporters %}
+        <li><a class="dropdown-item" href="?{% if url_params %}{{ url_params }}&{% endif %}export&export_format={{ exporter.slug }}">{{ exporter.label }}</a></li>
+      {% endfor %}
+    {% endif %}
     {% if export_templates %}
       <li>
         <hr class="dropdown-divider">

netbox/utilities/templatetags/buttons.py

@@ -4,8 +4,7 @@ from django.urls import NoReverseMatch, reverse
 from django.utils.safestring import mark_safe
 from django.utils.translation import gettext as _
 
-from core.models import ObjectType
-from extras.models import Bookmark, ExportTemplate, Subscription
+from extras.models import Bookmark, Subscription
 from netbox.models.features import NotificationsMixin
 from utilities.querydict import prepare_cloned_fields
 from utilities.views import get_action_url
@@ -19,7 +18,6 @@ __all__ = (
     'clone_button',
     'delete_button',
     'edit_button',
-    'export_button',
     'import_button',
     'subscribe_button',
     'sync_button',
@@ -191,28 +189,6 @@ def import_button(model, action='bulk_import'):
     }
 
 
-# TODO: Remove in NetBox v4.7
-@register.inclusion_tag('buttons/export.html', takes_context=True)
-def export_button(context, model):
-    object_type = ObjectType.objects.get_for_model(model)
-    user = context['request'].user
-
-    # Determine if the "all data" export returns CSV or YAML
-    data_format = 'YAML' if hasattr(object_type.model_class(), 'to_yaml') else 'CSV'
-
-    # Retrieve all export templates for this model
-    export_templates = ExportTemplate.objects.restrict(user, 'view').filter(object_types=object_type)
-
-    return {
-        'label': _('Export'),
-        'perms': context['perms'],
-        'object_type': object_type,
-        'url_params': context['request'].GET.urlencode() if context['request'].GET else '',
-        'export_templates': export_templates,
-        'data_format': data_format,
-    }
-
-
 # TODO: Remove in NetBox v4.7
 @register.inclusion_tag('buttons/bulk_edit.html', takes_context=True)
 def bulk_edit_button(context, model, action='bulk_edit', query_params=None):

netbox/utilities/testing/views.py

@@ -11,6 +11,7 @@ from django.utils.translation import gettext as _
 from core.choices import ObjectChangeActionChoices
 from core.models import ObjectChange, ObjectType
 from netbox.choices import CSVDelimiterChoices, ImportFormatChoices
+from netbox.exporters import get_exporters
 from netbox.models.features import ChangeLoggingMixin, CustomFieldsMixin
 from users.models import ObjectPermission
 
@@ -517,6 +518,17 @@ class ViewTestCases:
             self.assertHttpStatus(response, 200)
             self.assertEqual(response.get('Content-Type'), 'text/csv; charset=utf-8')
 
+            # Test each supported exporter for the current view (table exporters only) and for all data
+            for exporter in get_exporters(self.model):
+                scopes = ('export=table', 'export') if exporter.tabular else ('export',)
+                for scope in scopes:
+                    with self.subTest(exporter=exporter.slug, scope=scope):
+                        response = self.client.get(f'{url}?{scope}&export_format={exporter.slug}')
+                        self.assertHttpStatus(response, 200)
+                        self.assertEqual(response.get('Content-Type'), exporter.content_type)
+                        self.assertIn(f'.{exporter.file_extension}', response.get('Content-Disposition'))
+                        self.assertTrue(response.getvalue())
+
         @override_settings(EXEMPT_VIEW_PERMISSIONS=['*'], LOGIN_REQUIRED=False)
         def test_export_objects_anonymous(self):
             # Ensure we are logged out.

netbox/utilities/tests/test_export.py

@@ -0,0 +1,41 @@
+import importlib
+import sys
+
+from django.test import TestCase
+
+from dcim.models import Site
+from dcim.tables import SiteTable
+
+
+# TODO: Remove in NetBox v5.0 along with utilities.export
+class DeprecatedExportModuleTestCase(TestCase):
+
+    @classmethod
+    def setUpTestData(cls):
+        Site.objects.create(name='Site 1', slug='site-1')
+
+    def setUp(self):
+        # Re-import the module so that its deprecation warning is raised again
+        sys.modules.pop('utilities.export', None)
+        with self.assertWarns(DeprecationWarning):
+            self.export = importlib.import_module('utilities.export')
+        self.table = SiteTable(Site.objects.all())
+
+    def test_table_export(self):
+        exporter = self.export.TableExport(
+            export_format=self.export.TableExport.CSV,
+            table=self.table,
+            exclude_columns={'pk', 'actions'},
+            delimiter='semicolon',
+        )
+        response = exporter.response(filename='sites.csv')
+        self.assertIn('Site 1;', response.content.decode())
+
+    def test_stream_table_csv_response(self):
+        response = self.export.stream_table_csv_response(
+            table=self.table,
+            exclude_columns={'pk', 'actions'},
+            filename='sites.csv',
+        )
+        self.assertEqual(response['Content-Disposition'], 'attachment; filename="sites.csv"')
+        self.assertIn('Site 1,', response.getvalue().decode())