Closes #22942: Upgrade to Django 6.1 (#22943)

base_requirements.txt

@@ -4,7 +4,7 @@ colorama
 
 # The Python web framework on which NetBox is built
 # https://docs.djangoproject.com/en/stable/releases/
-Django==6.0.*
+Django==6.1.*
 
 # Django middleware which permits cross-domain API requests
 # https://github.com/adamchainz/django-cors-headers/blob/main/CHANGELOG.rst
@@ -74,7 +74,7 @@ django-timezone-field
 # A REST API framework for Django projects
 # https://www.django-rest-framework.org/community/release-notes/
 # TODO: Re-evaluate the monkey-patch of get_unique_validators() before upgrading
-djangorestframework==3.17.1
+djangorestframework==3.18.0
 
 # Sane and flexible OpenAPI 3 schema generation for Django REST framework.
 # https://github.com/tfranzel/drf-spectacular/blob/master/CHANGELOG.rst

docs/configuration/system.md

@@ -54,7 +54,7 @@ The filesystem path to NetBox's documentation. This is used when presenting cont
 
 In order to send email, NetBox needs an email server configured. The following items can be defined within the `EMAIL` configuration parameter:
 
-* `SERVER` - Hostname or IP address of the email server (use `localhost` if running locally)
+* `SERVER` - Hostname or IP address of the email server (required; use `localhost` if running locally)
 * `PORT` - TCP port to use for the connection (default: `25`)
 * `USERNAME` - Username with which to authenticate
 * `PASSWORD` - Password with which to authenticate
@@ -68,6 +68,9 @@ In order to send email, NetBox needs an email server configured. The following i
 !!! note
     The `USE_SSL` and `USE_TLS` parameters are mutually exclusive.
 
+!!! warning
+    `SERVER` must be defined in order to send email: A deployment which omits it raises an `InvalidMailer` exception when attempting to send. Note that this is raised at send time rather than at startup, so a misconfiguration here will not be apparent until NetBox first tries to send mail.
+
 Email is sent from NetBox only for critical events or if configured for [logging](#logging). If you would like to test the email server configuration, Django provides a convenient [send_mail()](https://docs.djangoproject.com/en/stable/topics/email/#send-mail) function accessible within the NetBox shell:
 
 ```no-highlight
@@ -77,8 +80,7 @@ Email is sent from NetBox only for critical events or if configured for [logging
   'Test Email Subject',
   'Test Email Body',
   'noreply-netbox@example.com',
-  ['users@example.com'],
-  fail_silently=False
+  ['users@example.com']
 )
 ```
 

docs/release-notes/version-4.7.md

@@ -4,7 +4,7 @@
 
 ### Breaking Changes
 
-* PostgreSQL 14 is no longer supported. NetBox now requires PostgreSQL 15 or later.
+* PostgreSQL 14 is no longer supported. NetBox now requires PostgreSQL 15 or later: The upgrade script will abort when connected to an earlier release. (NetBox v4.6 reported this as a warning.)
 * Redis 5.x is no longer supported. NetBox now requires Redis 6.0 or later.
 * Selection and multiple selection custom field values are now returned as objects specifying both the raw value and its human-friendly label (e.g. `{"value": "datacenter", "label": "Data Center"}`) in both the REST and GraphQL APIs. These fields continue to accept the raw value on write.
 * The `protocol` and `ports` fields on the ApplicationService and ApplicationServiceTemplate models have been replaced by a unified `port_mappings` field, which supports multiple protocols per service. The legacy fields are retained (as deprecated) in the REST and GraphQL APIs, but at the ORM level they are now read-only properties derived from `port_mappings`: Passing `protocol` or `ports` to the model raises a `TypeError`, and assigning to `service.ports` raises an `AttributeError`.
@@ -20,6 +20,7 @@
 * The `request` object passed to custom link templates is now a sanitized subset of the current request. Only the `id`, `path`, `path_info`, `method`, `GET`, and `user` attributes are available; cookies, headers, and session state are no longer accessible.
 * URL custom field values are now validated against the [`ALLOWED_URL_SCHEMES`](../configuration/security.md#allowed_url_schemes) configuration parameter. A value entered without a scheme is assumed to use `https` and stored as an absolute URL.
 * Webhooks now support a configurable timeout. If you have lowered `RQ_DEFAULT_TIMEOUT` to 60 seconds or less, you must also set [`WEBHOOK_DEFAULT_TIMEOUT`](../configuration/miscellaneous.md#webhook_default_timeout) to a lower value; NetBox will refuse to start otherwise.
+* Specifying an email server under the [`EMAIL`](../configuration/system.md#email) configuration parameter is now mandatory in order to send mail: A deployment which does not define `EMAIL['SERVER']` will raise an `InvalidMailer` exception when attempting to send, rather than failing at the SMTP connection.
 * The upgrade script now runs the `rebuild_config_context_cache` management command to populate the new config context cache. This may extend the duration of the upgrade for deployments with a large number of devices and virtual machines.
 * Removal of deprecated behavior
     * The `housekeeping` management command has been removed. (Its constituent tasks are performed by the individual management commands introduced in NetBox v4.6.)
@@ -140,6 +141,7 @@ Event rule conditions can now inspect the pre-change and post-change snapshots c
 * [#22571](https://github.com/netbox-community/netbox/issues/22571) - Migrate from django-pglocks to django-pgware
 * [#22615](https://github.com/netbox-community/netbox/issues/22615) - Remove the legacy `request_id` and `username` keys from the webhook context
 * [#22909](https://github.com/netbox-community/netbox/issues/22909) - Tolerate an undefined column when flushing deferred search cache updates
+* [#22942](https://github.com/netbox-community/netbox/issues/22942) - Upgrade to Django 6.1
 
 ### REST API Changes
 

netbox/core/apps.py

@@ -29,7 +29,7 @@ class CoreConfig(AppConfig):
 
     def ready(self):
         from core.api import schema  # noqa: F401
-        from core.checks import check_duplicate_indexes, check_postgresql_version, check_redis_version  # noqa: F401
+        from core.checks import check_duplicate_indexes, check_redis_version  # noqa: F401
         from netbox import context_managers  # noqa: F401
         from netbox.models.features import register_models
         from netbox.search import signals as search_signals  # noqa: F401

netbox/core/checks.py

@@ -1,12 +1,10 @@
 from django.apps import apps
 from django.core.cache import cache
-from django.core.checks import Error, Tags, Warning, register
-from django.db import connection
+from django.core.checks import Error, Tags, register
 from django.db.models import Index, UniqueConstraint
 
 __all__ = (
     'check_duplicate_indexes',
-    'check_postgresql_version',
     'check_redis_version',
 )
 
@@ -45,32 +43,6 @@ def check_duplicate_indexes(app_configs, **kwargs):
     return errors
 
 
-@register(Tags.database)
-def check_postgresql_version(app_configs, **kwargs):
-    """
-    Warn if the PostgreSQL version is less than 15, as support for PostgreSQL 14
-    will be removed in NetBox v4.7.
-    """
-    warnings = []
-    try:
-        with connection.cursor() as cursor:
-            cursor.execute('SHOW server_version_num')
-            row = cursor.fetchone()
-            pg_version = int(row[0])
-        if pg_version < 150000:
-            major_version = pg_version // 10000
-            warnings.append(
-                Warning(
-                    f'Support for PostgreSQL {major_version} is deprecated and will be removed in NetBox v4.7.',
-                    hint='Please upgrade to PostgreSQL 15 or later.',
-                    id='netbox.W001',
-                )
-            )
-    except Exception:
-        pass
-    return warnings
-
-
 @register(Tags.caches)
 def check_redis_version(app_configs, **kwargs):
     """

netbox/extras/querysets.py

@@ -74,7 +74,7 @@ class ConfigContextQuerySet(RestrictedQuerySet):
 
         if aggregate_data:
             return queryset.aggregate(
-                config_context_data=JSONBAgg('data', ordering=['weight', 'name'])
+                config_context_data=JSONBAgg('data', order_by=['weight', 'name'])
             )['config_context_data']
 
         return queryset

netbox/ipam/api/views.py

@@ -22,7 +22,7 @@ from netbox.api.viewsets import NetBoxModelViewSet
 from netbox.api.viewsets.mixins import ObjectValidationMixin, discard_events_on_rollback
 from netbox.config import get_config
 from netbox.constants import ADVISORY_LOCK_KEYS
-from utilities.api import get_serializer_for_model
+from utilities.api import get_positional_errors, get_serializer_for_model
 from virtualization.models import VMInterface
 
 from . import serializers
@@ -265,8 +265,10 @@ class AvailableObjectsView(ObjectValidationMixin, APIView):
             **self.get_extra_context(parent),
         })
         if not serializer.is_valid():
+            # Report the errors by the position of each entry in the request, as the serializer is
+            # always bound to a list (a single object having been wrapped in one above)
             return Response(
-                serializer.errors,
+                get_positional_errors(serializer.errors, len(requested_objects)),
                 status=status.HTTP_400_BAD_REQUEST
             )
 
@@ -292,7 +294,12 @@ class AvailableObjectsView(ObjectValidationMixin, APIView):
                 serializer = serializer_class(data=requested_objects[0], context=context)
 
             if not serializer.is_valid():
-                return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)
+                # A list request is reported by position; a single object carries no position, and
+                # its errors pass through unchanged
+                return Response(
+                    get_positional_errors(serializer.errors, len(requested_objects)),
+                    status=status.HTTP_400_BAD_REQUEST
+                )
 
             # Create the new IP address(es)
             using = router.db_for_write(self.queryset.model)

netbox/ipam/tests/test_api.py

@@ -652,6 +652,44 @@ class PrefixTestCase(APIViewTestCases.APIViewTestCase):
         response = self.client.post(url, data, format='json', **self.header)
         self.assertHttpStatus(response, status.HTTP_400_BAD_REQUEST)
 
+    @tag('regression')
+    def test_create_available_ips_errors_by_position(self):
+        """
+        Test that the errors for a request creating multiple IP addresses are correlated to the
+        positions of the entries which failed validation.
+        """
+        prefix = Prefix.objects.create(prefix=IPNetwork('192.0.2.0/24'))
+        url = reverse('ipam-api:prefix-available-ips', kwargs={'pk': prefix.pk})
+        self.add_permissions('ipam.view_prefix', 'ipam.add_ipaddress')
+
+        # An invalid request attribute, rejected before any address has been allocated
+        data = [
+            {'description': 'Test IP 1'},
+            {'prefix_length': 23},  # Parent prefix is a /24
+        ]
+        response = self.client.post(url, data, format='json', **self.header)
+        self.assertHttpStatus(response, status.HTTP_400_BAD_REQUEST)
+        self.assertEqual(len(response.data), 2)
+        self.assertEqual(response.data[0], {})
+        self.assertIn('prefix_length', response.data[1])
+
+        # An invalid object attribute, rejected after the addresses have been allocated
+        data = [
+            {'description': 'Test IP 1'},
+            {'status': 'not-a-valid-status'},
+        ]
+        response = self.client.post(url, data, format='json', **self.header)
+        self.assertHttpStatus(response, status.HTTP_400_BAD_REQUEST)
+        self.assertEqual(len(response.data), 2)
+        self.assertEqual(response.data[0], {})
+        self.assertIn('status', response.data[1])
+
+        # A single object is wrapped in a list, so its errors are reported in the same form
+        response = self.client.post(url, {'prefix_length': 23}, format='json', **self.header)
+        self.assertHttpStatus(response, status.HTTP_400_BAD_REQUEST)
+        self.assertEqual(len(response.data), 1)
+        self.assertIn('prefix_length', response.data[0])
+
     @tag('regression')
     def test_graphql_tenant_prefixes_contains_nested_skips_invalid(self):
         """

netbox/netbox/api/viewsets/mixins.py

@@ -109,7 +109,7 @@ def _as_field_errors(item_errors):
     return {api_settings.NON_FIELD_ERRORS_KEY: item_errors}
 
 
-def get_invalid_entries_response(entry_errors):
+def get_invalid_entries_response(entry_errors, total):
     """
     Return a structured error Response for the entries of a bulk request which could not be
     interpreted, or None if every entry was interpretable.
@@ -122,13 +122,20 @@ def get_invalid_entries_response(entry_errors):
 
     Passing this stage is what allows every later error to be correlated by `id` instead.
 
-    :param entry_errors: The `errors` of a BulkOperationSerializer bound to a list, which holds one
-        entry per object in the request (an empty dict where that object was interpretable).
+    :param entry_errors: The `errors` of a BulkOperationSerializer bound to a list. These are
+        reported as a mapping of the position of each uninterpretable entry in the request to that
+        entry's errors; the entries which were interpretable are omitted.
+    :param total: The number of entries in the request, for the summary message.
     """
+    # Ignore any error not correlated to a position, as it does not pertain to a single entry
+    indexed_errors = {
+        index: item_errors
+        for index, item_errors in entry_errors.items()
+        if isinstance(index, int)
+    }
     errors = [
-        {'index': i, 'errors': _as_field_errors(item_errors)}
-        for i, item_errors in enumerate(entry_errors)
-        if item_errors
+        {'index': index, 'errors': _as_field_errors(item_errors)}
+        for index, item_errors in sorted(indexed_errors.items())
     ]
     if not errors:
         return None
@@ -137,7 +144,7 @@ def get_invalid_entries_response(entry_errors):
         {
             'detail': _('{failed_count} of {total} objects failed validation.').format(
                 failed_count=len(errors),
-                total=len(entry_errors),
+                total=total,
             ),
             'errors': errors,
         },
@@ -528,7 +535,7 @@ class BulkUpdateModelMixin:
         # a malformed entry is reported in the same form as every other bulk error
         serializer = BulkOperationSerializer(data=request.data, many=True)
         if not serializer.is_valid():
-            return get_invalid_entries_response(serializer.errors)
+            return get_invalid_entries_response(serializer.errors, len(request.data))
 
         object_ids = [o['id'] for o in serializer.validated_data]
 
@@ -673,7 +680,7 @@ class BulkDestroyModelMixin:
         # a malformed entry is reported in the same form as every other bulk error
         serializer = BulkOperationSerializer(data=request.data, many=True)
         if not serializer.is_valid():
-            return get_invalid_entries_response(serializer.errors)
+            return get_invalid_entries_response(serializer.errors, len(request.data))
 
         object_ids = [o['id'] for o in serializer.validated_data]
 

netbox/netbox/models/ltree.py

@@ -304,8 +304,8 @@ class LtreeModel(models.Model, metaclass=LtreeModelBase):
         self._loaded_name = self.__dict__.get('name')
 
     @classmethod
-    def from_db(cls, db, field_names, values):
-        instance = super().from_db(db, field_names, values)
+    def from_db(cls, db, field_names, values, **kwargs):
+        instance = super().from_db(db, field_names, values, **kwargs)
         instance._loaded_parent_id = instance.__dict__.get('parent_id')
         instance._loaded_name = instance.__dict__.get('name')
         return instance

netbox/netbox/settings.py

@@ -464,16 +464,23 @@ if SESSION_FILE_PATH is not None:
 # Email
 #
 
-EMAIL_HOST = EMAIL.get('SERVER')
-EMAIL_HOST_USER = EMAIL.get('USERNAME')
-EMAIL_HOST_PASSWORD = EMAIL.get('PASSWORD')
-EMAIL_PORT = EMAIL.get('PORT', 25)
-EMAIL_SSL_CERTFILE = EMAIL.get('SSL_CERTFILE')
-EMAIL_SSL_KEYFILE = EMAIL.get('SSL_KEYFILE')
+MAILERS = {
+    'default': {
+        'BACKEND': 'django.core.mail.backends.smtp.EmailBackend',
+        'OPTIONS': {
+            'host': EMAIL.get('SERVER'),
+            'port': EMAIL.get('PORT', 25),
+            'username': EMAIL.get('USERNAME'),
+            'password': EMAIL.get('PASSWORD'),
+            'use_ssl': EMAIL.get('USE_SSL', False),
+            'use_tls': EMAIL.get('USE_TLS', False),
+            'ssl_certfile': EMAIL.get('SSL_CERTFILE'),
+            'ssl_keyfile': EMAIL.get('SSL_KEYFILE'),
+            'timeout': EMAIL.get('TIMEOUT', 10),
+        },
+    },
+}
 EMAIL_SUBJECT_PREFIX = '[NetBox] '
-EMAIL_USE_SSL = EMAIL.get('USE_SSL', False)
-EMAIL_USE_TLS = EMAIL.get('USE_TLS', False)
-EMAIL_TIMEOUT = EMAIL.get('TIMEOUT', 10)
 SERVER_EMAIL = EMAIL.get('FROM_EMAIL')
 
 

netbox/utilities/api.py

@@ -30,6 +30,7 @@ __all__ = (
     'IsSuperuser',
     'get_annotations_for_serializer',
     'get_graphql_type_for_model',
+    'get_positional_errors',
     'get_prefetches_for_serializer',
     'get_related_object_by_attrs',
     'get_serializer_for_model',
@@ -127,6 +128,29 @@ def get_view_name(view):
     return drf_get_view_name(view)
 
 
+def get_positional_errors(errors, count):
+    """
+    Return the errors reported by a serializer bound to a list of `count` entries as a list
+    correlated to the positions of those entries, with an empty dict standing in for each entry
+    which validated.
+
+    DRF 3.18 reports the errors of a ListSerializer as a mapping of the index of each failed entry
+    to that entry's errors, omitting the entries which passed; earlier releases reported a list
+    aligned with the request body. Restoring the positional form keeps the response shape stable for
+    API consumers which index into it.
+
+    Errors which pertain to the list as a whole rather than to any one entry (e.g. a body which is
+    not a list at all) carry no position, and are returned unchanged.
+
+    :param errors: The `errors` of a serializer instantiated with many=True.
+    :param count: The number of entries the serializer was bound to.
+    """
+    if not isinstance(errors, dict) or not any(isinstance(index, int) for index in errors):
+        return errors
+
+    return [errors.get(index, {}) for index in range(count)]
+
+
 def _get_nested_serializer(serializer_field):
     """
     Return the nested serializer instance for a declared serializer field.

netbox/utilities/fields.py

@@ -1,6 +1,6 @@
 from collections import defaultdict
 
-from django.contrib.contenttypes.fields import GenericForeignKey
+from django.contrib.contenttypes.fields import GenericForeignKey, GenericForeignKeyDescriptor
 from django.contrib.contenttypes.models import ContentType
 from django.core.exceptions import ObjectDoesNotExist
 from django.db import models
@@ -68,9 +68,25 @@ class NaturalOrderingField(models.CharField):
         )
 
 
+class RestrictedGenericForeignKeyDescriptor(GenericForeignKeyDescriptor):
+    """
+    Django 6.1 moved get_prefetch_querysets() off GenericForeignKey and onto a separate
+    descriptor, which prefetch_related() now prefers over the field itself. Delegate back to
+    the field so that RestrictedGenericForeignKey's restrict()-aware implementation is used.
+    """
+    def get_prefetch_querysets(self, instances, querysets=None):
+        return self.field.get_prefetch_querysets(instances, querysets)
+
+
 class RestrictedGenericForeignKey(GenericForeignKey):
 
-    # Replicated largely from GenericForeignKey. Changes include:
+    def contribute_to_class(self, cls, name, **kwargs):
+        super().contribute_to_class(cls, name, **kwargs)
+        # Replace the descriptor installed by GenericForeignKey with one which defers to
+        # get_prefetch_querysets() below.
+        setattr(cls, self.attname, RestrictedGenericForeignKeyDescriptor(self))
+
+    # Replicated largely from GenericForeignKeyDescriptor. Changes include:
     #  1. Capture restrict_params from RestrictedPrefetch (hack)
     #  2. If restrict_params is set, call restrict() on the queryset for
     #     the related model
@@ -116,14 +132,18 @@ class RestrictedGenericForeignKey(GenericForeignKey):
         for ct_id, fkeys in fk_dict.items():
             if ct_id in custom_queryset_dict:
                 # Return values from the custom queryset, if provided.
-                ret_val.extend(custom_queryset_dict[ct_id].filter(pk__in=fkeys))
+                qs = custom_queryset_dict[ct_id].filter(pk__in=fkeys)
             else:
                 instance = instance_dict[ct_id]
                 ct = self.get_content_type(id=ct_id, using=instance._state.db)
                 qs = ct.model_class().objects.filter(pk__in=fkeys)
                 if restrict_params:
                     qs = qs.restrict(**restrict_params)
-                ret_val.extend(qs)
+            # Carry the fetch mode of the objects being prefetched over to the objects prefetched
+            # onto them. Every instance in a batch shares one fetch mode, so the first is
+            # representative; it is safe to index because fk_dict is populated from `instances`,
+            # and so is empty (skipping this loop entirely) whenever `instances` is.
+            ret_val.extend(qs.fetch_mode(instances[0]._state.fetch_mode))
 
         # For doing the join in Python, we have to match both the FK val and the
         # content type, so we use a callable that returns a (fk, class) pair.

netbox/utilities/testing/base.py

@@ -47,7 +47,7 @@ class TestCase(_TestCase):
         Context manager that wraps subTest with automatic cleanup.
         All database changes within the context will be rolled back.
         """
-        sid = transaction.savepoint()
+        sid = transaction.savepoint_create()
 
         try:
             with self.subTest(**params):

netbox/utilities/tests/test_prefetch.py

@@ -1,5 +1,10 @@
+from django.db.models import FETCH_ONE, FETCH_RAISE
+
 from circuits.models import Circuit, Provider
+from core.models import ObjectType
+from extras.models import CachedValue
 from utilities.prefetch import get_prefetchable_fields
+from utilities.querysets import RestrictedPrefetch
 from utilities.testing.base import TestCase
 
 
@@ -15,3 +20,45 @@ class GetPrefetchableFieldsTestCase(TestCase):
 
         field_names = get_prefetchable_fields(Circuit)
         self.assertIn('group_assignments', field_names)  # Generic relation
+
+
+class RestrictedGenericForeignKeyTestCase(TestCase):
+    """
+    Verify the prefetching behavior of RestrictedGenericForeignKey.
+    """
+    user_permissions = ('circuits.view_provider',)
+
+    def setUp(self):
+        super().setUp()
+
+        self.provider = Provider.objects.create(name='Provider 1', slug='provider-1')
+        CachedValue.objects.create(
+            object_type=ObjectType.objects.get_for_model(Provider),
+            object_id=self.provider.pk,
+            field='name',
+            type='string',
+            value=self.provider.name,
+        )
+
+    def _prefetch_object(self, queryset):
+        cached_value = list(queryset.prefetch_related(RestrictedPrefetch('object', self.user, 'view')))[0]
+        return cached_value.object
+
+    def test_prefetch_propagates_fetch_mode(self):
+        """
+        The fetch mode of the objects being prefetched is carried over to the objects prefetched
+        onto them, as Django's GenericForeignKeyDescriptor does.
+        """
+        obj = self._prefetch_object(CachedValue.objects.fetch_mode(FETCH_RAISE))
+
+        self.assertEqual(obj, self.provider)
+        self.assertIs(obj._state.fetch_mode, FETCH_RAISE)
+
+    def test_prefetch_default_fetch_mode(self):
+        """
+        A queryset which sets no fetch mode yields prefetched objects using the default mode.
+        """
+        obj = self._prefetch_object(CachedValue.objects.all())
+
+        self.assertEqual(obj, self.provider)
+        self.assertIs(obj._state.fetch_mode, FETCH_ONE)

requirements.txt

@@ -1,5 +1,5 @@
 colorama==0.4.6
-Django==6.0.7
+Django==6.1
 django-cors-headers==4.9.0
 django-debug-toolbar==7.0.0
 django-filter==26.1
@@ -15,7 +15,7 @@ django-storages==1.14.6
 django-tables2==3.0.0
 django-taggit==6.1.0
 django-timezone-field==7.2.2
-djangorestframework==3.17.1
+djangorestframework==3.18.0
 drf-spectacular==0.30.0
 drf-spectacular-sidecar==2026.7.1
 feedparser==6.0.14