소스 검색

Add a rebuild_ltree_paths management command

Reinstalling the cascade triggers corrects every subsequent write, but it does
not repair path and sort_path values which went stale while those triggers were
missing. Repairing them meant calling populate_paths_sql(), which returns a SQL
string for use in a migration rather than something an operator can run, or
rebuild_sort_paths(), which covers only half the problem: a stale path misplaces
an object in the hierarchy, while a stale sort_path only misorders a list.

The command wraps populate_paths_sql() for every core hierarchical model, or for
those named on the command line. It deliberately does nothing else. Detection
lives in the v4.7.1 release notes, as queries an operator can run against a
replica without installing anything, which is both a better home for it and
avoids re-deriving the path label width and sort_path separator that
mptt_to_ltree already owns.

Plugin models are excluded. populate_paths_sql() reads the name column by name
while InstallLtreeTriggers accepts any column, so a plugin maintaining sort_path
from another column cannot be rebuilt correctly here and needs its own repair
path.

Relates to #23130.
Jason Novinger 9 시간 전
부모
커밋
eb57625a91

+ 13 - 0
docs/administration/management-commands.md

@@ -44,6 +44,19 @@ By default, only those objects whose cache is empty are rendered, so the command
 python3 netbox/manage.py rebuild_config_context_cache [--force]
 ```
 
+## rebuild_ltree_paths
+
+Recompute the `path` and `sort_path` columns of the hierarchical models (regions, site groups, locations, device roles, platforms, tenant groups, contact groups, wireless LAN groups, module bays, inventory items, and inventory item templates) from their parent relationships. These columns are maintained by PostgreSQL triggers, so this is needed only where a write bypassed them: a bulk `COPY`, a direct `UPDATE`, or a database restored from a NetBox v4.7.0 dump (see [#23130](https://github.com/netbox-community/netbox/issues/23130)).
+
+Pass one or more models as `app_label.ModelName` to limit the rebuild.
+
+```
+python3 netbox/manage.py rebuild_ltree_paths [app_label.ModelName ...]
+```
+
+!!! warning
+    A rebuild rewrites every row of each named table in a single statement, locking those rows until it commits. On a large table this blocks concurrent writes for minutes, so run it during a maintenance window. The detection queries in the [v4.7.1 release notes](../release-notes/version-4.7.md) take no locks, and can be used first to find which tables need it.
+
 ## rebuild_prefixes
 
 Rebuild the IPAM prefix hierarchy, recalculating the depth and child counts for all prefixes.

+ 7 - 1
docs/release-notes/version-4.7.md

@@ -21,7 +21,13 @@
     WHERE c.sort_path <> p.sort_path || chr(9) || c.name;
     ```
 
-    Stale `sort_path` values affect only the order in which objects are listed. A stale `path`, by contrast, misplaces an object within the hierarchy, so it can be omitted from its ancestor's list of descendants.
+    Stale `sort_path` values affect only the order in which objects are listed. A stale `path`, by contrast, misplaces an object within the hierarchy, so it can be omitted from its ancestor's list of descendants. Repair an affected table with the [`rebuild_ltree_paths`](../administration/management-commands.md#rebuild_ltree_paths) management command, naming the models the queries above flagged:
+
+    ```no-highlight
+    python netbox/manage.py rebuild_ltree_paths dcim.region
+    ```
+
+    A rebuild rewrites every row of the named tables, locking those rows until it commits, so run it during a maintenance window.
 
     Plugins which maintain their own `ltree` models via the `InstallLtreeTriggers` migration operation are affected in the same way, and their tables are not touched by the migrations above. Where such a database was restored from a dump, the plugin's cascade triggers are missing entirely; where it was upgraded in place, they carry the old definition and will be lost by its next dump. Either way, applying `InstallLtreeTriggers` again from a new plugin migration reinstalls them: as of this release the operation drops each trigger before recreating it, so it is safe to re-run.
 

+ 66 - 0
netbox/utilities/management/commands/rebuild_ltree_paths.py

@@ -0,0 +1,66 @@
+from django.apps import apps
+from django.core.management.base import BaseCommand, CommandError
+from django.db import connection, transaction
+
+from netbox.models.ltree import LtreeModel
+from netbox.plugins import PluginConfig
+from utilities.mptt_to_ltree import populate_paths_sql
+
+
+class Command(BaseCommand):
+    help = (
+        "Recompute the trigger-maintained path (and sort_path) columns of hierarchical models "
+        "from their parent relationships"
+    )
+
+    def add_arguments(self, parser):
+        parser.add_argument(
+            'model', nargs='*',
+            help="Limit the rebuild to these models, as app_label.ModelName (default: all)",
+        )
+
+    def get_models(self, names):
+        """
+        Return the concrete core hierarchical models to operate on, ordered by table name.
+
+        Plugin models are excluded, including when named explicitly: the SQL which rebuilds
+        `sort_path` reads the name column by name, while `InstallLtreeTriggers` lets a plugin
+        maintain it from any column, so rebuilding one is not something this command can do
+        correctly. A plugin in that position needs its own repair path.
+        """
+        def concrete_subclasses(base):
+            for subclass in base.__subclasses__():
+                if subclass._meta.abstract:
+                    yield from concrete_subclasses(subclass)
+                elif not isinstance(apps.get_app_config(subclass._meta.app_label), PluginConfig):
+                    yield subclass
+
+        candidates = {
+            model._meta.label_lower: model for model in concrete_subclasses(LtreeModel)
+        }
+
+        if not names:
+            return sorted(candidates.values(), key=lambda model: model._meta.db_table)
+
+        models = []
+        for name in names:
+            model = candidates.get(name.lower())
+            if model is None:
+                raise CommandError(f"{name} is not a core hierarchical (ltree-backed) model")
+            models.append(model)
+        return models
+
+    def handle(self, *args, **options):
+        for model in self.get_models(options['model']):
+            # populate_paths_sql() is the same SQL which backfilled these columns during the
+            # ltree migrations. It relies on SET LOCAL, so it must run inside a transaction,
+            # and the UPDATE it emits locks every row in the table until it commits.
+            self.stdout.write(f'{model._meta.label_lower}: rebuilding... ', ending='')
+            self.stdout.flush()
+            with transaction.atomic(), connection.cursor() as cursor:
+                cursor.execute(
+                    populate_paths_sql(model._meta.db_table, sort_path=model._has_sort_path())
+                )
+            self.stdout.write(self.style.SUCCESS('done'))
+
+        self.stdout.write(self.style.SUCCESS('Finished.'))

+ 56 - 0
netbox/utilities/tests/test_management_commands.py

@@ -2,8 +2,10 @@ from io import StringIO
 from unittest.mock import MagicMock, patch
 
 from django.core.management import call_command
+from django.core.management.base import CommandError
 from django.test import TestCase
 
+from dcim.models import Region
 from utilities.management.commands.calculate_cached_counts import Command
 
 
@@ -49,3 +51,57 @@ class CalculateCachedCountsTestCase(TestCase):
         ChildModel._meta.get_field.assert_called_once_with('parent')
         fk_field.related_query_name.assert_called_once_with()
         self.assertEqual(dict(models), {ParentModel: {'child_count': 'children'}})
+
+
+class RebuildLtreePathsTestCase(TestCase):
+    """
+    The command must repair path/sort_path values the triggers did not maintain.
+
+    Corruption is injected by writing the path columns directly: the triggers fire on
+    parent_id and the name column, so a raw UPDATE of path bypasses them, reproducing a
+    database whose cascade trigger went missing across a restore.
+    """
+
+    @classmethod
+    def setUpTestData(cls):
+        cls.parent = Region.objects.create(name='Alpha', slug='alpha-rlp')
+        cls.child = Region.objects.create(name='Beta', slug='beta-rlp', parent=cls.parent)
+
+    def test_rebuilds_stale_path_and_sort_path(self):
+        Region.objects.filter(pk=self.child.pk).update(
+            path='9999999999999999999', sort_path='stale',
+        )
+
+        call_command('rebuild_ltree_paths', 'dcim.region', stdout=StringIO())
+
+        self.child.refresh_from_db()
+        self.assertEqual(
+            self.child.path,
+            f'{str(self.parent.pk).zfill(19)}.{str(self.child.pk).zfill(19)}',
+        )
+        self.assertEqual(self.child.sort_path, f'Alpha{chr(9)}Beta')
+
+    def test_rebuilds_a_stale_sort_path_alone(self):
+        # What a rename leaves behind: the renamed row's own sort_path is rewritten by the
+        # BEFORE trigger, its descendants' are not, and no path changes.
+        Region.objects.filter(pk=self.child.pk).update(sort_path='stale')
+
+        call_command('rebuild_ltree_paths', 'dcim.region', stdout=StringIO())
+
+        self.child.refresh_from_db()
+        self.assertEqual(self.child.sort_path, f'Alpha{chr(9)}Beta')
+
+    def test_rebuilds_every_core_hierarchical_model_by_default(self):
+        out = StringIO()
+
+        call_command('rebuild_ltree_paths', stdout=out)
+
+        output = out.getvalue()
+        for label in ('dcim.region', 'dcim.inventoryitem', 'dcim.inventoryitemtemplate',
+                      'tenancy.tenantgroup', 'wireless.wirelesslangroup'):
+            self.assertIn(label, output)
+        self.assertIn('Finished.', output)
+
+    def test_rejects_a_model_which_is_not_hierarchical(self):
+        with self.assertRaises(CommandError):
+            call_command('rebuild_ltree_paths', 'dcim.site')