Browse Source

Entropy trace (#1659)

* I do an entropy check earlier than the others and add the info to the trace

Maybe I'm wrong, but it seems to me that if the regex allowlist is large, it's easier to first check for entropy, since we won't have to go through all these regexes. Plus, we need to check entropy anyway.
Additionally, when using -l trace, there was no information that the find was missed precisely because of entropy. Now it will be shown

* adding info into trace about entropy

* Fixed name of variables

---------

Co-authored-by: Almaz Vakhitov <a.vakhitov@vk.team>
almaz045 1 năm trước cách đây
mục cha
commit
5494c3ed33
1 tập tin đã thay đổi với 24 bổ sung23 xóa
  1. 24 23
      detect/detect.go

+ 24 - 23
detect/detect.go

@@ -420,6 +420,30 @@ MatchLoop:
 			}
 		}
 
+		// check entropy
+		entropy := shannonEntropy(finding.Secret)
+		finding.Entropy = float32(entropy)
+		if r.Entropy != 0.0 {
+			if entropy <= r.Entropy {
+				logger.Trace().
+					Float32("entropy", finding.Entropy).
+					Msg("Skipping finding due to low entropy")
+				// entropy is too low, skip this finding
+				continue
+			}
+			// NOTE: this is a goofy hack to get around the fact there golang's regex engine
+			// does not support positive lookaheads. Ideally we would want to add a
+			// restriction on generic rules regex that requires the secret match group
+			// contains both numbers and alphabetical characters, not just alphabetical characters.
+			// What this bit of code does is check if the ruleid is prepended with "generic" and enforces the
+			// secret contains both digits and alphabetical characters.
+			// TODO: this should be replaced with stop words
+			if strings.HasPrefix(r.RuleID, "generic") {
+				if !containsDigit(finding.Secret) {
+					continue
+				}
+			}
+		}
 		// check if the regexTarget is defined in the allowlist "regexes" entry
 		// or if the secret is in the list of stopwords
 		globalAllowlistTarget := finding.Secret
@@ -489,29 +513,6 @@ MatchLoop:
 				continue MatchLoop
 			}
 		}
-
-		// check entropy
-		entropy := shannonEntropy(finding.Secret)
-		finding.Entropy = float32(entropy)
-		if r.Entropy != 0.0 {
-			if entropy <= r.Entropy {
-				// entropy is too low, skip this finding
-				continue
-			}
-			// NOTE: this is a goofy hack to get around the fact there golang's regex engine
-			// does not support positive lookaheads. Ideally we would want to add a
-			// restriction on generic rules regex that requires the secret match group
-			// contains both numbers and alphabetical characters, not just alphabetical characters.
-			// What this bit of code does is check if the ruleid is prepended with "generic" and enforces the
-			// secret contains both digits and alphabetical characters.
-			// TODO: this should be replaced with stop words
-			if strings.HasPrefix(r.RuleID, "generic") {
-				if !containsDigit(finding.Secret) {
-					continue
-				}
-			}
-		}
-
 		findings = append(findings, finding)
 	}
 	return findings