Ver código fonte

Entropy trace (#1659)

* I do an entropy check earlier than the others and add the info to the trace

Maybe I'm wrong, but it seems to me that if the regex allowlist is large, it's easier to first check for entropy, since we won't have to go through all these regexes. Plus, we need to check entropy anyway.
Additionally, when using -l trace, there was no information that the find was missed precisely because of entropy. Now it will be shown

* adding info into trace about entropy

* Fixed name of variables

---------

Co-authored-by: Almaz Vakhitov <a.vakhitov@vk.team>
almaz045 1 ano atrás
pai
commit
b69b5157a0
1 arquivos alterados com 24 adições e 23 exclusões
  1. 24 23
      detect/detect.go

+ 24 - 23
detect/detect.go

@@ -420,6 +420,30 @@ MatchLoop:
 			}
 			}
 		}
 		}
 
 
+		// check entropy
+		entropy := shannonEntropy(finding.Secret)
+		finding.Entropy = float32(entropy)
+		if r.Entropy != 0.0 {
+			if entropy <= r.Entropy {
+				logger.Trace().
+					Float32("entropy", finding.Entropy).
+					Msg("Skipping finding due to low entropy")
+				// entropy is too low, skip this finding
+				continue
+			}
+			// NOTE: this is a goofy hack to get around the fact there golang's regex engine
+			// does not support positive lookaheads. Ideally we would want to add a
+			// restriction on generic rules regex that requires the secret match group
+			// contains both numbers and alphabetical characters, not just alphabetical characters.
+			// What this bit of code does is check if the ruleid is prepended with "generic" and enforces the
+			// secret contains both digits and alphabetical characters.
+			// TODO: this should be replaced with stop words
+			if strings.HasPrefix(r.RuleID, "generic") {
+				if !containsDigit(finding.Secret) {
+					continue
+				}
+			}
+		}
 		// check if the regexTarget is defined in the allowlist "regexes" entry
 		// check if the regexTarget is defined in the allowlist "regexes" entry
 		// or if the secret is in the list of stopwords
 		// or if the secret is in the list of stopwords
 		globalAllowlistTarget := finding.Secret
 		globalAllowlistTarget := finding.Secret
@@ -489,29 +513,6 @@ MatchLoop:
 				continue MatchLoop
 				continue MatchLoop
 			}
 			}
 		}
 		}
-
-		// check entropy
-		entropy := shannonEntropy(finding.Secret)
-		finding.Entropy = float32(entropy)
-		if r.Entropy != 0.0 {
-			if entropy <= r.Entropy {
-				// entropy is too low, skip this finding
-				continue
-			}
-			// NOTE: this is a goofy hack to get around the fact there golang's regex engine
-			// does not support positive lookaheads. Ideally we would want to add a
-			// restriction on generic rules regex that requires the secret match group
-			// contains both numbers and alphabetical characters, not just alphabetical characters.
-			// What this bit of code does is check if the ruleid is prepended with "generic" and enforces the
-			// secret contains both digits and alphabetical characters.
-			// TODO: this should be replaced with stop words
-			if strings.HasPrefix(r.RuleID, "generic") {
-				if !containsDigit(finding.Secret) {
-					continue
-				}
-			}
-		}
-
 		findings = append(findings, finding)
 		findings = append(findings, finding)
 	}
 	}
 	return findings
 	return findings