소스 검색

Entropy trace (#1659)

* I do an entropy check earlier than the others and add the info to the trace

Maybe I'm wrong, but it seems to me that if the regex allowlist is large, it's easier to first check for entropy, since we won't have to go through all these regexes. Plus, we need to check entropy anyway.
Additionally, when using -l trace, there was no information that the find was missed precisely because of entropy. Now it will be shown

* adding info into trace about entropy

* Fixed name of variables

---------

Co-authored-by: Almaz Vakhitov <a.vakhitov@vk.team>
almaz045 1 년 전
부모
커밋
b69b5157a0
1개의 변경된 파일24개의 추가작업 그리고 23개의 파일을 삭제
  1. 24 23
      detect/detect.go

+ 24 - 23
detect/detect.go

@@ -420,6 +420,30 @@ MatchLoop:
 			}
 		}
 
+		// check entropy
+		entropy := shannonEntropy(finding.Secret)
+		finding.Entropy = float32(entropy)
+		if r.Entropy != 0.0 {
+			if entropy <= r.Entropy {
+				logger.Trace().
+					Float32("entropy", finding.Entropy).
+					Msg("Skipping finding due to low entropy")
+				// entropy is too low, skip this finding
+				continue
+			}
+			// NOTE: this is a goofy hack to get around the fact there golang's regex engine
+			// does not support positive lookaheads. Ideally we would want to add a
+			// restriction on generic rules regex that requires the secret match group
+			// contains both numbers and alphabetical characters, not just alphabetical characters.
+			// What this bit of code does is check if the ruleid is prepended with "generic" and enforces the
+			// secret contains both digits and alphabetical characters.
+			// TODO: this should be replaced with stop words
+			if strings.HasPrefix(r.RuleID, "generic") {
+				if !containsDigit(finding.Secret) {
+					continue
+				}
+			}
+		}
 		// check if the regexTarget is defined in the allowlist "regexes" entry
 		// or if the secret is in the list of stopwords
 		globalAllowlistTarget := finding.Secret
@@ -489,29 +513,6 @@ MatchLoop:
 				continue MatchLoop
 			}
 		}
-
-		// check entropy
-		entropy := shannonEntropy(finding.Secret)
-		finding.Entropy = float32(entropy)
-		if r.Entropy != 0.0 {
-			if entropy <= r.Entropy {
-				// entropy is too low, skip this finding
-				continue
-			}
-			// NOTE: this is a goofy hack to get around the fact there golang's regex engine
-			// does not support positive lookaheads. Ideally we would want to add a
-			// restriction on generic rules regex that requires the secret match group
-			// contains both numbers and alphabetical characters, not just alphabetical characters.
-			// What this bit of code does is check if the ruleid is prepended with "generic" and enforces the
-			// secret contains both digits and alphabetical characters.
-			// TODO: this should be replaced with stop words
-			if strings.HasPrefix(r.RuleID, "generic") {
-				if !containsDigit(finding.Secret) {
-					continue
-				}
-			}
-		}
-
 		findings = append(findings, finding)
 	}
 	return findings