Explorar o código

Global allowlist regex (#422)

* work in progress

* Update test configs

* adding test

* golint and change to allowlist

* fix race condition, this mistakenly got committed

* Updating tests and examples

* bump to v6 as this is a breaking change to the config

* More consistent variables
Zachary Rice %!s(int64=5) %!d(string=hai) anos
pai
achega
fae366a404
Modificáronse 69 ficheiros con 449 adicións e 204 borrados
  1. 1 1
      Makefile
  2. 83 71
      config/config.go
  3. 2 2
      config/config_test.go
  4. 21 29
      examples/leaky-repo.toml
  5. 3 3
      examples/simple_regex_and_allowlist_config.toml
  6. 1 1
      go.mod
  7. 3 3
      hosts/github.go
  8. 3 3
      hosts/gitlab.go
  9. 1 1
      hosts/host.go
  10. 3 3
      hosts/hosts_test.go
  11. 5 5
      main.go
  12. 3 3
      manager/manager.go
  13. 2 2
      manager/manager_test.go
  14. 1 1
      options/options.go
  15. 5 5
      scan/repo.go
  16. 17 39
      scan/rule.go
  17. 4 3
      scan/scan.go
  18. 13 3
      scan/scan_test.go
  19. 4 10
      test_data/test_configs/aws_key_allowlist_files.toml
  20. 2 2
      test_data/test_configs/aws_key_allowlist_python_files.toml
  21. 2 2
      test_data/test_configs/aws_key_aws_allowlisted.toml
  22. 8 8
      test_data/test_configs/large.toml
  23. 151 0
      test_data/test_configs/large_with_global_allowlist_regex.toml
  24. 1 1
      test_data/test_configs/regex_filename.toml
  25. 1 1
      test_data/test_configs/regex_filepath.toml
  26. 2 2
      test_data/test_configs/regex_filepath_filename.toml
  27. 15 0
      test_data/test_local_owner_aws_leak.json
  28. 15 0
      test_data/test_local_owner_aws_leak_allowlist_repo.json
  29. 15 0
      test_data/test_local_owner_aws_leak_depth_2.json
  30. 17 0
      test_data/test_local_repo_nine_aws_leak.json
  31. 1 0
      test_data/test_repos/test_repo_9/dotGit/COMMIT_EDITMSG
  32. 1 0
      test_data/test_repos/test_repo_9/dotGit/HEAD
  33. 7 0
      test_data/test_repos/test_repo_9/dotGit/config
  34. 1 0
      test_data/test_repos/test_repo_9/dotGit/description
  35. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/index
  36. 6 0
      test_data/test_repos/test_repo_9/dotGit/info/exclude
  37. 3 0
      test_data/test_repos/test_repo_9/dotGit/logs/HEAD
  38. 3 0
      test_data/test_repos/test_repo_9/dotGit/logs/refs/heads/master
  39. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/10/fa14c5ab0134436e2ae435138bf921eb477c60
  40. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/3a/76f3781306faf5612017bf18a4b4bdb9f927bf
  41. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/41/42082bcb939bbc17985a69ba748491ac6b62a5
  42. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/49/8b267a8c7812490d6479839c5577eaaec79d62
  43. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/60/c9e47f150a6b713e247e6105b77f1b961f844f
  44. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/61/87dbf4390fc6e28445dd3d988aefb9d1111988
  45. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/65/57c92612d3b35979bd426d429255b3bf9fab74
  46. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/6a/756416384c210ada2631f17862f5c01fffa478
  47. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/6c/9406b7d9320db083eca69b3f8bee9a6c7b50d4
  48. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/6c/bef5c370d8c3486ca85423dd70440c5e0a2aa2
  49. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/80/8b12c5ca4b142367932e7045d555a639fc148c
  50. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/80/ba94135cc378364af9d3cb2450df48e51faf2c
  51. 2 0
      test_data/test_repos/test_repo_9/dotGit/objects/8d/1fb60d2d80f0590f191ed5ace1e45ef780909a
  52. 3 0
      test_data/test_repos/test_repo_9/dotGit/objects/9e/523225b31add24e72f2feb0b2645cfb36542dc
  53. 3 0
      test_data/test_repos/test_repo_9/dotGit/objects/a1/fd29ec14823d8bc4a8d1a2cfe35451580f5118
  54. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/a4/fa2187727281aea78d7c3aaebdb4b924fc4e4d
  55. 1 0
      test_data/test_repos/test_repo_9/dotGit/objects/a5/196d1be8fb59edf8062bef36d3a602e0812139
  56. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/a5/d7b84a673458d14d9aab082183a1968c2c7492
  57. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/b5/8d1184a9d43a39c0d95f32453efc78581877d6
  58. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/c1/f3c6341548a65be1eded56b4f7cacec7fe9090
  59. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/c9/8e6c52cbd1f50de572ff12a3441271fccff705
  60. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/cb/089cd89a7d7686d284d8761201649346b5aa1c
  61. 2 0
      test_data/test_repos/test_repo_9/dotGit/objects/cb/19a50e8cdeb7011eccdb13f3b739f00d775bab
  62. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/d2/74003914c707212cbe84e3e466a00013ccb639
  63. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/e6/73bb3980f3c286291809e05f80873852bc3e9c
  64. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/e9/e5396f7e52aa48de485b4836ebb041cc7f7c46
  65. 2 0
      test_data/test_repos/test_repo_9/dotGit/objects/ec/17ec1939b7c3e86b7cb6c0c4de6b0818a7e75e
  66. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/f9/f2f36962fa38ca4525b99ef702fba03e3a79a7
  67. BIN=BIN
      test_data/test_repos/test_repo_9/dotGit/objects/fe/f3190e94723e78847ab9f0daaa15add4ffd4ef
  68. 1 0
      test_data/test_repos/test_repo_9/dotGit/refs/heads/master
  69. 9 0
      test_data/test_repos/test_repo_9/server.test.py

+ 1 - 1
Makefile

@@ -3,7 +3,7 @@
 VERSION := `git fetch --tags && git tag | sort -V | tail -1`
 VERSION := `git fetch --tags && git tag | sort -V | tail -1`
 PKG=github.com/zricethezav/gitleaks
 PKG=github.com/zricethezav/gitleaks
 LDFLAGS=-ldflags "-X=github.com/zricethezav/gitleaks/v5/version.Version=$(VERSION)"
 LDFLAGS=-ldflags "-X=github.com/zricethezav/gitleaks/v5/version.Version=$(VERSION)"
-_LDFLAGS="github.com/zricethezav/gitleaks/v5/version.Version=$(VERSION)"
+_LDFLAGS="github.com/zricethezav/gitleaks/v6/version.Version=$(VERSION)"
 COVER=--cover --coverprofile=cover.out
 COVER=--cover --coverprofile=cover.out
 
 
 test-cover:
 test-cover:

+ 83 - 71
config/config.go

@@ -6,19 +6,21 @@ import (
 	"regexp"
 	"regexp"
 	"strconv"
 	"strconv"
 
 
-	"github.com/zricethezav/gitleaks/v5/options"
+	"github.com/zricethezav/gitleaks/v6/options"
 
 
 	"github.com/BurntSushi/toml"
 	"github.com/BurntSushi/toml"
 	log "github.com/sirupsen/logrus"
 	log "github.com/sirupsen/logrus"
 )
 )
 
 
-// Allowlist is struct containing items that if encountered will allowlist
+// AllowList is struct containing items that if encountered will allowlist
 // a commit/line of code that would be considered a leak.
 // a commit/line of code that would be considered a leak.
 type Allowlist struct {
 type Allowlist struct {
 	Description string
 	Description string
-	Regex       *regexp.Regexp
-	File        *regexp.Regexp
-	Path        *regexp.Regexp
+	Regexes     []*regexp.Regexp
+	Commits     []string
+	Files       []*regexp.Regexp
+	Paths       []*regexp.Regexp
+	Repos       []*regexp.Regexp
 }
 }
 
 
 // Entropy represents an entropy range
 // Entropy represents an entropy range
@@ -34,56 +36,50 @@ type Entropy struct {
 // that match is not allowlisted (globally or locally), then a leak will be appended
 // that match is not allowlisted (globally or locally), then a leak will be appended
 // to the final scan report.
 // to the final scan report.
 type Rule struct {
 type Rule struct {
-	Description   string
-	Regex         *regexp.Regexp
-	FileNameRegex *regexp.Regexp
-	FilePathRegex *regexp.Regexp
-	Tags          []string
-	Allowlist     []Allowlist
-	Entropies     []Entropy
+	Description string
+	Regex       *regexp.Regexp
+	File        *regexp.Regexp
+	Path        *regexp.Regexp
+	Tags        []string
+	Allowlist   Allowlist
+	Entropies   []Entropy
 }
 }
 
 
 // Config is a composite struct of Rules and Allowlists
 // Config is a composite struct of Rules and Allowlists
 // Each Rule contains a description, regular expression, tags, and allowlists if available
 // Each Rule contains a description, regular expression, tags, and allowlists if available
 type Config struct {
 type Config struct {
 	Rules     []Rule
 	Rules     []Rule
-	Allowlist struct {
-		Description string
-		Commits     []string
-		Files       []*regexp.Regexp
-		Paths       []*regexp.Regexp
-		Repos       []*regexp.Regexp
-	}
+	Allowlist Allowlist
+}
+
+// TomlAllowList is a struct used in the TomlLoader that loads in allowlists from
+// specific rules or globally at the top level config
+type TomlAllowList struct {
+	Description string
+	Regexes     []string
+	Commits     []string
+	Files       []string
+	Paths       []string
+	Repos       []string
 }
 }
 
 
 // TomlLoader gets loaded with the values from a gitleaks toml config
 // TomlLoader gets loaded with the values from a gitleaks toml config
 // see the config in config/defaults.go for an example. TomlLoader is used
 // see the config in config/defaults.go for an example. TomlLoader is used
 // to generate Config values (compiling regexes, etc).
 // to generate Config values (compiling regexes, etc).
 type TomlLoader struct {
 type TomlLoader struct {
-	Allowlist struct {
+	AllowList TomlAllowList
+	Rules     []struct {
 		Description string
 		Description string
-		Commits     []string
-		Files       []string
-		Paths       []string
-		Repos       []string
-	}
-	Rules []struct {
-		Description   string
-		Regex         string
-		FileNameRegex string
-		FilePathRegex string
-		Tags          []string
-		Entropies     []struct {
+		Regex       string
+		File        string
+		Path        string
+		Tags        []string
+		Entropies   []struct {
 			Min   string
 			Min   string
 			Max   string
 			Max   string
 			Group string
 			Group string
 		}
 		}
-		Allowlist []struct {
-			Description string
-			Regex       string
-			File        string
-			Path        string
-		}
+		AllowList TomlAllowList
 	}
 	}
 }
 }
 
 
@@ -99,7 +95,7 @@ func NewConfig(options options.Options) (Config, error) {
 	if options.Config != "" {
 	if options.Config != "" {
 		_, err = toml.DecodeFile(options.Config, &tomlLoader)
 		_, err = toml.DecodeFile(options.Config, &tomlLoader)
 		// append a allowlist rule for allowlisting the config
 		// append a allowlist rule for allowlisting the config
-		tomlLoader.Allowlist.Files = append(tomlLoader.Allowlist.Files, path.Base(options.Config))
+		tomlLoader.AllowList.Files = append(tomlLoader.AllowList.Files, path.Base(options.Config))
 	} else {
 	} else {
 		_, err = toml.Decode(DefaultConfig, &tomlLoader)
 		_, err = toml.Decode(DefaultConfig, &tomlLoader)
 	}
 	}
@@ -121,7 +117,7 @@ func (tomlLoader TomlLoader) Parse() (Config, error) {
 	var cfg Config
 	var cfg Config
 	for _, rule := range tomlLoader.Rules {
 	for _, rule := range tomlLoader.Rules {
 		// check and make sure the rule is valid
 		// check and make sure the rule is valid
-		if rule.Regex == "" && rule.FilePathRegex == "" && rule.FileNameRegex == "" && len(rule.Entropies) == 0 {
+		if rule.Regex == "" && rule.Path == "" && rule.File == "" && len(rule.Entropies) == 0 {
 			log.Warnf("Rule %s does not define any actionable data", rule.Description)
 			log.Warnf("Rule %s does not define any actionable data", rule.Description)
 			continue
 			continue
 		}
 		}
@@ -129,36 +125,43 @@ func (tomlLoader TomlLoader) Parse() (Config, error) {
 		if err != nil {
 		if err != nil {
 			return cfg, fmt.Errorf("problem loading config: %v", err)
 			return cfg, fmt.Errorf("problem loading config: %v", err)
 		}
 		}
-		fileNameRe, err := regexp.Compile(rule.FileNameRegex)
+		fileNameRe, err := regexp.Compile(rule.File)
 		if err != nil {
 		if err != nil {
 			return cfg, fmt.Errorf("problem loading config: %v", err)
 			return cfg, fmt.Errorf("problem loading config: %v", err)
 		}
 		}
-		filePathRe, err := regexp.Compile(rule.FilePathRegex)
+		filePathRe, err := regexp.Compile(rule.Path)
 		if err != nil {
 		if err != nil {
 			return cfg, fmt.Errorf("problem loading config: %v", err)
 			return cfg, fmt.Errorf("problem loading config: %v", err)
 		}
 		}
 
 
 		// rule specific allowlists
 		// rule specific allowlists
-		var allowlists []Allowlist
-		for _, wl := range rule.Allowlist {
-			wlRe, err := regexp.Compile(wl.Regex)
+		var allowList Allowlist
+
+		// rule specific regexes
+		for _, re := range rule.AllowList.Regexes {
+			allowListedRegex, err := regexp.Compile(re)
 			if err != nil {
 			if err != nil {
 				return cfg, fmt.Errorf("problem loading config: %v", err)
 				return cfg, fmt.Errorf("problem loading config: %v", err)
 			}
 			}
-			wlFileNameRe, err := regexp.Compile(wl.File)
+			allowList.Regexes = append(allowList.Regexes, allowListedRegex)
+		}
+
+		// rule specific filenames
+		for _, re := range rule.AllowList.Files {
+			allowListedRegex, err := regexp.Compile(re)
 			if err != nil {
 			if err != nil {
 				return cfg, fmt.Errorf("problem loading config: %v", err)
 				return cfg, fmt.Errorf("problem loading config: %v", err)
 			}
 			}
-			wlFilePathRe, err := regexp.Compile(wl.Path)
+			allowList.Files = append(allowList.Files, allowListedRegex)
+		}
+
+		// rule specific paths
+		for _, re := range rule.AllowList.Paths {
+			allowListedRegex, err := regexp.Compile(re)
 			if err != nil {
 			if err != nil {
 				return cfg, fmt.Errorf("problem loading config: %v", err)
 				return cfg, fmt.Errorf("problem loading config: %v", err)
 			}
 			}
-			allowlists = append(allowlists, Allowlist{
-				Description: wl.Description,
-				File:        wlFileNameRe,
-				Path:        wlFilePathRe,
-				Regex:       wlRe,
-			})
+			allowList.Paths = append(allowList.Paths, allowListedRegex)
 		}
 		}
 
 
 		var entropies []Entropy
 		var entropies []Entropy
@@ -191,47 +194,56 @@ func (tomlLoader TomlLoader) Parse() (Config, error) {
 		}
 		}
 
 
 		r := Rule{
 		r := Rule{
-			Description:   rule.Description,
-			Regex:         re,
-			FileNameRegex: fileNameRe,
-			FilePathRegex: filePathRe,
-			Tags:          rule.Tags,
-			Allowlist:     allowlists,
-			Entropies:     entropies,
+			Description: rule.Description,
+			Regex:       re,
+			File:        fileNameRe,
+			Path:        filePathRe,
+			Tags:        rule.Tags,
+			Allowlist:   allowList,
+			Entropies:   entropies,
 		}
 		}
 
 
 		cfg.Rules = append(cfg.Rules, r)
 		cfg.Rules = append(cfg.Rules, r)
 	}
 	}
 
 
-	// global file name allowlists
-	for _, wlFileName := range tomlLoader.Allowlist.Files {
-		re, err := regexp.Compile(wlFileName)
+	// global regex allowLists
+	for _, allowListRegex := range tomlLoader.AllowList.Regexes {
+		re, err := regexp.Compile(allowListRegex)
+		if err != nil {
+			return cfg, fmt.Errorf("problem loading config: %v", err)
+		}
+		cfg.Allowlist.Regexes = append(cfg.Allowlist.Regexes, re)
+	}
+
+	// global file name allowLists
+	for _, allowListFileName := range tomlLoader.AllowList.Files {
+		re, err := regexp.Compile(allowListFileName)
 		if err != nil {
 		if err != nil {
 			return cfg, fmt.Errorf("problem loading config: %v", err)
 			return cfg, fmt.Errorf("problem loading config: %v", err)
 		}
 		}
 		cfg.Allowlist.Files = append(cfg.Allowlist.Files, re)
 		cfg.Allowlist.Files = append(cfg.Allowlist.Files, re)
 	}
 	}
 
 
-	// global file path allowlists
-	for _, wlFilePath := range tomlLoader.Allowlist.Paths {
-		re, err := regexp.Compile(wlFilePath)
+	// global file path allowLists
+	for _, allowListFilePath := range tomlLoader.AllowList.Paths {
+		re, err := regexp.Compile(allowListFilePath)
 		if err != nil {
 		if err != nil {
 			return cfg, fmt.Errorf("problem loading config: %v", err)
 			return cfg, fmt.Errorf("problem loading config: %v", err)
 		}
 		}
 		cfg.Allowlist.Paths = append(cfg.Allowlist.Paths, re)
 		cfg.Allowlist.Paths = append(cfg.Allowlist.Paths, re)
 	}
 	}
 
 
-	// global repo allowlists
-	for _, wlRepo := range tomlLoader.Allowlist.Repos {
-		re, err := regexp.Compile(wlRepo)
+	// global repo allowLists
+	for _, allowListRepo := range tomlLoader.AllowList.Repos {
+		re, err := regexp.Compile(allowListRepo)
 		if err != nil {
 		if err != nil {
 			return cfg, fmt.Errorf("problem loading config: %v", err)
 			return cfg, fmt.Errorf("problem loading config: %v", err)
 		}
 		}
 		cfg.Allowlist.Repos = append(cfg.Allowlist.Repos, re)
 		cfg.Allowlist.Repos = append(cfg.Allowlist.Repos, re)
 	}
 	}
 
 
-	cfg.Allowlist.Commits = tomlLoader.Allowlist.Commits
-	cfg.Allowlist.Description = tomlLoader.Allowlist.Description
+	cfg.Allowlist.Commits = tomlLoader.AllowList.Commits
+	cfg.Allowlist.Description = tomlLoader.AllowList.Description
 
 
 	return cfg, nil
 	return cfg, nil
 }
 }

+ 2 - 2
config/config_test.go

@@ -5,7 +5,7 @@ import (
 	"regexp"
 	"regexp"
 	"testing"
 	"testing"
 
 
-	"github.com/zricethezav/gitleaks/v5/options"
+	"github.com/zricethezav/gitleaks/v6/options"
 )
 )
 
 
 func TestParse(t *testing.T) {
 func TestParse(t *testing.T) {
@@ -122,7 +122,7 @@ func TestParse(t *testing.T) {
 		_, err := NewConfig(test.opts)
 		_, err := NewConfig(test.opts)
 		if err != nil {
 		if err != nil {
 			if test.wantErr == nil {
 			if test.wantErr == nil {
-				t.Error(err)
+				t.Error(test.description, err)
 			} else if test.wantErr.Error() != err.Error() {
 			} else if test.wantErr.Error() != err.Error() {
 				t.Errorf("expected err: %s, got %s", test.wantErr, err)
 				t.Errorf("expected err: %s, got %s", test.wantErr, err)
 			}
 			}

+ 21 - 29
examples/leaky-repo.toml

@@ -129,8 +129,8 @@ title = "gitleaks config"
 [[rules]]
 [[rules]]
 	description = "Port"
 	description = "Port"
 	regex = '''(?i)port(.{0,4})?[0-9]{1,10}'''
 	regex = '''(?i)port(.{0,4})?[0-9]{1,10}'''
-	[[rules.allowlist]]
-		regex = '''(?i)port '''
+	[rules.allowlist]
+		regexes = ['''(?i)port ''']
 		description = "ignore export "
 		description = "ignore export "
 
 
 
 
@@ -139,8 +139,8 @@ title = "gitleaks config"
 	description = "Email"
 	description = "Email"
 	regex = '''[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,4}'''
 	regex = '''[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,4}'''
 	tags = ["email"]
 	tags = ["email"]
-	[[rules.allowlist]]
-		file = '''(?i)bashrc'''
+	[rules.allowlist]
+		files = ['''(?i)bashrc''']
 		description = "ignore bashrc emails"
 		description = "ignore bashrc emails"
 
 
 
 
@@ -149,35 +149,26 @@ title = "gitleaks config"
 	regex = '''(?i)(dbpasswd|dbuser|dbname|dbhost|api_key|apikey|secret|key|api|password|user|guid|hostname|pw|auth)(.{0,20})?['|"]([0-9a-zA-Z-_\/+!{}/=]{4,120})['|"]'''
 	regex = '''(?i)(dbpasswd|dbuser|dbname|dbhost|api_key|apikey|secret|key|api|password|user|guid|hostname|pw|auth)(.{0,20})?['|"]([0-9a-zA-Z-_\/+!{}/=]{4,120})['|"]'''
 	tags = ["key", "API", "generic"]
 	tags = ["key", "API", "generic"]
 	# ignore leaks with specific identifiers like slack and aws
 	# ignore leaks with specific identifiers like slack and aws
-	[[rules.allowlist]]
-		regex = '''xox[baprs]-([0-9a-zA-Z]{10,48})'''
-		description = "ignore slack"
-	[[rules.allowlist]]
-		description = "MailChimp API key"
-		regex = '''(?i)(.{0,20})?['"][0-9a-f]{32}-us[0-9]{1,2}['"]'''
-	[[rules.allowlist]]
-		description = "AWS Manager ID"
-		regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
-
-#[[rules]]
-#	description = "s3config"
-#  	regex = '''(?i)(dbpasswd|dbuser|dbname|dbhost|api_key|apikey|key|api|password|user|guid|hostname|pw|auth)(.{0,3})?([0-9a-zA-Z-_\/+!{}=]{4,120})'''
-#	fileNameRegex = '''(?i)s3cfg$'''
+	[rules.allowlist]
+		description = "ignore slack, mailchimp, aws"
+		regexes = [
+		    '''xox[baprs]-([0-9a-zA-Z]{10,48})''',
+		    '''(?i)(.{0,20})?['"][0-9a-f]{32}-us[0-9]{1,2}['"]''',
+		    '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
+		]
 
 
 [[rules]]
 [[rules]]
 	description = "High Entropy"
 	description = "High Entropy"
 	regex = '''[0-9a-zA-Z-_!{}/=]{4,120}'''
 	regex = '''[0-9a-zA-Z-_!{}/=]{4,120}'''
-  	fileNameRegex = '''(?i)(dump.sql|high-entropy-misc.txt)$'''
+  	file = '''(?i)(dump.sql|high-entropy-misc.txt)$'''
 	tags = ["entropy"]
 	tags = ["entropy"]
-        [[rules.Entropies]]
-            Min = "4.3"
-            Max = "7.0"
-        [[rules.allowlist]]
-            description = "ignore ssh key and pems"
-            file = '''(pem|ppk|env)$'''
-            path = '''(.*)?ssh'''
-
-
+    [[rules.Entropies]]
+        Min = "4.3"
+        Max = "7.0"
+    [rules.allowlist]
+        description = "ignore ssh key and pems"
+        files = ['''(pem|ppk|env)$''']
+        paths = ['''(.*)?ssh''']
 
 
 [[rules]]
 [[rules]]
 	description = "Potential bash var"
 	description = "Potential bash var"
@@ -195,8 +186,9 @@ title = "gitleaks config"
 
 
 [[rules]]
 [[rules]]
 	description = "Files with keys and credentials"
 	description = "Files with keys and credentials"
-    fileNameRegex = '''(?i)(id_rsa|passwd|id_rsa.pub|pgpass|pem|key|shadow)'''
+    file = '''(?i)(id_rsa|passwd|id_rsa.pub|pgpass|pem|key|shadow)'''
 
 
+# Global allowlist
 [allowlist]
 [allowlist]
 	description = "image allowlists"
 	description = "image allowlists"
 	files = ['''(.*?)(jpg|gif|doc|pdf|bin)$''']
 	files = ['''(.*?)(jpg|gif|doc|pdf|bin)$''']

+ 3 - 3
examples/simple_regex_and_allowlist_config.toml

@@ -8,6 +8,6 @@
     description = "AWS Manager ID"
     description = "AWS Manager ID"
     regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
     regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
     tags = ["key", "AWS"]
     tags = ["key", "AWS"]
-        [[rules.allowlist]]
-            regex = '''AKIAIO5FODNN7EXAMPLE.*'''
-            description = "ignore example aws key"
+    [rules.allowlist]
+        regexes = ['''AKIAIO5FODNN7EXAMPLE.*''']
+        description = "ignore example aws key"

+ 1 - 1
go.mod

@@ -1,4 +1,4 @@
-module github.com/zricethezav/gitleaks/v5
+module github.com/zricethezav/gitleaks/v6
 
 
 go 1.14
 go 1.14
 
 

+ 3 - 3
hosts/github.go

@@ -6,9 +6,9 @@ import (
 	"strings"
 	"strings"
 	"sync"
 	"sync"
 
 
-	"github.com/zricethezav/gitleaks/v5/manager"
-	"github.com/zricethezav/gitleaks/v5/options"
-	"github.com/zricethezav/gitleaks/v5/scan"
+	"github.com/zricethezav/gitleaks/v6/manager"
+	"github.com/zricethezav/gitleaks/v6/options"
+	"github.com/zricethezav/gitleaks/v6/scan"
 
 
 	"github.com/go-git/go-git/v5"
 	"github.com/go-git/go-git/v5"
 	"github.com/go-git/go-git/v5/plumbing"
 	"github.com/go-git/go-git/v5/plumbing"

+ 3 - 3
hosts/gitlab.go

@@ -4,9 +4,9 @@ import (
 	"context"
 	"context"
 	"sync"
 	"sync"
 
 
-	"github.com/zricethezav/gitleaks/v5/manager"
-	"github.com/zricethezav/gitleaks/v5/options"
-	"github.com/zricethezav/gitleaks/v5/scan"
+	"github.com/zricethezav/gitleaks/v6/manager"
+	"github.com/zricethezav/gitleaks/v6/options"
+	"github.com/zricethezav/gitleaks/v6/scan"
 
 
 	log "github.com/sirupsen/logrus"
 	log "github.com/sirupsen/logrus"
 	"github.com/xanzy/go-gitlab"
 	"github.com/xanzy/go-gitlab"

+ 1 - 1
hosts/host.go

@@ -3,7 +3,7 @@ package hosts
 import (
 import (
 	"strings"
 	"strings"
 
 
-	"github.com/zricethezav/gitleaks/v5/manager"
+	"github.com/zricethezav/gitleaks/v6/manager"
 )
 )
 
 
 const (
 const (

+ 3 - 3
hosts/hosts_test.go

@@ -6,9 +6,9 @@ import (
 	"os"
 	"os"
 	"testing"
 	"testing"
 
 
-	"github.com/zricethezav/gitleaks/v5/config"
-	"github.com/zricethezav/gitleaks/v5/manager"
-	"github.com/zricethezav/gitleaks/v5/options"
+	"github.com/zricethezav/gitleaks/v6/config"
+	"github.com/zricethezav/gitleaks/v6/manager"
+	"github.com/zricethezav/gitleaks/v6/options"
 )
 )
 
 
 var (
 var (

+ 5 - 5
main.go

@@ -5,11 +5,11 @@ import (
 	"os"
 	"os"
 	"time"
 	"time"
 
 
-	"github.com/zricethezav/gitleaks/v5/config"
-	"github.com/zricethezav/gitleaks/v5/hosts"
-	"github.com/zricethezav/gitleaks/v5/manager"
-	"github.com/zricethezav/gitleaks/v5/options"
-	"github.com/zricethezav/gitleaks/v5/scan"
+	"github.com/zricethezav/gitleaks/v6/config"
+	"github.com/zricethezav/gitleaks/v6/hosts"
+	"github.com/zricethezav/gitleaks/v6/manager"
+	"github.com/zricethezav/gitleaks/v6/options"
+	"github.com/zricethezav/gitleaks/v6/scan"
 
 
 	"github.com/hako/durafmt"
 	"github.com/hako/durafmt"
 	log "github.com/sirupsen/logrus"
 	log "github.com/sirupsen/logrus"

+ 3 - 3
manager/manager.go

@@ -14,8 +14,8 @@ import (
 	"text/tabwriter"
 	"text/tabwriter"
 	"time"
 	"time"
 
 
-	"github.com/zricethezav/gitleaks/v5/config"
-	"github.com/zricethezav/gitleaks/v5/options"
+	"github.com/zricethezav/gitleaks/v6/config"
+	"github.com/zricethezav/gitleaks/v6/options"
 
 
 	"github.com/go-git/go-git/v5"
 	"github.com/go-git/go-git/v5"
 	"github.com/hako/durafmt"
 	"github.com/hako/durafmt"
@@ -252,7 +252,7 @@ func (manager *Manager) DebugOutput() {
 	w := tabwriter.NewWriter(os.Stdout, 0, 0, padding, '.', 0)
 	w := tabwriter.NewWriter(os.Stdout, 0, 0, padding, '.', 0)
 
 
 	log.Debugf("--------------------------\n")
 	log.Debugf("--------------------------\n")
-	log.Debugf("| Individual Regex Times |\n")
+	log.Debugf("| Individual Regexes Times |\n")
 	log.Debugf("--------------------------\n")
 	log.Debugf("--------------------------\n")
 	for k, v := range manager.metadata.RegexTime {
 	for k, v := range manager.metadata.RegexTime {
 		_, _ = fmt.Fprintf(w, "%s\t%s\n", k, durafmt.Parse(time.Duration(v)*time.Nanosecond))
 		_, _ = fmt.Fprintf(w, "%s\t%s\n", k, durafmt.Parse(time.Duration(v)*time.Nanosecond))

+ 2 - 2
manager/manager_test.go

@@ -3,8 +3,8 @@ package manager
 import (
 import (
 	"crypto/rand"
 	"crypto/rand"
 	"fmt"
 	"fmt"
-	"github.com/zricethezav/gitleaks/v5/config"
-	"github.com/zricethezav/gitleaks/v5/options"
+	"github.com/zricethezav/gitleaks/v6/config"
+	"github.com/zricethezav/gitleaks/v6/options"
 	"io"
 	"io"
 	"testing"
 	"testing"
 )
 )

+ 1 - 1
options/options.go

@@ -7,7 +7,7 @@ import (
 	"os/user"
 	"os/user"
 	"strings"
 	"strings"
 
 
-	"github.com/zricethezav/gitleaks/v5/version"
+	"github.com/zricethezav/gitleaks/v6/version"
 
 
 	"github.com/go-git/go-git/v5"
 	"github.com/go-git/go-git/v5"
 	"github.com/go-git/go-git/v5/plumbing/transport/http"
 	"github.com/go-git/go-git/v5/plumbing/transport/http"

+ 5 - 5
scan/repo.go

@@ -11,8 +11,8 @@ import (
 	"runtime"
 	"runtime"
 	"time"
 	"time"
 
 
-	"github.com/zricethezav/gitleaks/v5/config"
-	"github.com/zricethezav/gitleaks/v5/manager"
+	"github.com/zricethezav/gitleaks/v6/config"
+	"github.com/zricethezav/gitleaks/v6/manager"
 
 
 	"github.com/BurntSushi/toml"
 	"github.com/BurntSushi/toml"
 	"github.com/go-git/go-billy/v5"
 	"github.com/go-git/go-billy/v5"
@@ -77,11 +77,11 @@ func Run(m *manager.Manager) error {
 
 
 func runHelper(r *Repo) error {
 func runHelper(r *Repo) error {
 	// Ignore allowlisted repos
 	// Ignore allowlisted repos
-	for _, wlRepo := range r.Manager.Config.Allowlist.Repos {
-		if RegexMatched(r.Manager.Opts.RepoPath, wlRepo) {
+	for _, allowListedRepo := range r.Manager.Config.Allowlist.Repos {
+		if RegexMatched(r.Manager.Opts.RepoPath, allowListedRepo) {
 			return nil
 			return nil
 		}
 		}
-		if RegexMatched(r.Manager.Opts.Repo, wlRepo) {
+		if RegexMatched(r.Manager.Opts.Repo, allowListedRepo) {
 			return nil
 			return nil
 		}
 		}
 	}
 	}

+ 17 - 39
scan/rule.go

@@ -11,8 +11,8 @@ import (
 	"strings"
 	"strings"
 	"time"
 	"time"
 
 
-	"github.com/zricethezav/gitleaks/v5/config"
-	"github.com/zricethezav/gitleaks/v5/manager"
+	"github.com/zricethezav/gitleaks/v6/config"
+	"github.com/zricethezav/gitleaks/v6/manager"
 
 
 	fdiff "github.com/go-git/go-git/v5/plumbing/format/diff"
 	fdiff "github.com/go-git/go-git/v5/plumbing/format/diff"
 	"github.com/go-git/go-git/v5/plumbing/object"
 	"github.com/go-git/go-git/v5/plumbing/object"
@@ -58,17 +58,17 @@ func (repo *Repo) CheckRules(bundle *Bundle) {
 		start := time.Now()
 		start := time.Now()
 
 
 		// For each rule we want to check filename allowlists
 		// For each rule we want to check filename allowlists
-		if isFileNameWhiteListed(filename, rule.Allowlist) || isFilePathWhiteListed(path, rule.Allowlist) {
+		if isAllowListed(filename, rule.Allowlist.Files) || isAllowListed(path, rule.Allowlist.Paths) {
 			continue
 			continue
 		}
 		}
 
 
 		// If it has fileNameRegex and it doesnt match we continue to next rule
 		// If it has fileNameRegex and it doesnt match we continue to next rule
-		if ruleContainFileNameRegex(rule) && !RegexMatched(filename, rule.FileNameRegex) {
+		if ruleContainFileRegex(rule) && !RegexMatched(filename, rule.File) {
 			continue
 			continue
 		}
 		}
 
 
 		// If it has filePathRegex and it doesnt match we continue to next rule
 		// If it has filePathRegex and it doesnt match we continue to next rule
-		if ruleContainFilePathRegex(rule) && !RegexMatched(path, rule.FilePathRegex) {
+		if ruleContainPathRegex(rule) && !RegexMatched(path, rule.Path) {
 			continue
 			continue
 		}
 		}
 
 
@@ -112,7 +112,7 @@ func (repo *Repo) CheckRules(bundle *Bundle) {
 					offender := bundle.Content[loc[0]:loc[1]]
 					offender := bundle.Content[loc[0]:loc[1]]
 					groups := rule.Regex.FindStringSubmatch(offender)
 					groups := rule.Regex.FindStringSubmatch(offender)
 
 
-					if isOffenderWhiteListed(offender, rule.Allowlist) {
+					if isAllowListed(line, append(rule.Allowlist.Regexes, repo.config.Allowlist.Regexes...)) {
 						continue
 						continue
 					}
 					}
 
 
@@ -146,7 +146,6 @@ func (repo *Repo) CheckRules(bundle *Bundle) {
 			}
 			}
 		}
 		}
 
 
-		//	TODO should return filenameRegex if only file rule
 		repo.Manager.RecordTime(manager.RegexTime{
 		repo.Manager.RecordTime(manager.RegexTime{
 			Time:  howLong(start),
 			Time:  howLong(start),
 			Regex: rule.Regex.String(),
 			Regex: rule.Regex.String(),
@@ -339,28 +338,28 @@ func ruleContainRegex(rule config.Rule) bool {
 }
 }
 
 
 // Checks if the given rule has a file name regex
 // Checks if the given rule has a file name regex
-func ruleContainFileNameRegex(rule config.Rule) bool {
-	if rule.FileNameRegex == nil {
+func ruleContainFileRegex(rule config.Rule) bool {
+	if rule.File == nil {
 		return false
 		return false
 	}
 	}
-	if rule.FileNameRegex.String() == "" {
+	if rule.File.String() == "" {
 		return false
 		return false
 	}
 	}
 	return true
 	return true
 }
 }
 
 
 // Checks if the given rule has a file path regex
 // Checks if the given rule has a file path regex
-func ruleContainFilePathRegex(rule config.Rule) bool {
-	if rule.FilePathRegex == nil {
+func ruleContainPathRegex(rule config.Rule) bool {
+	if rule.Path == nil {
 		return false
 		return false
 	}
 	}
-	if rule.FilePathRegex.String() == "" {
+	if rule.Path.String() == "" {
 		return false
 		return false
 	}
 	}
 	return true
 	return true
 }
 }
 
 
-func isCommitWhiteListed(commitHash string, allowlistedCommits []string) bool {
+func isCommitAllowListed(commitHash string, allowlistedCommits []string) bool {
 	for _, hash := range allowlistedCommits {
 	for _, hash := range allowlistedCommits {
 		if commitHash == hash {
 		if commitHash == hash {
 			return true
 			return true
@@ -369,35 +368,14 @@ func isCommitWhiteListed(commitHash string, allowlistedCommits []string) bool {
 	return false
 	return false
 }
 }
 
 
-func isOffenderWhiteListed(offender string, allowlist []config.Allowlist) bool {
-	if len(allowlist) != 0 {
-		for _, wl := range allowlist {
-			if wl.Regex.FindString(offender) != "" {
+func isAllowListed(target string, allowList []*regexp.Regexp) bool {
+	if len(allowList) != 0 {
+		for _, re := range allowList {
+			if re.FindString(target) != "" {
 				return true
 				return true
 			}
 			}
 		}
 		}
 	}
 	}
 	return false
 	return false
-}
 
 
-func isFileNameWhiteListed(filename string, allowlist []config.Allowlist) bool {
-	if len(allowlist) != 0 {
-		for _, wl := range allowlist {
-			if RegexMatched(filename, wl.File) {
-				return true
-			}
-		}
-	}
-	return false
-}
-
-func isFilePathWhiteListed(filepath string, allowlist []config.Allowlist) bool {
-	if len(allowlist) != 0 {
-		for _, wl := range allowlist {
-			if RegexMatched(filepath, wl.Path) {
-				return true
-			}
-		}
-	}
-	return false
 }
 }

+ 4 - 3
scan/scan.go

@@ -7,7 +7,7 @@ import (
 	"sync"
 	"sync"
 	"time"
 	"time"
 
 
-	"github.com/zricethezav/gitleaks/v5/manager"
+	"github.com/zricethezav/gitleaks/v6/manager"
 
 
 	"github.com/go-git/go-git/v5"
 	"github.com/go-git/go-git/v5"
 	"github.com/go-git/go-git/v5/plumbing"
 	"github.com/go-git/go-git/v5/plumbing"
@@ -96,7 +96,7 @@ func (repo *Repo) Scan() error {
 		}
 		}
 
 
 		// Check if Commit is allowlisted
 		// Check if Commit is allowlisted
-		if isCommitWhiteListed(c.Hash.String(), repo.config.Allowlist.Commits) {
+		if isCommitAllowListed(c.Hash.String(), repo.config.Allowlist.Commits) {
 			return nil
 			return nil
 		}
 		}
 
 
@@ -140,9 +140,10 @@ func (repo *Repo) Scan() error {
 		start := time.Now()
 		start := time.Now()
 		patch, err := parent.Patch(c)
 		patch, err := parent.Patch(c)
 		if err != nil {
 		if err != nil {
-			return fmt.Errorf("could not generate Patch")
+			log.Errorf("could not generate Patch")
 		}
 		}
 		repo.Manager.RecordTime(manager.PatchTime(howLong(start)))
 		repo.Manager.RecordTime(manager.PatchTime(howLong(start)))
+
 		wg.Add(1)
 		wg.Add(1)
 		semaphore <- true
 		semaphore <- true
 		go func(c *object.Commit, patch *object.Patch) {
 		go func(c *object.Commit, patch *object.Patch) {

+ 13 - 3
scan/scan_test.go

@@ -10,9 +10,9 @@ import (
 	"sort"
 	"sort"
 	"testing"
 	"testing"
 
 
-	"github.com/zricethezav/gitleaks/v5/config"
-	"github.com/zricethezav/gitleaks/v5/manager"
-	"github.com/zricethezav/gitleaks/v5/options"
+	"github.com/zricethezav/gitleaks/v6/config"
+	"github.com/zricethezav/gitleaks/v6/manager"
+	"github.com/zricethezav/gitleaks/v6/options"
 
 
 	"github.com/sergi/go-diff/diffmatchpatch"
 	"github.com/sergi/go-diff/diffmatchpatch"
 )
 )
@@ -385,6 +385,16 @@ func TestScan(t *testing.T) {
 			},
 			},
 			wantPath: "../test_data/test_local_repo_eight.json",
 			wantPath: "../test_data/test_local_repo_eight.json",
 		},
 		},
+		{
+			description: "test local repo nine",
+			opts: options.Options{
+				RepoPath:     "../test_data/test_repos/test_repo_9",
+				Report:       "../test_data/test_local_repo_nine_aws_leak.json.got",
+				Config:       "../test_data/test_configs/large_with_global_allowlist_regex.toml",
+				ReportFormat: "json",
+			},
+			wantPath: "../test_data/test_local_repo_nine_aws_leak.json",
+		},
 	}
 	}
 
 
 	for _, test := range tests {
 	for _, test := range tests {

+ 4 - 10
test_data/test_configs/aws_key_allowlist_files.toml

@@ -2,13 +2,7 @@
     description = "AWS Manager ID"
     description = "AWS Manager ID"
     regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
     regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
     tags = ["key", "AWS"]
     tags = ["key", "AWS"]
-        [[rules.allowlist]]
-            description = "ignore md files"
-            file = '''(.*)?md$'''
-        [[rules.allowlist]]
-            description = "ignore this regex"
-            regex = '''ignore$'''
-        [[rules.allowlist]]
-            description = "ignore regex and md files"
-            regex = '''ignore$'''
-            file = '''(.*)?md$'''
+        [rules.allowlist]
+            description = "ignore sample regex md files"
+            files = ['''(.*)?md$''']
+            regexes = ['''ignore$''']

+ 2 - 2
test_data/test_configs/aws_key_allowlist_python_files.toml

@@ -2,6 +2,6 @@
     description = "AWS Manager ID"
     description = "AWS Manager ID"
     regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
     regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
     tags = ["key", "AWS"]
     tags = ["key", "AWS"]
-        [[rules.allowlist]]
+        [rules.allowlist]
             description = "ignore python files"
             description = "ignore python files"
-            file = '''(.*)?py$'''
+            files = ['''(.*)?py$''']

+ 2 - 2
test_data/test_configs/aws_key_aws_allowlisted.toml

@@ -2,6 +2,6 @@
     description = "AWS Manager ID"
     description = "AWS Manager ID"
     regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
     regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
     tags = ["key", "AWS"]
     tags = ["key", "AWS"]
-        [[rules.allowlist]]
-            regex = '''AKIAIO5FODNN7EXAMPLE.*'''
+        [rules.allowlist]
             description = "ignore aws key"
             description = "ignore aws key"
+            regexes = ['''AKIAIO5FODNN7EXAMPLE.*''']

+ 8 - 8
test_data/test_configs/large.toml

@@ -125,19 +125,19 @@ title = "gitleaks config"
 	description = "AWS Manager ID"
 	description = "AWS Manager ID"
 	regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
 	regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
 	tags = ["key", "AWS"]
 	tags = ["key", "AWS"]
-		[[rules.allowlist]]
+		[rules.allowlist]
 			description = "ignore common jenkins functions"
 			description = "ignore common jenkins functions"
-			regex = '''allowlistme'''
-			file = '''(.*)?Jenkinsfile$'''
-		[[rules.allowlist]]
-			description = "ignore common jenkins functions2"
-			regex = '''allowlistme'''
-			file = '''(.*)?Jenkinsfile$'''
+			regexes = [
+			    '''allowlistme'''
+			]
+			files = [
+			    '''(.*)?Jenkinsfile$'''
+			]
 
 
 # Global rules. This instructs gitleaks to ignore all .pem files or if a message with
 # Global rules. This instructs gitleaks to ignore all .pem files or if a message with
 [[rules]]
 [[rules]]
 	description = "Files with keys and credentials"
 	description = "Files with keys and credentials"
-    fileRegex = '''(.*?)(pem)'''
+    file = '''(.*?)(pem)'''
 
 
 [allowlist]
 [allowlist]
 	description = "File allowlists"
 	description = "File allowlists"

+ 151 - 0
test_data/test_configs/large_with_global_allowlist_regex.toml

@@ -0,0 +1,151 @@
+title = "gitleaks config"
+
+[[rules]]
+	description = "AWS Secret Key"
+	regex = '''(?i)aws(.{0,20})?(?-i)['\"][0-9a-zA-Z\/+]{40}['\"]'''
+	tags = ["key", "AWS"]
+
+[[rules]]
+	description = "AWS MWS key"
+	regex = '''amzn\.mws\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'''
+	tags = ["key", "AWS", "MWS"]
+
+[[rules]]
+	description = "Facebook Secret Key"
+	regex = '''(?i)(facebook|fb)(.{0,20})?(?-i)['\"][0-9a-f]{32}['\"]'''
+	tags = ["key", "Facebook"]
+
+[[rules]]
+	description = "Facebook Client ID"
+	regex = '''(?i)(facebook|fb)(.{0,20})?['\"][0-9]{13,17}['\"]'''
+	tags = ["key", "Facebook"]
+
+[[rules]]
+	description = "Twitter Secret Key"
+	regex = '''(?i)twitter(.{0,20})?['\"][0-9a-z]{35,44}['\"]'''
+	tags = ["key", "Twitter"]
+
+[[rules]]
+	description = "Twitter Client ID"
+	regex = '''(?i)twitter(.{0,20})?['\"][0-9a-z]{18,25}['\"]'''
+	tags = ["client", "Twitter"]
+
+[[rules]]
+	description = "Github"
+	regex = '''(?i)github(.{0,20})?(?-i)['\"][0-9a-zA-Z]{35,40}['\"]'''
+	tags = ["key", "Github"]
+
+[[rules]]
+	description = "LinkedIn Client ID"
+	regex = '''(?i)linkedin(.{0,20})?(?-i)['\"][0-9a-z]{12}['\"]'''
+	tags = ["client", "LinkedIn"]
+
+[[rules]]
+	description = "LinkedIn Secret Key"
+	regex = '''(?i)linkedin(.{0,20})?['\"][0-9a-z]{16}['\"]'''
+	tags = ["secret", "LinkedIn"]
+
+[[rules]]
+	description = "Slack"
+	regex = '''xox[baprs]-([0-9a-zA-Z]{10,48})?'''
+	tags = ["key", "Slack"]
+
+[[rules]]
+	description = "EC"
+	regex = '''-----BEGIN EC PRIVATE KEY-----'''
+	tags = ["key", "EC"]
+
+[[rules]]
+	description = "Generic API key"
+	regex = '''(?i)(api_key|apikey)(.{0,20})?['|"][0-9a-zA-Z]{32,45}['|"]'''
+	tags = ["key", "API", "generic"]
+
+[[rules]]
+	description = "Generic Secret"
+	regex = '''(?i)secret(.{0,20})?['|"][0-9a-zA-Z]{32,45}['|"]'''
+	tags = ["key", "Secret", "generic"]
+
+[[rules]]
+	description = "Google API key"
+	regex = '''AIza[0-9A-Za-z\\-_]{35}'''
+	tags = ["key", "Google"]
+
+
+[[rules]]
+	description = "Heroku API key"
+	regex = '''(?i)heroku(.{0,20})?['"][0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}['"]'''
+	tags = ["key", "Heroku"]
+
+[[rules]]
+	description = "MailChimp API key"
+	regex = '''(?i)(mailchimp|mc)(.{0,20})?['"][0-9a-f]{32}-us[0-9]{1,2}['"]'''
+	tags = ["key", "Mailchimp"]
+
+[[rules]]
+	description = "Mailgun API key"
+	regex = '''(?i)(mailgun|mg)(.{0,20})?['"][0-9a-z]{32}['"]'''
+	tags = ["key", "Mailgun"]
+
+[[rules]]
+	description = "PayPal Braintree access token"
+	regex = '''access_token\$production\$[0-9a-z]{16}\$[0-9a-f]{32}'''
+	tags = ["key", "Paypal"]
+
+[[rules]]
+	description = "Picatic API key"
+	regex = '''sk_live_[0-9a-z]{32}'''
+	tags = ["key", "Picatic"]
+
+[[rules]]
+	description = "Slack Webhook"
+	regex = '''https://hooks.slack.com/services/T[a-zA-Z0-9_]{8}/B[a-zA-Z0-9_]{8}/[a-zA-Z0-9_]{24}'''
+	tags = ["key", "slack"]
+
+[[rules]]
+	description = "Stripe API key"
+	regex = '''(?i)stripe(.{0,20})?['\"][sk|rk]_live_[0-9a-zA-Z]{24}'''
+	tags = ["key", "Stripe"]
+
+[[rules]]
+	description = "Square access token"
+	regex = '''sq0atp-[0-9A-Za-z\-_]{22}'''
+	tags = ["key", "square"]
+
+[[rules]]
+	description = "Square OAuth secret"
+	regex = '''sq0csp-[0-9A-Za-z\\-_]{43}'''
+	tags = ["key", "square"]
+
+[[rules]]
+	description = "Twilio API key"
+	regex = '''(?i)twilio(.{0,20})?['\"][0-9a-f]{32}['\"]'''
+	tags = ["key", "twilio"]
+
+[[rules]]
+	description = "AWS Manager ID"
+	regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
+	tags = ["key", "AWS"]
+		[rules.allowlist]
+			description = "ignore common jenkins functions"
+			regexes = [
+			    '''allowlistme'''
+			]
+			files = [
+			    '''(.*)?Jenkinsfile$'''
+			]
+
+[[rules]]
+	description = "Files with keys and credentials"
+    file = '''(.*?)(pem)'''
+
+# Global allowlist
+[allowlist]
+	description = "global allowlists"
+	files = [
+		'''(.*?)(jpg|gif)$''',
+		'''(.*?)(doc|pdf|bin)$'''
+		]
+	regexes = [
+	    '''(.*?)gitleaks:allow'''
+	]
+

+ 1 - 1
test_data/test_configs/regex_filename.toml

@@ -2,5 +2,5 @@
 [[rules]]
 [[rules]]
 	description = "Generic password"
 	description = "Generic password"
 	regex = '''(?i)(password|token).{0,20}?\s*[:|=]\s*?(?-i)([0-9a-zA-Z\/+]{4,40})'''
 	regex = '''(?i)(password|token).{0,20}?\s*[:|=]\s*?(?-i)([0-9a-zA-Z\/+]{4,40})'''
-	fileNameRegex = '''.*\.ya?ml'''
+	file = '''.*\.ya?ml'''
 	tags = ["key", "Yml", "Yaml"]
 	tags = ["key", "Yml", "Yaml"]

+ 1 - 1
test_data/test_configs/regex_filepath.toml

@@ -2,5 +2,5 @@
 [[rules]]
 [[rules]]
     description = "AWS Manager ID"
     description = "AWS Manager ID"
     regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
     regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
-    filePathRegex = '''config(guration)?'''
+    path = '''config(guration)?'''
     tags = ["key", "AWS"]
     tags = ["key", "AWS"]

+ 2 - 2
test_data/test_configs/regex_filepath_filename.toml

@@ -2,6 +2,6 @@
 [[rules]]
 [[rules]]
 	description = "Generic password"
 	description = "Generic password"
 	regex = '''(?i)(password|token).{0,20}?\s*[:|=]\s*?(?-i)([0-9a-zA-Z\/+]{4,40})'''
 	regex = '''(?i)(password|token).{0,20}?\s*[:|=]\s*?(?-i)([0-9a-zA-Z\/+]{4,40})'''
-	fileNameRegex = '''.*\.properties'''
-	filePathRegex = '''config(guration)?'''
+	file = '''.*\.properties'''
+	path = '''config(guration)?'''
 	tags = ["key", "Yml", "Yaml"]
 	tags = ["key", "Yml", "Yaml"]

+ 15 - 0
test_data/test_local_owner_aws_leak.json

@@ -178,5 +178,20 @@
   "date": "2020-07-25T14:41:11-04:00",
   "date": "2020-07-25T14:41:11-04:00",
   "tags": "key, AWS",
   "tags": "key, AWS",
   "operation": "addition"
   "operation": "addition"
+ },
+ {
+  "line": "    aws_access_key_id='AKIAIO5FODNN7EXAMPLE', #gitleaks:allow",
+  "lineNumber": 6,
+  "offender": "AKIAIO5FODNN7EXAMPLE",
+  "commit": "8d1fb60d2d80f0590f191ed5ace1e45ef780909a",
+  "repo": "test_repo_9",
+  "rule": "AWS Manager ID",
+  "commitMessage": "gitleaks allow secret\n",
+  "author": "Zach Rice",
+  "email": "zrice@gitlab.com",
+  "file": "server.test.py",
+  "date": "2020-08-12T13:36:20-04:00",
+  "tags": "key, AWS",
+  "operation": "addition"
  }
  }
 ]
 ]

+ 15 - 0
test_data/test_local_owner_aws_leak_allowlist_repo.json

@@ -178,5 +178,20 @@
   "date": "2020-07-25T14:41:11-04:00",
   "date": "2020-07-25T14:41:11-04:00",
   "tags": "key, AWS",
   "tags": "key, AWS",
   "operation": "addition"
   "operation": "addition"
+ },
+ {
+  "line": "    aws_access_key_id='AKIAIO5FODNN7EXAMPLE', #gitleaks:allow",
+  "lineNumber": 6,
+  "offender": "AKIAIO5FODNN7EXAMPLE",
+  "commit": "8d1fb60d2d80f0590f191ed5ace1e45ef780909a",
+  "repo": "test_repo_9",
+  "rule": "AWS Manager ID",
+  "commitMessage": "gitleaks allow secret\n",
+  "author": "Zach Rice",
+  "email": "zrice@gitlab.com",
+  "file": "server.test.py",
+  "date": "2020-08-12T13:36:20-04:00",
+  "tags": "key, AWS",
+  "operation": "addition"
  }
  }
 ]
 ]

+ 15 - 0
test_data/test_local_owner_aws_leak_depth_2.json

@@ -88,5 +88,20 @@
   "date": "2020-07-25T14:44:48-04:00",
   "date": "2020-07-25T14:44:48-04:00",
   "tags": "key, AWS",
   "tags": "key, AWS",
   "operation": "addition"
   "operation": "addition"
+ },
+ {
+  "line": "    aws_access_key_id='AKIAIO5FODNN7EXAMPLE', #gitleaks:allow",
+  "lineNumber": 6,
+  "offender": "AKIAIO5FODNN7EXAMPLE",
+  "commit": "8d1fb60d2d80f0590f191ed5ace1e45ef780909a",
+  "repo": "test_repo_9",
+  "rule": "AWS Manager ID",
+  "commitMessage": "gitleaks allow secret\n",
+  "author": "Zach Rice",
+  "email": "zrice@gitlab.com",
+  "file": "server.test.py",
+  "date": "2020-08-12T13:36:20-04:00",
+  "tags": "key, AWS",
+  "operation": "addition"
  }
  }
 ]
 ]

+ 17 - 0
test_data/test_local_repo_nine_aws_leak.json

@@ -0,0 +1,17 @@
+[
+ {
+  "line": "    aws_access_key_id='AKIAIO5FODNN7EXAMPLE',",
+  "lineNumber": 5,
+  "offender": "AKIAIO5FODNN7EXAMPLE",
+  "commit": "6557c92612d3b35979bd426d429255b3bf9fab74",
+  "repo": "test_repo_9",
+  "rule": "AWS Manager ID",
+  "commitMessage": "commit 1 with secrets\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "server.test.py",
+  "date": "2019-10-24T09:29:27-04:00",
+  "tags": "key, AWS",
+  "operation": "addition"
+ }
+]

+ 1 - 0
test_data/test_repos/test_repo_9/dotGit/COMMIT_EDITMSG

@@ -0,0 +1 @@
+gitleaks allow secret

+ 1 - 0
test_data/test_repos/test_repo_9/dotGit/HEAD

@@ -0,0 +1 @@
+ref: refs/heads/master

+ 7 - 0
test_data/test_repos/test_repo_9/dotGit/config

@@ -0,0 +1,7 @@
+[core]
+	repositoryformatversion = 0
+	filemode = true
+	bare = false
+	logallrefupdates = true
+	ignorecase = true
+	precomposeunicode = true

+ 1 - 0
test_data/test_repos/test_repo_9/dotGit/description

@@ -0,0 +1 @@
+Unnamed repository; edit this file 'description' to name the repository.

BIN=BIN
test_data/test_repos/test_repo_9/dotGit/index


+ 6 - 0
test_data/test_repos/test_repo_9/dotGit/info/exclude

@@ -0,0 +1,6 @@
+# git ls-files --others --exclude-from=.git/info/exclude
+# Lines that start with '#' are comments.
+# For a project mostly in C, the following would be a good set of
+# exclude patterns (uncomment them if you want to use them):
+# *.[oa]
+# *~

+ 3 - 0
test_data/test_repos/test_repo_9/dotGit/logs/HEAD

@@ -0,0 +1,3 @@
+0000000000000000000000000000000000000000 6557c92612d3b35979bd426d429255b3bf9fab74 zach rice <zricer@protonmail.com> 1571923767 -0400	commit (initial): commit 1 with secrets
+6557c92612d3b35979bd426d429255b3bf9fab74 d274003914c707212cbe84e3e466a00013ccb639 zach rice <zricer@protonmail.com> 1571925818 -0400	commit: comment
+d274003914c707212cbe84e3e466a00013ccb639 8d1fb60d2d80f0590f191ed5ace1e45ef780909a Zach Rice <zrice@gitlab.com> 1597253780 -0400	commit: gitleaks allow secret

+ 3 - 0
test_data/test_repos/test_repo_9/dotGit/logs/refs/heads/master

@@ -0,0 +1,3 @@
+0000000000000000000000000000000000000000 6557c92612d3b35979bd426d429255b3bf9fab74 zach rice <zricer@protonmail.com> 1571923767 -0400	commit (initial): commit 1 with secrets
+6557c92612d3b35979bd426d429255b3bf9fab74 d274003914c707212cbe84e3e466a00013ccb639 zach rice <zricer@protonmail.com> 1571925818 -0400	commit: comment
+d274003914c707212cbe84e3e466a00013ccb639 8d1fb60d2d80f0590f191ed5ace1e45ef780909a Zach Rice <zrice@gitlab.com> 1597253780 -0400	commit: gitleaks allow secret

BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/10/fa14c5ab0134436e2ae435138bf921eb477c60


BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/3a/76f3781306faf5612017bf18a4b4bdb9f927bf


BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/41/42082bcb939bbc17985a69ba748491ac6b62a5


BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/49/8b267a8c7812490d6479839c5577eaaec79d62


BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/60/c9e47f150a6b713e247e6105b77f1b961f844f


BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/61/87dbf4390fc6e28445dd3d988aefb9d1111988


BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/65/57c92612d3b35979bd426d429255b3bf9fab74


BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/6a/756416384c210ada2631f17862f5c01fffa478


BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/6c/9406b7d9320db083eca69b3f8bee9a6c7b50d4


BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/6c/bef5c370d8c3486ca85423dd70440c5e0a2aa2


BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/80/8b12c5ca4b142367932e7045d555a639fc148c


BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/80/ba94135cc378364af9d3cb2450df48e51faf2c


+ 2 - 0
test_data/test_repos/test_repo_9/dotGit/objects/8d/1fb60d2d80f0590f191ed5ace1e45ef780909a

@@ -0,0 +1,2 @@
+x•ŽM
+Â0F]çseò߀ˆgpén2�Úbk$�žÞzW<x��˲L
Œ	»VE€õ`9X§½ë(ø,Zzé}ÈnˆL,I˜P=©Ê£Ao¢C´I;Ž�6œ¥sbÅ…@ˆ¨-s6)zµ±T¸�p™Xàø©ÛœoS›)¸,'Ð>Eãmìö¸eÕF·wMþõÔ¯*t_�æ¹¼a®ÒÔ­¼G*

+ 3 - 0
test_data/test_repos/test_repo_9/dotGit/objects/9e/523225b31add24e72f2feb0b2645cfb36542dc

@@ -0,0 +1,3 @@
+xMŽO‚@Ĺ;ď§č`‘%:XjE	˘<Éş;�¤»°ł~űĚ
+z§÷`~ďMVęFţ´Ó…P�ŇnÜHZ"�•-xIL”Eca™¶Ú÷>±Ç ‘Cľă¶®«,�¬)TNŔéżČm7
+]U¨$JŻ%ů“R.Ąw¬ÓBÎś`łÖ‡q|÷űIt
vÇmôÝy_6ďŮ?¨!ć‹0Šb´ťÔW\>ε9ć	…ęr:-†Aň+a}Ć^K§N$

+ 3 - 0
test_data/test_repos/test_repo_9/dotGit/objects/a1/fd29ec14823d8bc4a8d1a2cfe35451580f5118

@@ -0,0 +1,3 @@
+xuPMOÃ0åÚüŠGV1lc.LEB`†Ä¸!Miç*a]R5!þ;N‡øââÄö{ÏÏÎ+—ãìôt§³;Ì�z-:¢ƒKzQ«º"hç–ðEcê€à°RKÂÚ\‰zí5\Œ³~À¬™¦/š7Ì~Ú‘‡ªª_h­|PMÀÆ�næªXf]Rv�†ž¨ßTVÙh²Ì$ÈmSþÔ„ñÑׂ�D+dUÎþƒæF\âˆ%­Z}VJ{²n¨ßPAæ™ä@S"�è[Èôz2›ß=Üß̧w³Éôv~5}¸�IÁ¬HLv,’�nu"C¦F¢_ÅÏ?¼dáD’гª 9¬){üeÆŠ/,Ó$axZòu ƒgm
+‹(·jÏòc•~Üøé[+Ñ9ÈÞ%.öN"s<æ¸=àVéÅŒâ¯
+~L–îïbÔãþÂY¥eMª$

BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/a4/fa2187727281aea78d7c3aaebdb4b924fc4e4d


+ 1 - 0
test_data/test_repos/test_repo_9/dotGit/objects/a5/196d1be8fb59edf8062bef36d3a602e0812139

@@ -0,0 +1 @@
+x-�±jÃ0„;ç)<¤
µSJ§B¦@¦¾Aé Û¿lI¤_¸Yú쑃·ãø¾»Þ³ÇûÇÛSƒÉ)|n­ó’ѶÔYÒäoðe”Ö&†SW©£‹–Ç­Þ5ør±:EV“‹Óûf“C�¨¹«ä…	×Ä_�Yý
.âüZu�¥÷\\œ°°ø½Tz"GdQÐÖ…íW£*)f<—¸=¬ÎâÆ‚ÅD…%Ë£ù¬ò¡û¦ùYÃÿîá›SC

BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/a5/d7b84a673458d14d9aab082183a1968c2c7492


BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/b5/8d1184a9d43a39c0d95f32453efc78581877d6


BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/c1/f3c6341548a65be1eded56b4f7cacec7fe9090


BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/c9/8e6c52cbd1f50de572ff12a3441271fccff705


BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/cb/089cd89a7d7686d284d8761201649346b5aa1c


+ 2 - 0
test_data/test_repos/test_repo_9/dotGit/objects/cb/19a50e8cdeb7011eccdb13f3b739f00d775bab

@@ -0,0 +1,2 @@
+x��Í
+1„½êS䨥MÚf+"¾J“­XØY‚OïŠ^¼íÀ00‡áéFB\™…‚à=kÄ`±%!9Jë0ÌŽè½�\ã5	;x%½ÁT4Ãéõ‰érŸÆ:}*ÝAÇþÖ³�HöƳžÛ¾TØ–¡Ô’ºÝ~�…g©7xd�r}l3´Èó.Eë”
£E•ܸLÙ…�æÇ–T%P\ëÛüÁ~óP7oq,\é

BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/d2/74003914c707212cbe84e3e466a00013ccb639


BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/e6/73bb3980f3c286291809e05f80873852bc3e9c


BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/e9/e5396f7e52aa48de485b4836ebb041cc7f7c46


+ 2 - 0
test_data/test_repos/test_repo_9/dotGit/objects/ec/17ec1939b7c3e86b7cb6c0c4de6b0818a7e75e

@@ -0,0 +1,2 @@
+x-ÍK‚0P×9E[�r§Kïà&0H
+Ȥ&ƒ…·7~–Ý]õÚ/âq¾\õ±ó!vyrµ«q‹à�Ö´0&‘¹×�&Hʉ”AHÔÏ< dÉÁD_E±e†<Y2l«‡)ÅœD-·_ù.àH¾À6…üÕOň´þ›1|6AUXk¶4�qÕ:Ç;÷xïj2k¹iBŽvÐBü

BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/f9/f2f36962fa38ca4525b99ef702fba03e3a79a7


BIN=BIN
test_data/test_repos/test_repo_9/dotGit/objects/fe/f3190e94723e78847ab9f0daaa15add4ffd4ef


+ 1 - 0
test_data/test_repos/test_repo_9/dotGit/refs/heads/master

@@ -0,0 +1 @@
+8d1fb60d2d80f0590f191ed5ace1e45ef780909a

+ 9 - 0
test_data/test_repos/test_repo_9/server.test.py

@@ -0,0 +1,9 @@
+# Do not hard code credentials
+client = boto3.client(
+    's3',
+    # Hard coded strings as credentials, not recommended.
+    aws_access_key_id='AKIAIO5FODNN7EXAMPLE', #gitleaks:allow
+    aws_secret_access_key='ABCDEF+c2L7yXeGvUyrPgYsDnWRRC1AYEXAMPLE'
+)
+
+# adding another line