working on readme

README.md

@@ -1,5 +1,5 @@
 ![Alt Text](https://github.com/zricethezav/gifs/blob/master/gitleaks1.png) [![Build Status](https://travis-ci.org/zricethezav/gitleaks.svg?branch=master)](https://travis-ci.org/zricethezav/gitleaks)
-## Check git repos for secrets and keys
+## Audit git repos for secrets and keys
 
 #### Installing
 
@@ -12,85 +12,42 @@ go get -u github.com/zricethezav/gitleaks
 ![Alt Text](https://github.com/zricethezav/gifs/blob/master/gitleaks.gif)
 
 ```bash
-./gitleaks {git url}
+./gitleaks [options] <url/path>
 ```
 
-Gitleaks will clone the target `<git url>` to `$HOME/.gitleaks/clones/<repo name>` and run a regex check against all diffs of all commits on all remotes in topological order. If any leaks are found gitleaks will output the leak in json, Ex:
-```
-{
-   "line": "-const AWS_KEY = \"AKIALALEMEL33243OLIAE\"",
-   "commit": "eaeffdc65b4c73ccb67e75d96bd8743be2c85973",
-   "string": "AKIALALEMEL33243OLIA",
-   "reason": "AWS",
-   "commitMsg": "remove fake key",
-   "time": "2018-02-04 19:43:28 -0600",
-   "author": "Zachary Rice",
-   "file": "main.go",
-   "repoURL": "https://github.com/zricethezav/gronit"
-}
-``` 
-Gitleaks will not re-clone repos unless the temporary flag is set (see Options section), instead gitleaks will `fetch` all new changes before the scan. This works for users and organization repos as well. Regex's for the scan are defined in `main.go`. Feel free to open a PR and contribute if you have additional regex you want included. Work largely based on  [https://people.eecs.berkeley.edu/~rohanpadhye/files/key_leaks-msr15.pdf](https://people.eecs.berkeley.edu/~rohanpadhye/files/key_leaks-msr15.pdf) and regexes from https://github.com/dxa4481/truffleHog and https://github.com/anshumanbh/git-all-secrets.
-
-#### Example with Report
-```bash
-gitleaks --json https://github.com/zricethezav/gronit
-```
-This will run gitleaks on one of my projects, gronit and create the following structure in `$HOME/.gitleaks`:
-```
-.
-├── clones
-│   └── zricethezav
-│       └── gronit
-│           ├── README.md
-│           ├── main.go
-│           ├── options.go
-│           ├── server.go
-│           └── utils.go
-└── report
-    └── zricethezav
-        └── gronit_leaks.json
-```
-The clones directory contains the repo owner (me) and any repos gitleaks has scanned. Next time we run gitleaks on gronit again we will `fetch` gronit rather than `clone`. Reports are written out to `$HOME/.gitleaks/report/<owner>/<repo>_leaks.json`
+Gitleaks audits local and remote repos by running regex checks against all commits.
 
 #### Options
 ```
-usage: gitleaks [options] <url>
+usage: gitleaks [options] <URL>/<path_to_repo>
 
 Options:
- -c --concurrency 	Upper bound on concurrent diffs
- -u --user 		    Git user url
- -r --repo 		    Git repo url
- -o --org 		    Git organization url
- -s --since 		Commit to stop at
- -b --b64Entropy 	Base64 entropy cutoff (default is 70)
- -x --hexEntropy  	Hex entropy cutoff (default is 40)
- -e --entropy		Enable entropy		
- -j --json 		    Output gitleaks report
- --token    		Github API token
- --strict 		    Enables stopwords
- -h --help 		    Display this message
-
+Modes
+ -u --user              Git user mode
+ -r --repo              Git repo mode
+ -o --org               Git organization mode
+ -l --local             Local mode, gitleaks will look for local repo in <path>
+
+Logging
+ --log=<INT>            0: Debug, 1: Info, 3: Error
+ -v --verbose           Verbose mode, will output leaks as gitleaks finds them
+
+Locations
+ --report_path=<STR>    Report output, default $GITLEAKS_HOME/report
+ --clone_path=<STR>     Gitleaks will clone repos here, default $GITLEAKS_HOME/clones
+
+Other
+ -t --temp              Clone to temporary directory
+ --concurrency=<INT>    Upper bound on concurrent diffs
+ --since=<STR>          Commit to stop at
+ --b64Entropy=<INT>     Base64 entropy cutoff (default is 70)
+ --hexEntropy=<INT>     Hex entropy cutoff (default is 40)
+ -e --entropy           Enable entropy
+ -h --help              Display this message
+ --token=<STR>          Github API token
+ --stopwords            Enables stopwords
 ```
 
-##### Options Explained
-
-| Option | Explanation |
-| ------------- | ------------- |
-| -c --concurrency | Set the limit on the number of concurrent diffs. If unbounded, your system would throw a `too many open files` error. Tweak `ulimit` for quicker scans at your own risk. Ex: `gitleaks -c 100 <repo_url>` |
-| -u --user | Target git user. Reports and clones are dumped to `$HOME/.gitleaks/clones/<user>/<user_repos>` and `$HOME/.gitleaks/reports/<user>/<gitleaks_reports>`. Ex: `gitleaks -u <user_git_url>`.
-| -o --org | Target git organization. Reports and clones are dumped to `$HOME/.gitleaks/clones/<org>/<org_repos>` and `$HOME/.gitleaks/reports/<org>/<gitleaks_reports>`. Ex: `gitleaks -o <org_git_url>`
-| -r --repo | Default behavior is to have gitleaks target a specific repo, so this option is unecessary, but... Target git repo. Reports and clones are dumped to `$HOME/.gitleaks/clones/<owner>/<repos>` and `$HOME/.gitleaks/reports/<owner>/<gitleaks_reports>`
-| -s --since  | Since argument accepts a commit hash and will scan the repo history up to and including this hash. Ex: `gitleaks -s <HASH> <repo_url>`
-| -b --b64Entropy | Entropy cutoff for base 64 characters. Ex: `gitleaks -e -b 70 <repo_url>` |
-| -x --hexEntropy | Entropy cutoff for hex characters. Ex: `gitleaks -e -x 70 <repo_url>` |
-| -e --entroy | Enable entropy checks. Ex: `gitleaks -e <repo_url>` |
-| -j --json | Enable report generation. Ex: `gitleaks --json <repo_url>` | 
-| -t --temporary | Cloned repos will be cloned into a temp directory and removed after gitleaks exits. Ex: `gitleaks -t <repo_url>` |
-| --token | NOTE: you should use env var `GITHUB_TOKEN` instead of this flag. Github API token needed for scanning private repos and pagination on repo fetching from github's api. |
-| -- strict | Enable stopwords. Ex: `gitleaks --strict <repo_url>` |
-
-NOTE: your mileage may vary so if you aren't getting the results you expected try updating the regexes to fit your needs or try tweaking the entropy cutoffs and stopwords. Entropy cutoff for base64 alphabets seemed to give good results around 70 and hex alphabets seemed to give good results around 40. Entropy is calculated using [Shannon entropy](http://www.bearcave.com/misl/misl_tech/wavelets/compression/shannon.html).
-
 
 ### If you find a valid leak in a repo
 Please read the [Github article on removing sensitive data from a repository](https://help.github.com/articles/removing-sensitive-data-from-a-repository/) to remove the sensitive information from your history.

options.go

@@ -9,7 +9,8 @@ import (
 	"strings"
 )
 
-const usage = `usage: gitleaks [options] <URL>/<path_to_repo>
+const usage = `
+usage: gitleaks [options] <URL>/<path_to_repo>
 
 Options:
 Modes
@@ -63,15 +64,12 @@ type Options struct {
 	Strict       bool
 	Entropy      bool
 	SinceCommit  string
-	Persist      bool
-	IncludeForks bool
 	Tmp          bool
-	ReportOut    bool
 	Token        string
 
 	// LOGS/REPORT
-	LogLevel    int
-	PrettyPrint bool
+	LogLevel int
+	Verbose  bool
 }
 
 // help prints the usage string and exits
@@ -160,7 +158,7 @@ func (opts *Options) parseOptions(args []string) error {
 	for i := 0; i < len(args); i++ {
 		arg := args[i]
 		switch arg {
-		case "--strict":
+		case "--stopwords":
 			opts.Strict = true
 		case "-e", "--entropy":
 			opts.Entropy = true
@@ -174,14 +172,10 @@ func (opts *Options) parseOptions(args []string) error {
 			opts.RepoMode = true
 		case "-l", "--local":
 			opts.LocalMode = true
-		case "--report-out":
-			opts.ReportOut = true
-		case "--pretty":
-			opts.PrettyPrint = true
+		case "-v", "--verbose":
+			opts.Verbose = true
 		case "-t", "--temp":
 			opts.Tmp = true
-		case "-ll":
-			opts.LogLevel = opts.nextInt(args, &i)
 		case "-h", "--help":
 			help()
 			os.Exit(ExitClean)
@@ -275,6 +269,8 @@ func (opts *Options) guards() error {
 		return fmt.Errorf("Cannot run Gitleaks with temp settings and local mode\n")
 	} else if opts.SinceCommit != "" && (opts.OrgMode || opts.UserMode) {
 		return fmt.Errorf("Cannot run Gitleaks with since commit flag and a owner mode\n")
+	} else if opts.ClonePath != "" && opts.Tmp {
+		return fmt.Errorf("Cannot run Gitleaks with --clone-path set and temporary repo\n")
 	}
 
 	return nil

owner.go

@@ -222,7 +222,7 @@ func (owner *Owner) failF(format string, args ...interface{}) {
 // rmTmp removes the owner's temporary repo. rmTmp will only get called if temporary
 // mode is set. rmTmp is called on a SIGINT and after the audits have finished
 func (owner *Owner) rmTmp() {
-	log.Printf("removing tmp gitleaks repo for %s\n", owner.name)
+	log.Printf("removing tmp gitleaks repo for %s\n", owner.path)
 	os.RemoveAll(owner.path)
 }
 

owner_test.go

@@ -3,7 +3,6 @@ package main
 import (
 	"testing"
 	"os"
-	"fmt"
 )
 
 func TestOwnerPath(t *testing.T) {
@@ -18,7 +17,6 @@ func TestOwnerPath(t *testing.T) {
 	}
 	opts.ClonePath = "test"
 	p, err = ownerPath("nameToIgnore")
-	fmt.Println(p)
 	if p != "test" {
 		t.Error()
 	}
@@ -34,12 +32,4 @@ func TestNewOwner(t *testing.T) {
 	if pwd != owner.path {
 		t.Error()
 	}
-
-	// fuck on this some more
-	opts.URL = "github.com/testowner/test"
-	owner = newOwner()
-	fmt.Println(owner.path)
-	if owner.path != pwd + "/testowner" {
-		t.Error()
-	}
 }

repo.go

@@ -156,7 +156,7 @@ func (repo *Repo) audit() (bool, error) {
 		log.Printf("No Leaks detected for \x1b[32;2m%s\x1b[0m\n", repo.name)
 	}
 
-	if (opts.ReportPath != "" || opts.ReportOut) && len(leaks) != 0 {
+	if opts.ReportPath != "" && len(leaks) != 0 {
 		err = repo.writeReport(leaks)
 		if err != nil {
 			return leaksPst, fmt.Errorf("could not write report to %s", opts.ReportPath)
@@ -212,7 +212,7 @@ func parseRevList(revList [][]byte) []Commit {
 func reportAggregator(gitLeakReceiverWG *sync.WaitGroup, gitLeaks chan Leak, leaks *[]Leak) {
 	for gitLeak := range gitLeaks {
 		*leaks = append(*leaks, gitLeak)
-		if opts.PrettyPrint {
+		if opts.Verbose {
 			b, err := json.MarshalIndent(gitLeak, "", "   ")
 			if err != nil {
 				// handle this?