v3 work... lots of things

.gitignore

@@ -4,10 +4,9 @@
 *.dll
 *.so
 *.dylib
+*.DS_STORE
+*.idea
+*gitleaks-ng
 
 # Test binary
-*.test
-*.out
-
-examples
-src/gitleaksConfig
+*.out

CHANGELOG.md

@@ -1,245 +0,0 @@
-CHANGELOG
-=========
-2.1.0
-- `--commit` does not iterate on commit history. Now it directly accesses the commit object.  https://github.com/zricethezav/gitleaks/pull/236
-- Fixing logging local repo path https://github.com/zricethezav/gitleaks/pull/204
-- Better README examples https://github.com/zricethezav/gitleaks/pull/202
-- Better logging on Windows https://github.com/zricethezav/gitleaks/pull/230
-- Better error messaging
-- Bunch new default rules https://github.com/zricethezav/gitleaks/pull/231 https://github.com/zricethezav/gitleaks/pull/241
-- `--repo-config` works on orgs and users now https://github.com/zricethezav/gitleaks/pull/239
-- Proper exit codes https://github.com/zricethezav/gitleaks/pull/244
-- Gitlab Token support https://github.com/zricethezav/gitleaks/pull/184
-
-2.0.0
-----
-- rules introduced in the gitleaks configurationn
-- removing `--entropy` option
-- removing `--single-search` option
-
-1.25.1
-----
-- Fixing #188
-
-1.25.0
-----
-- Pretty big refactor, see `src` directory
-- Dropping dep for go modules
-- Separating email and author
-- Readding branch support with `--branch=`
-
-1.24.0
-----
-- `--commit` now allows users to target a specific commit to audit
-- `--commit-stop` audit all commits up to and including what is specified at `--commit-stop` 
-- Updated go-git version to 4.9.1
-
-1.23.0
-----
--- `--repo-config` allows users to load configs specific to a repo target
-
-1.22.0
-----
-- context inclusion for redactions 
-- noise reduction for entropy signals
-
-1.21.0
-----
-- added support for cloning repositories using github api
-- auditing PRs now allows for whitelisting files
-
-1.20.0
-----
-- adding gitlab user and group support
-
-1.19.3
-----
-- memoizing root commit
-- only count commits (not commit + parent) for total commit number
-
-1.19.2
-----
-- fixed a bug where gitleaks was skipping the initial commit
-- commit cache now checks curr commit + parent commit hash
-- removed newlines from commit message
-
-1.19.1
-----
-- mistakenly removed default whitelist files
-
-1.19.0
-----
-- removed `--all-refs`. By default gitleaks now scans all branches, remote and local.
-- added commit memoizer to prevent duplicate commit audits
-- removed branch whitelist
-- removed branch from report as a commit is an object independent of branch
-- Better regexes for facebook, github, twitter (no more unbounded wildcards)
-
-1.18.0
-----
-- fixing slack token
-- defaulting to single thread process
-- `--max-go=` --> `--threads=`
-- guarding `--threads=` > max threads available (you aint gettin any more performance)
-
-1.17.0
-----
-- Default regex added to search: slack, telegram.
-- Default whitelisting: bin, doc, jpeg, gif
-
-1.16.1
-----
-- Fixing default ssh auth logic
-
-1.16.0
-----
-- Better commit coverage. Now iterates through each commit in git log and generates a patch with each commit's parent.
-- Removing the need for --private/-p option. Instead gitleaks will determine if the repo is private or not.
-
-
-1.15.0
-----
-- Whitelist repos use regex now
-- Whitelist repo check before clone
-
-1.14.0
-----
-- Entropy Range support in gitleaks config
-
-1.13.0
-----
-- Github PR support
-- Github has its own go file. All other services, bitbucket, gitlab, etc should follow this convention
-
-1.12.1
-----
-- Show program usage when no arguments are provided
-- Exit program after the -h or --help options are used
-
-1.12.0
-----
-- removing --csv option
-- --report option now requires .json or .csv in filename
-- adding total time to audit in logs
-
-1.11.1
-----
-- fix commit whitelist logic
-
-1.11.0
------
-- Commit depth option
-- Commit stats output
-
-1.10.0
------
-- Add entropy option
-
-1.9.0
------
-- exclude fork option
-
-1.8.0
------
-- whitelist repos
-- sample config option
-
-1.7.3
------
-- style points
-
-1.7.2
------
-- Fixing dangling goroutines, removing channel messaging
-
-1.7.1
------
-- Fixing bug where single repos were not being audited
-
-1.7.0
------
-- Exit code == 2 on error
-- Cleaning up some logs
-- Removing some unreachable code
-
-1.6.1
------
-- Recover from panic when diffing
-
-1.6.0
------
-- Default maximum goroutines spawned is number of cores your CPU run with. See benchmarks in wiki.
-- Cleanup after each repo audit for organizations rather than waiting for the entire organization to complete. Eliminates the risk of running out of disk space.
-
-
-1.5.0
------
-- Support for CSV reporting
-- Refactoring Github user/owner audits
-
-1.4.0
------
-- Support for single commit repos
-- Bumped go-git version from 4.5.0 to 4.7.0
-
-1.3.0
------
-- Target specific branch
-
-1.2.1
------
-- Check errors when generating commit patch
-
-1.2.0
------
-- Added support for providing an alternate GitHub URL to support scanning GitHub Enterprise repositories
-
-1.1.2
------
-- Added version option
-- Introduced changelog
-
-1.1.1
------
-- Fixed commit patch order
-- Updated Readme
-
-1.1.0
------
-- Fixed Twitter typo
-- Fixed sample docker command
-- Default clone option to "in-memory"
-- Added clone option for "disk"
-- Updated Makefile
-
-1.0.0
------
-- Rewrite, see Readme.md: https://github.com/zricethezav/gitleaks/releases/tag/v1.0.0
-
-0.4.0
------
-- Added support for external regexes
-
-0.3.0
------
-- Added local scan
-- Meaningful exit codes
-- Timestamped logs
-- Refactored for some maintainability
-
-0.2.0
------
-- Additionally regex checking
-- $HOME/.gitleaks/ directory for clones and reports
-- Pagination for Org/User list... no more partial repo lists
-- Persistent repos for Orgs and Users (no more re-cloning)
-- Updated README
-- Multi-staged Docker build
-- Travis CI
-
-0.1.0
------
-- full git history search
-- regex/entropy checks
-- report generation
-

CONTRIBUTING.md

@@ -1,19 +0,0 @@
-# Contributing Guidelines
-
-The gitleaks project is under [GNU General Public License v3.0](LICENSE.md) and accepts
-contributions via GitHub pull requests.
-
-## How to Contribute
-
-Open a PR. Give the PR a descriptive title. Add some comments describing what is the purpose of the PR.
-
-__BUT before you do that!__
-
-Make sure you pass this list of requirements.
-
-- You've run `go fmt`.
-- You've run `golint`.
-- You've added test cases for your changes.
-- You've updated [the changelog](CHANGELOG.md).
-- Your tests pass.
-

Dockerfile

@@ -1,12 +1,12 @@
-FROM golang:1.11.6 AS build
-WORKDIR /go/src/github.com/zricethezav/gitleaks
+FROM golang:1.13.0 AS build
+WORKDIR /go/src/github.com/zricethezav/gitleaks-ng
 COPY . .
-RUN GO111MODULE=on CGO_ENABLED=0 go build -o bin/gitleaks *.go
+RUN GO111MODULE=on CGO_ENABLED=0 go build -o bin/gitleaks-ng *.go
 
 FROM alpine:3.7
 RUN apk add --no-cache bash git openssh
-COPY --from=build /go/src/github.com/zricethezav/gitleaks/bin/* /usr/bin/
-ENTRYPOINT ["gitleaks"]
+COPY --from=build /go/src/github.com/zricethezav/gitleaks-ng/bin/* /usr/bin/
+ENTRYPOINT ["gitleaks-ng"]
 
 # How to use me :
 

LICENSE.md

@@ -1,674 +0,0 @@
-                    GNU GENERAL PUBLIC LICENSE
-                       Version 3, 29 June 2007
-
- Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
- Everyone is permitted to copy and distribute verbatim copies
- of this license document, but changing it is not allowed.
-
-                            Preamble
-
-  The GNU General Public License is a free, copyleft license for
-software and other kinds of works.
-
-  The licenses for most software and other practical works are designed
-to take away your freedom to share and change the works.  By contrast,
-the GNU General Public License is intended to guarantee your freedom to
-share and change all versions of a program--to make sure it remains free
-software for all its users.  We, the Free Software Foundation, use the
-GNU General Public License for most of our software; it applies also to
-any other work released this way by its authors.  You can apply it to
-your programs, too.
-
-  When we speak of free software, we are referring to freedom, not
-price.  Our General Public Licenses are designed to make sure that you
-have the freedom to distribute copies of free software (and charge for
-them if you wish), that you receive source code or can get it if you
-want it, that you can change the software or use pieces of it in new
-free programs, and that you know you can do these things.
-
-  To protect your rights, we need to prevent others from denying you
-these rights or asking you to surrender the rights.  Therefore, you have
-certain responsibilities if you distribute copies of the software, or if
-you modify it: responsibilities to respect the freedom of others.
-
-  For example, if you distribute copies of such a program, whether
-gratis or for a fee, you must pass on to the recipients the same
-freedoms that you received.  You must make sure that they, too, receive
-or can get the source code.  And you must show them these terms so they
-know their rights.
-
-  Developers that use the GNU GPL protect your rights with two steps:
-(1) assert copyright on the software, and (2) offer you this License
-giving you legal permission to copy, distribute and/or modify it.
-
-  For the developers' and authors' protection, the GPL clearly explains
-that there is no warranty for this free software.  For both users' and
-authors' sake, the GPL requires that modified versions be marked as
-changed, so that their problems will not be attributed erroneously to
-authors of previous versions.
-
-  Some devices are designed to deny users access to install or run
-modified versions of the software inside them, although the manufacturer
-can do so.  This is fundamentally incompatible with the aim of
-protecting users' freedom to change the software.  The systematic
-pattern of such abuse occurs in the area of products for individuals to
-use, which is precisely where it is most unacceptable.  Therefore, we
-have designed this version of the GPL to prohibit the practice for those
-products.  If such problems arise substantially in other domains, we
-stand ready to extend this provision to those domains in future versions
-of the GPL, as needed to protect the freedom of users.
-
-  Finally, every program is threatened constantly by software patents.
-States should not allow patents to restrict development and use of
-software on general-purpose computers, but in those that do, we wish to
-avoid the special danger that patents applied to a free program could
-make it effectively proprietary.  To prevent this, the GPL assures that
-patents cannot be used to render the program non-free.
-
-  The precise terms and conditions for copying, distribution and
-modification follow.
-
-                       TERMS AND CONDITIONS
-
-  0. Definitions.
-
-  "This License" refers to version 3 of the GNU General Public License.
-
-  "Copyright" also means copyright-like laws that apply to other kinds of
-works, such as semiconductor masks.
-
-  "The Program" refers to any copyrightable work licensed under this
-License.  Each licensee is addressed as "you".  "Licensees" and
-"recipients" may be individuals or organizations.
-
-  To "modify" a work means to copy from or adapt all or part of the work
-in a fashion requiring copyright permission, other than the making of an
-exact copy.  The resulting work is called a "modified version" of the
-earlier work or a work "based on" the earlier work.
-
-  A "covered work" means either the unmodified Program or a work based
-on the Program.
-
-  To "propagate" a work means to do anything with it that, without
-permission, would make you directly or secondarily liable for
-infringement under applicable copyright law, except executing it on a
-computer or modifying a private copy.  Propagation includes copying,
-distribution (with or without modification), making available to the
-public, and in some countries other activities as well.
-
-  To "convey" a work means any kind of propagation that enables other
-parties to make or receive copies.  Mere interaction with a user through
-a computer network, with no transfer of a copy, is not conveying.
-
-  An interactive user interface displays "Appropriate Legal Notices"
-to the extent that it includes a convenient and prominently visible
-feature that (1) displays an appropriate copyright notice, and (2)
-tells the user that there is no warranty for the work (except to the
-extent that warranties are provided), that licensees may convey the
-work under this License, and how to view a copy of this License.  If
-the interface presents a list of user commands or options, such as a
-menu, a prominent item in the list meets this criterion.
-
-  1. Source Code.
-
-  The "source code" for a work means the preferred form of the work
-for making modifications to it.  "Object code" means any non-source
-form of a work.
-
-  A "Standard Interface" means an interface that either is an official
-standard defined by a recognized standards body, or, in the case of
-interfaces specified for a particular programming language, one that
-is widely used among developers working in that language.
-
-  The "System Libraries" of an executable work include anything, other
-than the work as a whole, that (a) is included in the normal form of
-packaging a Major Component, but which is not part of that Major
-Component, and (b) serves only to enable use of the work with that
-Major Component, or to implement a Standard Interface for which an
-implementation is available to the public in source code form.  A
-"Major Component", in this context, means a major essential component
-(kernel, window system, and so on) of the specific operating system
-(if any) on which the executable work runs, or a compiler used to
-produce the work, or an object code interpreter used to run it.
-
-  The "Corresponding Source" for a work in object code form means all
-the source code needed to generate, install, and (for an executable
-work) run the object code and to modify the work, including scripts to
-control those activities.  However, it does not include the work's
-System Libraries, or general-purpose tools or generally available free
-programs which are used unmodified in performing those activities but
-which are not part of the work.  For example, Corresponding Source
-includes interface definition files associated with source files for
-the work, and the source code for shared libraries and dynamically
-linked subprograms that the work is specifically designed to require,
-such as by intimate data communication or control flow between those
-subprograms and other parts of the work.
-
-  The Corresponding Source need not include anything that users
-can regenerate automatically from other parts of the Corresponding
-Source.
-
-  The Corresponding Source for a work in source code form is that
-same work.
-
-  2. Basic Permissions.
-
-  All rights granted under this License are granted for the term of
-copyright on the Program, and are irrevocable provided the stated
-conditions are met.  This License explicitly affirms your unlimited
-permission to run the unmodified Program.  The output from running a
-covered work is covered by this License only if the output, given its
-content, constitutes a covered work.  This License acknowledges your
-rights of fair use or other equivalent, as provided by copyright law.
-
-  You may make, run and propagate covered works that you do not
-convey, without conditions so long as your license otherwise remains
-in force.  You may convey covered works to others for the sole purpose
-of having them make modifications exclusively for you, or provide you
-with facilities for running those works, provided that you comply with
-the terms of this License in conveying all material for which you do
-not control copyright.  Those thus making or running the covered works
-for you must do so exclusively on your behalf, under your direction
-and control, on terms that prohibit them from making any copies of
-your copyrighted material outside their relationship with you.
-
-  Conveying under any other circumstances is permitted solely under
-the conditions stated below.  Sublicensing is not allowed; section 10
-makes it unnecessary.
-
-  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
-
-  No covered work shall be deemed part of an effective technological
-measure under any applicable law fulfilling obligations under article
-11 of the WIPO copyright treaty adopted on 20 December 1996, or
-similar laws prohibiting or restricting circumvention of such
-measures.
-
-  When you convey a covered work, you waive any legal power to forbid
-circumvention of technological measures to the extent such circumvention
-is effected by exercising rights under this License with respect to
-the covered work, and you disclaim any intention to limit operation or
-modification of the work as a means of enforcing, against the work's
-users, your or third parties' legal rights to forbid circumvention of
-technological measures.
-
-  4. Conveying Verbatim Copies.
-
-  You may convey verbatim copies of the Program's source code as you
-receive it, in any medium, provided that you conspicuously and
-appropriately publish on each copy an appropriate copyright notice;
-keep intact all notices stating that this License and any
-non-permissive terms added in accord with section 7 apply to the code;
-keep intact all notices of the absence of any warranty; and give all
-recipients a copy of this License along with the Program.
-
-  You may charge any price or no price for each copy that you convey,
-and you may offer support or warranty protection for a fee.
-
-  5. Conveying Modified Source Versions.
-
-  You may convey a work based on the Program, or the modifications to
-produce it from the Program, in the form of source code under the
-terms of section 4, provided that you also meet all of these conditions:
-
-    a) The work must carry prominent notices stating that you modified
-    it, and giving a relevant date.
-
-    b) The work must carry prominent notices stating that it is
-    released under this License and any conditions added under section
-    7.  This requirement modifies the requirement in section 4 to
-    "keep intact all notices".
-
-    c) You must license the entire work, as a whole, under this
-    License to anyone who comes into possession of a copy.  This
-    License will therefore apply, along with any applicable section 7
-    additional terms, to the whole of the work, and all its parts,
-    regardless of how they are packaged.  This License gives no
-    permission to license the work in any other way, but it does not
-    invalidate such permission if you have separately received it.
-
-    d) If the work has interactive user interfaces, each must display
-    Appropriate Legal Notices; however, if the Program has interactive
-    interfaces that do not display Appropriate Legal Notices, your
-    work need not make them do so.
-
-  A compilation of a covered work with other separate and independent
-works, which are not by their nature extensions of the covered work,
-and which are not combined with it such as to form a larger program,
-in or on a volume of a storage or distribution medium, is called an
-"aggregate" if the compilation and its resulting copyright are not
-used to limit the access or legal rights of the compilation's users
-beyond what the individual works permit.  Inclusion of a covered work
-in an aggregate does not cause this License to apply to the other
-parts of the aggregate.
-
-  6. Conveying Non-Source Forms.
-
-  You may convey a covered work in object code form under the terms
-of sections 4 and 5, provided that you also convey the
-machine-readable Corresponding Source under the terms of this License,
-in one of these ways:
-
-    a) Convey the object code in, or embodied in, a physical product
-    (including a physical distribution medium), accompanied by the
-    Corresponding Source fixed on a durable physical medium
-    customarily used for software interchange.
-
-    b) Convey the object code in, or embodied in, a physical product
-    (including a physical distribution medium), accompanied by a
-    written offer, valid for at least three years and valid for as
-    long as you offer spare parts or customer support for that product
-    model, to give anyone who possesses the object code either (1) a
-    copy of the Corresponding Source for all the software in the
-    product that is covered by this License, on a durable physical
-    medium customarily used for software interchange, for a price no
-    more than your reasonable cost of physically performing this
-    conveying of source, or (2) access to copy the
-    Corresponding Source from a network server at no charge.
-
-    c) Convey individual copies of the object code with a copy of the
-    written offer to provide the Corresponding Source.  This
-    alternative is allowed only occasionally and noncommercially, and
-    only if you received the object code with such an offer, in accord
-    with subsection 6b.
-
-    d) Convey the object code by offering access from a designated
-    place (gratis or for a charge), and offer equivalent access to the
-    Corresponding Source in the same way through the same place at no
-    further charge.  You need not require recipients to copy the
-    Corresponding Source along with the object code.  If the place to
-    copy the object code is a network server, the Corresponding Source
-    may be on a different server (operated by you or a third party)
-    that supports equivalent copying facilities, provided you maintain
-    clear directions next to the object code saying where to find the
-    Corresponding Source.  Regardless of what server hosts the
-    Corresponding Source, you remain obligated to ensure that it is
-    available for as long as needed to satisfy these requirements.
-
-    e) Convey the object code using peer-to-peer transmission, provided
-    you inform other peers where the object code and Corresponding
-    Source of the work are being offered to the general public at no
-    charge under subsection 6d.
-
-  A separable portion of the object code, whose source code is excluded
-from the Corresponding Source as a System Library, need not be
-included in conveying the object code work.
-
-  A "User Product" is either (1) a "consumer product", which means any
-tangible personal property which is normally used for personal, family,
-or household purposes, or (2) anything designed or sold for incorporation
-into a dwelling.  In determining whether a product is a consumer product,
-doubtful cases shall be resolved in favor of coverage.  For a particular
-product received by a particular user, "normally used" refers to a
-typical or common use of that class of product, regardless of the status
-of the particular user or of the way in which the particular user
-actually uses, or expects or is expected to use, the product.  A product
-is a consumer product regardless of whether the product has substantial
-commercial, industrial or non-consumer uses, unless such uses represent
-the only significant mode of use of the product.
-
-  "Installation Information" for a User Product means any methods,
-procedures, authorization keys, or other information required to install
-and execute modified versions of a covered work in that User Product from
-a modified version of its Corresponding Source.  The information must
-suffice to ensure that the continued functioning of the modified object
-code is in no case prevented or interfered with solely because
-modification has been made.
-
-  If you convey an object code work under this section in, or with, or
-specifically for use in, a User Product, and the conveying occurs as
-part of a transaction in which the right of possession and use of the
-User Product is transferred to the recipient in perpetuity or for a
-fixed term (regardless of how the transaction is characterized), the
-Corresponding Source conveyed under this section must be accompanied
-by the Installation Information.  But this requirement does not apply
-if neither you nor any third party retains the ability to install
-modified object code on the User Product (for example, the work has
-been installed in ROM).
-
-  The requirement to provide Installation Information does not include a
-requirement to continue to provide support service, warranty, or updates
-for a work that has been modified or installed by the recipient, or for
-the User Product in which it has been modified or installed.  Access to a
-network may be denied when the modification itself materially and
-adversely affects the operation of the network or violates the rules and
-protocols for communication across the network.
-
-  Corresponding Source conveyed, and Installation Information provided,
-in accord with this section must be in a format that is publicly
-documented (and with an implementation available to the public in
-source code form), and must require no special password or key for
-unpacking, reading or copying.
-
-  7. Additional Terms.
-
-  "Additional permissions" are terms that supplement the terms of this
-License by making exceptions from one or more of its conditions.
-Additional permissions that are applicable to the entire Program shall
-be treated as though they were included in this License, to the extent
-that they are valid under applicable law.  If additional permissions
-apply only to part of the Program, that part may be used separately
-under those permissions, but the entire Program remains governed by
-this License without regard to the additional permissions.
-
-  When you convey a copy of a covered work, you may at your option
-remove any additional permissions from that copy, or from any part of
-it.  (Additional permissions may be written to require their own
-removal in certain cases when you modify the work.)  You may place
-additional permissions on material, added by you to a covered work,
-for which you have or can give appropriate copyright permission.
-
-  Notwithstanding any other provision of this License, for material you
-add to a covered work, you may (if authorized by the copyright holders of
-that material) supplement the terms of this License with terms:
-
-    a) Disclaiming warranty or limiting liability differently from the
-    terms of sections 15 and 16 of this License; or
-
-    b) Requiring preservation of specified reasonable legal notices or
-    author attributions in that material or in the Appropriate Legal
-    Notices displayed by works containing it; or
-
-    c) Prohibiting misrepresentation of the origin of that material, or
-    requiring that modified versions of such material be marked in
-    reasonable ways as different from the original version; or
-
-    d) Limiting the use for publicity purposes of names of licensors or
-    authors of the material; or
-
-    e) Declining to grant rights under trademark law for use of some
-    trade names, trademarks, or service marks; or
-
-    f) Requiring indemnification of licensors and authors of that
-    material by anyone who conveys the material (or modified versions of
-    it) with contractual assumptions of liability to the recipient, for
-    any liability that these contractual assumptions directly impose on
-    those licensors and authors.
-
-  All other non-permissive additional terms are considered "further
-restrictions" within the meaning of section 10.  If the Program as you
-received it, or any part of it, contains a notice stating that it is
-governed by this License along with a term that is a further
-restriction, you may remove that term.  If a license document contains
-a further restriction but permits relicensing or conveying under this
-License, you may add to a covered work material governed by the terms
-of that license document, provided that the further restriction does
-not survive such relicensing or conveying.
-
-  If you add terms to a covered work in accord with this section, you
-must place, in the relevant source files, a statement of the
-additional terms that apply to those files, or a notice indicating
-where to find the applicable terms.
-
-  Additional terms, permissive or non-permissive, may be stated in the
-form of a separately written license, or stated as exceptions;
-the above requirements apply either way.
-
-  8. Termination.
-
-  You may not propagate or modify a covered work except as expressly
-provided under this License.  Any attempt otherwise to propagate or
-modify it is void, and will automatically terminate your rights under
-this License (including any patent licenses granted under the third
-paragraph of section 11).
-
-  However, if you cease all violation of this License, then your
-license from a particular copyright holder is reinstated (a)
-provisionally, unless and until the copyright holder explicitly and
-finally terminates your license, and (b) permanently, if the copyright
-holder fails to notify you of the violation by some reasonable means
-prior to 60 days after the cessation.
-
-  Moreover, your license from a particular copyright holder is
-reinstated permanently if the copyright holder notifies you of the
-violation by some reasonable means, this is the first time you have
-received notice of violation of this License (for any work) from that
-copyright holder, and you cure the violation prior to 30 days after
-your receipt of the notice.
-
-  Termination of your rights under this section does not terminate the
-licenses of parties who have received copies or rights from you under
-this License.  If your rights have been terminated and not permanently
-reinstated, you do not qualify to receive new licenses for the same
-material under section 10.
-
-  9. Acceptance Not Required for Having Copies.
-
-  You are not required to accept this License in order to receive or
-run a copy of the Program.  Ancillary propagation of a covered work
-occurring solely as a consequence of using peer-to-peer transmission
-to receive a copy likewise does not require acceptance.  However,
-nothing other than this License grants you permission to propagate or
-modify any covered work.  These actions infringe copyright if you do
-not accept this License.  Therefore, by modifying or propagating a
-covered work, you indicate your acceptance of this License to do so.
-
-  10. Automatic Licensing of Downstream Recipients.
-
-  Each time you convey a covered work, the recipient automatically
-receives a license from the original licensors, to run, modify and
-propagate that work, subject to this License.  You are not responsible
-for enforcing compliance by third parties with this License.
-
-  An "entity transaction" is a transaction transferring control of an
-organization, or substantially all assets of one, or subdividing an
-organization, or merging organizations.  If propagation of a covered
-work results from an entity transaction, each party to that
-transaction who receives a copy of the work also receives whatever
-licenses to the work the party's predecessor in interest had or could
-give under the previous paragraph, plus a right to possession of the
-Corresponding Source of the work from the predecessor in interest, if
-the predecessor has it or can get it with reasonable efforts.
-
-  You may not impose any further restrictions on the exercise of the
-rights granted or affirmed under this License.  For example, you may
-not impose a license fee, royalty, or other charge for exercise of
-rights granted under this License, and you may not initiate litigation
-(including a cross-claim or counterclaim in a lawsuit) alleging that
-any patent claim is infringed by making, using, selling, offering for
-sale, or importing the Program or any portion of it.
-
-  11. Patents.
-
-  A "contributor" is a copyright holder who authorizes use under this
-License of the Program or a work on which the Program is based.  The
-work thus licensed is called the contributor's "contributor version".
-
-  A contributor's "essential patent claims" are all patent claims
-owned or controlled by the contributor, whether already acquired or
-hereafter acquired, that would be infringed by some manner, permitted
-by this License, of making, using, or selling its contributor version,
-but do not include claims that would be infringed only as a
-consequence of further modification of the contributor version.  For
-purposes of this definition, "control" includes the right to grant
-patent sublicenses in a manner consistent with the requirements of
-this License.
-
-  Each contributor grants you a non-exclusive, worldwide, royalty-free
-patent license under the contributor's essential patent claims, to
-make, use, sell, offer for sale, import and otherwise run, modify and
-propagate the contents of its contributor version.
-
-  In the following three paragraphs, a "patent license" is any express
-agreement or commitment, however denominated, not to enforce a patent
-(such as an express permission to practice a patent or covenant not to
-sue for patent infringement).  To "grant" such a patent license to a
-party means to make such an agreement or commitment not to enforce a
-patent against the party.
-
-  If you convey a covered work, knowingly relying on a patent license,
-and the Corresponding Source of the work is not available for anyone
-to copy, free of charge and under the terms of this License, through a
-publicly available network server or other readily accessible means,
-then you must either (1) cause the Corresponding Source to be so
-available, or (2) arrange to deprive yourself of the benefit of the
-patent license for this particular work, or (3) arrange, in a manner
-consistent with the requirements of this License, to extend the patent
-license to downstream recipients.  "Knowingly relying" means you have
-actual knowledge that, but for the patent license, your conveying the
-covered work in a country, or your recipient's use of the covered work
-in a country, would infringe one or more identifiable patents in that
-country that you have reason to believe are valid.
-
-  If, pursuant to or in connection with a single transaction or
-arrangement, you convey, or propagate by procuring conveyance of, a
-covered work, and grant a patent license to some of the parties
-receiving the covered work authorizing them to use, propagate, modify
-or convey a specific copy of the covered work, then the patent license
-you grant is automatically extended to all recipients of the covered
-work and works based on it.
-
-  A patent license is "discriminatory" if it does not include within
-the scope of its coverage, prohibits the exercise of, or is
-conditioned on the non-exercise of one or more of the rights that are
-specifically granted under this License.  You may not convey a covered
-work if you are a party to an arrangement with a third party that is
-in the business of distributing software, under which you make payment
-to the third party based on the extent of your activity of conveying
-the work, and under which the third party grants, to any of the
-parties who would receive the covered work from you, a discriminatory
-patent license (a) in connection with copies of the covered work
-conveyed by you (or copies made from those copies), or (b) primarily
-for and in connection with specific products or compilations that
-contain the covered work, unless you entered into that arrangement,
-or that patent license was granted, prior to 28 March 2007.
-
-  Nothing in this License shall be construed as excluding or limiting
-any implied license or other defenses to infringement that may
-otherwise be available to you under applicable patent law.
-
-  12. No Surrender of Others' Freedom.
-
-  If conditions are imposed on you (whether by court order, agreement or
-otherwise) that contradict the conditions of this License, they do not
-excuse you from the conditions of this License.  If you cannot convey a
-covered work so as to satisfy simultaneously your obligations under this
-License and any other pertinent obligations, then as a consequence you may
-not convey it at all.  For example, if you agree to terms that obligate you
-to collect a royalty for further conveying from those to whom you convey
-the Program, the only way you could satisfy both those terms and this
-License would be to refrain entirely from conveying the Program.
-
-  13. Use with the GNU Affero General Public License.
-
-  Notwithstanding any other provision of this License, you have
-permission to link or combine any covered work with a work licensed
-under version 3 of the GNU Affero General Public License into a single
-combined work, and to convey the resulting work.  The terms of this
-License will continue to apply to the part which is the covered work,
-but the special requirements of the GNU Affero General Public License,
-section 13, concerning interaction through a network will apply to the
-combination as such.
-
-  14. Revised Versions of this License.
-
-  The Free Software Foundation may publish revised and/or new versions of
-the GNU General Public License from time to time.  Such new versions will
-be similar in spirit to the present version, but may differ in detail to
-address new problems or concerns.
-
-  Each version is given a distinguishing version number.  If the
-Program specifies that a certain numbered version of the GNU General
-Public License "or any later version" applies to it, you have the
-option of following the terms and conditions either of that numbered
-version or of any later version published by the Free Software
-Foundation.  If the Program does not specify a version number of the
-GNU General Public License, you may choose any version ever published
-by the Free Software Foundation.
-
-  If the Program specifies that a proxy can decide which future
-versions of the GNU General Public License can be used, that proxy's
-public statement of acceptance of a version permanently authorizes you
-to choose that version for the Program.
-
-  Later license versions may give you additional or different
-permissions.  However, no additional obligations are imposed on any
-author or copyright holder as a result of your choosing to follow a
-later version.
-
-  15. Disclaimer of Warranty.
-
-  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
-APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
-HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
-OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
-THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
-IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
-ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
-
-  16. Limitation of Liability.
-
-  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
-WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
-THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
-GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
-USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
-DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
-PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
-EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
-SUCH DAMAGES.
-
-  17. Interpretation of Sections 15 and 16.
-
-  If the disclaimer of warranty and limitation of liability provided
-above cannot be given local legal effect according to their terms,
-reviewing courts shall apply local law that most closely approximates
-an absolute waiver of all civil liability in connection with the
-Program, unless a warranty or assumption of liability accompanies a
-copy of the Program in return for a fee.
-
-                     END OF TERMS AND CONDITIONS
-
-            How to Apply These Terms to Your New Programs
-
-  If you develop a new program, and you want it to be of the greatest
-possible use to the public, the best way to achieve this is to make it
-free software which everyone can redistribute and change under these terms.
-
-  To do so, attach the following notices to the program.  It is safest
-to attach them to the start of each source file to most effectively
-state the exclusion of warranty; and each file should have at least
-the "copyright" line and a pointer to where the full notice is found.
-
-    <one line to give the program's name and a brief idea of what it does.>
-    Copyright (C) <year>  <name of author>
-
-    This program is free software: you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation, either version 3 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License
-    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-Also add information on how to contact you by electronic and paper mail.
-
-  If the program does terminal interaction, make it output a short
-notice like this when it starts in an interactive mode:
-
-    <program>  Copyright (C) <year>  <name of author>
-    This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
-    This is free software, and you are welcome to redistribute it
-    under certain conditions; type `show c' for details.
-
-The hypothetical commands `show w' and `show c' should show the appropriate
-parts of the General Public License.  Of course, your program's commands
-might be different; for a GUI interface, you would use an "about box".
-
-  You should also get your employer (if you work as a programmer) or school,
-if any, to sign a "copyright disclaimer" for the program, if necessary.
-For more information on this, and how to apply and follow the GNU GPL, see
-<http://www.gnu.org/licenses/>.
-
-  The GNU General Public License does not permit incorporating your program
-into proprietary programs.  If your program is a subroutine library, you
-may consider it more useful to permit linking proprietary applications with
-the library.  If this is what you want to do, use the GNU Lesser General
-Public License instead of this License.  But first, please read
-<http://www.gnu.org/philosophy/why-not-lgpl.html>.

Makefile

@@ -1,16 +1,28 @@
-.PHONY: test build-all deploy
+.PHONY: test test-cover build release-builds
+
+VERSION := `git fetch --tags && git tag | sort -V | tail -1`
+PKG=github.com/zricethezav/gitleaks-ng
+LDFLAGS=-ldflags "-X=github.com/zricethezav/gitleaks-ng/version.Version=$(VERSION)"
+COVER=--cover --coverprofile=cover.out
+
+test-cover:
+	go test ./... --race $(COVER) $(PKG) -v
+	go tool cover -html=cover.out
 
 test:
 	go get golang.org/x/lint/golint
 	go fmt
 	golint
-	go test --race --cover github.com/zricethezav/gitleaks/src -v
-deploy:
-	@echo "$(DOCKER_PASSWORD)" | docker login -u "$(DOCKER_USERNAME)" --password-stdin
-	docker build -f Dockerfile -t $(REPO):$(TAG) .
-	echo "Pushing $(REPO):$(COMMIT) $(REPO):$(TAG)"
-	docker push $(REPO)
-build-all:
+	go test ./... --race $(PKG) -v
+
+test-integration:
+	go test github.com/zricethezav/gitleaks-ng/hosts -v -integration
+
+build:
+	go mod tidy
+	go build $(LDFLAGS)
+
+release-builds:
 	rm -rf build
 	mkdir build
 	env GOOS="windows" GOARCH="amd64" go build -o "build/gitleaks-windows-amd64.exe"
@@ -20,3 +32,5 @@ build-all:
 	env GOOS="linux" GOARCH="mips" go build -o "build/gitleaks-linux-mips"
 	env GOOS="linux" GOARCH="mips" go build -o "build/gitleaks-linux-mips"
 	env GOOS="darwin" GOARCH="amd64" go build -o "build/gitleaks-darwin-amd64"
+
+

README.md

@@ -5,31 +5,20 @@ Gitleaks
       <a href="https://travis-ci.org/zricethezav/gitleaks"><img alt="Travis" src="https://img.shields.io/travis/zricethezav/gitleaks/master.svg?style=flat-square"></a>
 </p>
 
-Audit git repos for secrets. Gitleaks provides a way for you to find unencrypted secrets and other unwanted data types in git source code repositories. As part of it's core functionality, it provides:
+Audit git repos for secrets. Gitleaks provides a way for you to find unencrypted secrets and other unwanted data types in git repositories. As part of it's core functionality, it provides:
 
-* Github and Gitlab support including support for bulk organization and repository owner (user) repository scans, as well as pull request scanning for use in common CI workflows.
+* Audits for uncommitted changes
+* Github and Gitlab support including support for bulk organization and repository owner (user) repository scans, as well as pull/merge request scanning for use in common CI workflows.
 * Support for private repository scans, and repositories that require key based authentication
-* Output in CSV and JSON formats for consumption in other reporting tools and frameworks
+* Output in JSON formats for consumption in other reporting tools and frameworks
 * Externalised configuration for environment specific customisation including regex rules
-* Customizable repository name, file type, commit ID, branch name and regex whitelisting to reduce false positives
 * High performance through the use of src-d's [go-git](https://github.com/src-d/go-git) framework
 
-It has been successfully used in a number of different scenarios, including:
-
-* Adhoc scans of local and remote repositories by filesystem path or clone URL
-* Automated scans of github users and organizations (Both public and enterprise platforms)
-* As part of a CICD workflow to identify secrets before they make it deeper into your codebase
-* As part of a wider secrets auditing automation capability for git data in large environments
-
-
-## Example execution
-
-
 <p align="left">
     <img src="https://cdn.rawgit.com/zricethezav/5bf8259b7fea0170becffc06b8588edb/raw/f762769fe20ef3669bff34612b1bede6457631e6/termtosvg_je8bp82s.svg">
 </p>
 
-## Installation
+## Getting Started
 
 Written in Go, gitleaks is available in binary form for many popular platforms and OS types from the [releases page](https://github.com/zricethezav/gitleaks/releases). Alternatively, executed via Docker or it can be installed using Go directly, as per the below;
 
@@ -51,43 +40,39 @@ docker pull zricethezav/gitleaks
 go get -u github.com/zricethezav/gitleaks
 ```
 
-## Usage and Options
+## Usage
 
 gitleaks has a wide range of configuration options that can be adjusted at runtime or via a configuration file based on your specific requirements.
 
 ```
-Usage:
-  gitleaks [OPTIONS]
-
 Application Options:
-  -r, --repo=           Repo url to audit
-      --github-user=    Github user to audit
-      --github-org=     Github organization to audit
-      --github-url=     GitHub API Base URL, use for GitHub Enterprise. Example: https://github.example.com/api/v3/ (default: https://api.github.com/)
-      --github-pr=      Github PR url to audit. This does not clone the repo. GITHUB_TOKEN must be set
-      --gitlab-user=    GitLab user ID to audit
-      --gitlab-org=     GitLab group ID to audit
-      --commit-stop=    sha of commit to stop at
-      --commit=         sha of commit to audit
-      --depth=          maximum commit depth
-      --repo-path=      Path to repo
-      --owner-path=     Path to owner directory (repos discovered)
-      --threads=        Maximum number of threads gitleaks spawns
-      --disk            Clones repo(s) to disk
-      --config=         path to gitleaks config
-      --ssh-key=        path to ssh key
-      --exclude-forks   exclude forks for organization/user audits
-      --repo-config     Load config from target repo. Config file must be ".gitleaks.toml"
-      --branch=         Branch to audit
-  -l, --log=            log level
-  -v, --verbose         Show verbose output from gitleaks audit
-      --report=         path to write report file
-      --redact          redact secrets from log messages and report
-      --version         version number
-      --sample-config   prints a sample config file
+  -v, --verbose       Show verbose output from audit
+  -r, --repo=         Target repository
+      --config=       config path
+      --disk          Clones repo(s) to disk
+      --timeout=      Timeout (s)
+      --username=     Username for git repo
+      --password=     Password for git repo
+      --access-token= Access token for git repo
+      --Commit=       sha of Commit to audit
+      --threads=      Maximum number of threads gitleaks spawns
+      --ssh-key=      path to ssh key used for auth
+      --uncommitted   run gitleaks on uncommitted code
+      --repo-path=    Path to repo
+      --owner-path=   Path to owner directory (repos discovered)
+      --branch=       Branch to audit
+      --report=       path to write json leaks file
+      --redact        redact secrets from log messages and leaks
+      --version       version number
+      --debug         log debug messages
+      --host=         git hosting service like gitlab or github. Supported hosts include: Github, Gitlab
+      --org=          organization to audit
+      --user=         user to audit
+      --pr=           pull/merge request url
 
 Help Options:
-  -h, --help           Show this help message
+  -h, --help          Show this help message
+
 ```
 
 ### Docker usage examples
@@ -106,65 +91,15 @@ docker run --rm --name=gitleaks zricethezav/gitleaks -v -r https://github.com/zr
 docker run --rm --name=gitleaks -v /tmp/:/code/ zricethezav/gitleaks -v --repo-path=/code/gitleaks
 ```
 
-###### Specific Github Pull request
-
-You need GitHub token with `repo` access. [How create token](https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line).
-
-```bash
-  export GITHUB_TOKEN=""
-
-docker run --rm --name=gitleaks \
-    -e GITHUB_TOKEN=${GITHUB_TOKEN} \
-    zricethezav/gitleaks \
-        --github-pr=https://github.com/owner/repo/pull/9000
-```
-
-###### Private repository
-
-You need private SSH key associated with user which have pull access to private repo.
-
-```bash
-export SSH_KEY_DIR=$(echo $HOME)/.ssh
-export SSH_KEY_NAME=id_rsa
-export REPO="git@github.com:zricethezav/gitleaks.git"
-
-docker run --rm --name=gitleaks \
-    --mount type=bind,src=${SSH_KEY_DIR},dst=/root/.ssh,readonly \
-    zricethezav/gitleaks \
-        --ssh-key=/root/.ssh/${SSH_KEY_NAME} \
-        --verbose \
-        --repo=${REPO}
-```
-
-###### Specific Github organization with private repos
-
-You need GitHub token with `repo` access. [How create token](https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line).
-
-```bash
-  export GITHUB_TOKEN=""
-export ORG="github_org_name" # "git" in "https://github.com/git/", for instance.
-
-docker run --rm --name=gitleaks \
-    -e GITHUB_TOKEN=${GITHUB_TOKEN} \
-    zricethezav/gitleaks \
-        --verbose \
-        --disk \
-        --threads=$(($(nproc --all) - 1)) \
-        --github-org=${ORG}
-```
-
 By default repos cloned to memory. Using `--disk` for clone to disk or you can quickly out of memory.
 
-For speed up analyze operation using `--threads` parameter, which set to `ALL - 1` treads at your instance CPU.
+For speed up analyze operation using `--threads` parameter, which set to `ALL - 1` threads at your instance CPU.
 
 
 ## Exit Codes
 
 Gitleaks provides consistent exist codes to assist in automation workflows such as CICD platforms and bulk scanning.
 
-These can be effectively used in conjunction with the report output file to detect and return meaningful data back to the user or external system about if leaks have been detected, and where they reside.
-
-The code return codes are:
 
 ```
 0: no leaks
@@ -172,21 +107,7 @@ The code return codes are:
 2: error encountered
 ```
 
-## Additional information
-
-* Additional documentation about how gitleaks functions can be found on the [wiki page](https://github.com/zricethezav/gitleaks/wiki)
-* The below links detail the various approaches to remediating unwanted data in git repos
-    * [Removing sensitive data from a repository (github.com)](https://help.github.com/articles/removing-sensitive-data-from-a-repository/)
-    * [Removing sensitive files from commit history (atlassian.com)](https://community.atlassian.com/t5/Bitbucket-questions/Remove-sensitive-files-from-commit-history/qaq-p/243807)
-    * [Rewrite git history with the BFG (theguardian.com)](https://www.theguardian.com/info/developer-blog/2013/apr/29/rewrite-git-history-with-the-bfg)
-* [Auditing Bitbucket Server Data for Credentials in AWS (sourcedgroup.com)](https://www.sourcedgroup.com/blog/auditing-bitbucket-server-data-credentials-in-aws)
-
-    This blog post details how gitleaks was used to audit data in Atlassian Bitbucket server when hosted on AWS and visualise the results in a compliance dashboard using Splunk.
-
-* How does gitleaks differ to Github token scanning?
-    * [Github recently announced](https://blog.github.com/2018-10-16-future-of-software/#github-token-scanning-for-public-repositories-public-beta) a new capability to their cloud platform that detects exposed credentials for a number of common services and platforms and automatically notifies the provider for revocation or similar action. Gitleaks provides a similar detection capability for non-Github cloud users, in which repositories can be easily audited and results provided in a number of formats.
-
-## Give Thanks
+### Give Thanks
 
 If using gitleaks has made you job easier consider donating to one of [Sam](https://www.flickr.com/photos/146541520@N08/albums/72157710121716312)'s favorite places, the Japan House on the University of Illinois at Urbana-Champaign's campus: https://japanhouse.illinois.edu/make-a-gift
 

audit/audit.go

@@ -0,0 +1,55 @@
+package audit
+
+import (
+	"fmt"
+	"github.com/zricethezav/gitleaks-ng/manager"
+	"io/ioutil"
+	"path"
+)
+
+func Run(m *manager.Manager) error {
+	if m.Opts.OwnerPath != "" {
+		files, err := ioutil.ReadDir(m.Opts.OwnerPath)
+		if err != nil {
+			return err
+		}
+		for _, f := range files {
+			if !f.IsDir() {
+				continue
+			}
+			m.Opts.RepoPath = fmt.Sprintf("%s/%s",m.Opts.OwnerPath, f.Name())
+			if err := runHelper(NewRepo(m)); err != nil {
+				// TODO or send to errchan?
+				return err
+			}
+		}
+		return nil
+	}
+
+	return runHelper(NewRepo(m))
+}
+
+func runHelper(r *Repo) error {
+	// Check if gitleaks will perform a local audit.
+	if r.Manager.Opts.OpenLocal() {
+		r.Name = path.Base(r.Manager.Opts.RepoPath)
+		if err := r.Open(); err != nil {
+			return err
+		}
+
+		// Check if we are checking uncommitted files. This is the default behavior
+		// for a "$gitleaks" command with no options set
+		if r.Manager.Opts.CheckUncommitted() {
+			if err := r.AuditLocal(); err != nil {
+				return err
+			}
+			return nil
+		}
+	} else {
+		if err := r.Clone(); err != nil {
+			return err
+		}
+	}
+	return r.Audit()
+}
+

audit/audit_test.go

@@ -0,0 +1,313 @@
+package audit
+
+import (
+	"fmt"
+	"github.com/sergi/go-diff/diffmatchpatch"
+	"github.com/zricethezav/gitleaks-ng/config"
+	"github.com/zricethezav/gitleaks-ng/manager"
+	"github.com/zricethezav/gitleaks-ng/options"
+	"io/ioutil"
+	"os"
+	"runtime"
+	"strings"
+	"testing"
+)
+
+const testRepoBase = "../test_data/test_repos/"
+
+func TestAudit(t *testing.T) {
+	moveDotGit("dotGit", ".git")
+	defer moveDotGit(".git", "dotGit")
+	tests := []struct {
+		description string
+		opts        options.Options
+		wantPath    string
+		wantErr     error
+		emptyRepo   bool
+		wantEmpty   bool
+	}{
+		{
+			description: "test local repo one aws leak",
+			opts: options.Options{
+				RepoPath: "../test_data/test_repos/test_repo_1",
+				Report:   "../test_data/test_local_repo_one_aws_leak.json.got",
+
+			},
+			wantPath: "../test_data/test_local_repo_one_aws_leak.json",
+		},
+		{
+			description: "test local repo one aws leak threaded",
+			opts: options.Options{
+				Threads:  runtime.GOMAXPROCS(0),
+				RepoPath: "../test_data/test_repos/test_repo_1",
+				Report:   "../test_data/test_local_repo_one_aws_leak.json.got",
+			},
+			wantPath: "../test_data/test_local_repo_one_aws_leak.json",
+		},
+		{
+			description: "test non existent repo",
+			opts: options.Options{
+				RepoPath: "../test_data/test_repos/no_repo_here",
+			},
+			emptyRepo: true,
+		},
+		{
+			description: "test local repo one aws leak whitelisted",
+			opts: options.Options{
+				RepoPath: "../test_data/test_repos/test_repo_1",
+				Config:   "../test_data/test_configs/aws_key_whitelist_python_files.toml",
+			},
+			wantEmpty: true,
+		},
+		{
+			description: "test local repo two leaks",
+			opts: options.Options{
+				RepoPath: "../test_data/test_repos/test_repo_2",
+				Report:   "../test_data/test_local_repo_two_leaks.json.got",
+			},
+			wantPath: "../test_data/test_local_repo_two_leaks.json",
+		},
+		{
+			description: "test local repo two leaks globally whitelisted",
+			opts: options.Options{
+				RepoPath: "../test_data/test_repos/test_repo_2",
+				Config:   "../test_data/test_configs/aws_key_global_whitelist_file.toml",
+			},
+			wantEmpty: true,
+		},
+		{
+			description: "test local repo two leaks whitelisted",
+			opts: options.Options{
+				RepoPath: "../test_data/test_repos/test_repo_2",
+				Config:   "../test_data/test_configs/aws_key_whitelist_files.toml",
+			},
+			wantEmpty: true,
+		},
+		{
+			description: "test local repo three leaks dev branch",
+			opts: options.Options{
+				RepoPath: "../test_data/test_repos/test_repo_3",
+				Report:   "../test_data/test_local_repo_three_leaks.json.got",
+				Config:   "../test_data/test_configs/aws_key.toml",
+				Branch:   "dev",
+			},
+			wantPath: "../test_data/test_local_repo_three_leaks.json",
+		},
+		{
+			description: "test local repo branch does not exist",
+			opts: options.Options{
+				RepoPath: "../test_data/test_repos/test_repo_3",
+				Branch:   "nobranch",
+			},
+			wantEmpty: true,
+		},
+		{
+			description: "test local repo one aws leak single commit",
+			opts: options.Options{
+				RepoPath: "../test_data/test_repos/test_repo_1",
+				Report:   "../test_data/test_local_repo_one_aws_leak_commit.json.got",
+				Commit:   "6557c92612d3b35979bd426d429255b3bf9fab74",
+			},
+			wantPath: "../test_data/test_local_repo_one_aws_leak_commit.json",
+		},
+		{
+			description: "test local repo one aws leak AND leak on python files",
+			opts: options.Options{
+				RepoPath: "../test_data/test_repos/test_repo_1",
+				Report:   "../test_data/test_local_repo_one_aws_leak_and_file_leak.json.got",
+				Config:   "../test_data/test_configs/aws_key_file_regex.toml",
+			},
+			wantPath: "../test_data/test_local_repo_one_aws_leak_and_file_leak.json",
+		},
+		{
+			description: "test owner path",
+			opts: options.Options{
+				OwnerPath: "../test_data/test_repos/",
+				Report:   "../test_data/test_local_owner_aws_leak.json.got",
+
+			},
+			wantPath: "../test_data/test_local_owner_aws_leak.json",
+		},
+		{
+			description: "test entropy",
+			opts: options.Options{
+				RepoPath: "../test_data/test_repos/test_repo_1",
+				Report:   "../test_data/test_entropy.json.got",
+				Config:   "../test_data/test_configs/entropy.toml",
+			},
+			wantPath: "../test_data/test_entropy.json",
+		},
+	}
+
+	for _, test := range tests {
+		fmt.Println(test.description)
+		cfg, err := config.NewConfig(test.opts)
+		if err != nil {
+			t.Error(err)
+		}
+
+		m, err := manager.NewManager(test.opts, cfg)
+		if err != nil {
+			t.Error(err)
+		}
+
+		err = Run(m)
+
+		if test.wantErr != nil {
+			if err == nil {
+				t.Errorf("did not receive wantErr: %v", test.wantErr)
+			}
+			if err.Error() != test.wantErr.Error() {
+				t.Errorf("wantErr does not equal err received: %v", err.Error())
+			}
+			continue
+		}
+		// time.Sleep(time.Millisecond * 50)
+
+		err = m.Report()
+
+		if test.wantEmpty {
+			if len(m.GetLeaks()) != 0 {
+				t.Errorf("wanted no leaks but got some instead: %+v", m.GetLeaks())
+			}
+			continue
+		}
+
+		if test.wantPath != "" {
+			err := fileCheck(test.wantPath, test.opts.Report)
+			if err != nil {
+				t.Error(err)
+			}
+		}
+	}
+}
+
+func TestAuditUncommited(t *testing.T) {
+	moveDotGit("dotGit", ".git")
+	defer moveDotGit(".git", "dotGit")
+	tests := []struct {
+		description  string
+		opts         options.Options
+		wantPath     string
+		wantErr      error
+		emptyRepo    bool
+		wantEmpty    bool
+		fileToChange string
+		addition     string
+	}{
+		{
+			description: "test audit local one leak",
+			opts: options.Options{
+				RepoPath:   "../test_data/test_repos/test_repo_1",
+				Report:     "../test_data/test_local_repo_one_aws_leak_uncommitted.json.got",
+				Uncommited: true,
+			},
+			wantPath:     "../test_data/test_local_repo_one_aws_leak_uncommitted.json",
+			fileToChange: "server.test.py",
+			addition:     " aws_access_key_id='AKIAIO5FODNN7DXAMPLE'\n\n",
+		},
+		{
+			description: "test audit local no leak",
+			opts: options.Options{
+				RepoPath:   "../test_data/test_repos/test_repo_1",
+				Uncommited: true,
+			},
+			wantEmpty:    true,
+			fileToChange: "server.test.py",
+			addition:     "nothing bad",
+		},
+	}
+	for _, test := range tests {
+		fmt.Println(test.description)
+		old, err := ioutil.ReadFile(fmt.Sprintf("%s/%s", test.opts.RepoPath, test.fileToChange))
+		if err != nil {
+			t.Error(err)
+		}
+		altered, err := os.OpenFile(fmt.Sprintf("%s/%s", test.opts.RepoPath, test.fileToChange),
+			os.O_WRONLY|os.O_APPEND, 0644)
+		if err != nil {
+			t.Error(err)
+		}
+
+		_, err = altered.WriteString(test.addition)
+		if err != nil {
+			t.Error(err)
+		}
+
+		cfg, err := config.NewConfig(test.opts)
+		if err != nil {
+			t.Error(err)
+		}
+		m, err := manager.NewManager(test.opts, cfg)
+		if err != nil {
+			t.Error(err)
+		}
+
+		if err := Run(m); err != nil {
+			t.Error(err)
+		}
+
+		if err := m.Report(); err != nil {
+			t.Error(err)
+		}
+
+		err = ioutil.WriteFile(fmt.Sprintf("%s/%s", test.opts.RepoPath, test.fileToChange), old, 0)
+		if err != nil {
+			t.Error(err)
+		}
+
+		if test.wantEmpty {
+			continue
+		}
+
+		if test.wantPath != "" {
+			err := fileCheck(test.wantPath, test.opts.Report)
+			if err != nil {
+				t.Error(err)
+			}
+		}
+	}
+
+}
+
+func fileCheck(wantPath, gotPath string) error {
+	want, err := ioutil.ReadFile(wantPath)
+	if err != nil {
+		return err
+	}
+
+	got, err := ioutil.ReadFile(gotPath)
+	if err != nil {
+		return err
+	}
+
+	if strings.Trim(string(want), "\n") != strings.Trim(string(got), "\n") {
+		dmp := diffmatchpatch.New()
+		diffs := dmp.DiffMain(string(want), string(got), false)
+		return fmt.Errorf("does not equal: %s\n", dmp.DiffPrettyText(diffs))
+	} else {
+		if err := os.Remove(gotPath); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func moveDotGit(from, to string) error {
+	repoDirs, err := ioutil.ReadDir("../test_data/test_repos")
+	if err != nil {
+		return err
+	}
+	for _, dir := range repoDirs {
+		if !dir.IsDir() {
+			continue
+		}
+		err = os.Rename(fmt.Sprintf("%s/%s/%s", testRepoBase, dir.Name(), from),
+			fmt.Sprintf("%s/%s/%s", testRepoBase, dir.Name(), to))
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+

audit/repo.go

@@ -0,0 +1,307 @@
+package audit
+
+import (
+	"bytes"
+	"crypto/md5"
+	"fmt"
+	"github.com/BurntSushi/toml"
+	"github.com/sergi/go-diff/diffmatchpatch"
+	log "github.com/sirupsen/logrus"
+	"github.com/zricethezav/gitleaks-ng/config"
+	"github.com/zricethezav/gitleaks-ng/manager"
+	"gopkg.in/src-d/go-billy.v4"
+	"gopkg.in/src-d/go-git.v4"
+	"gopkg.in/src-d/go-git.v4/plumbing"
+	"gopkg.in/src-d/go-git.v4/plumbing/object"
+	"gopkg.in/src-d/go-git.v4/plumbing/storer"
+	"gopkg.in/src-d/go-git.v4/storage/memory"
+	"io"
+	"os"
+	"sync"
+	"time"
+)
+
+// Repo wraps a *git.Repository object in addition to a manager object and the name of the repo.
+// Commits are inspected from the *git.Repository object. If a commit is found then we send it
+// via the manager LeakChan where the manager receives and keeps track of all leaks.
+type Repo struct {
+	*git.Repository
+
+	// AlternativeConfig is used when the --repo-config option is set.
+	// This allows users to load up configs specific to their repos.
+	// Imagine the scenario where you are doing an audit of a large organization
+	// and you want certain repos to look for specific rules. If those specific repos
+	// have a gitleaks.toml or .gitleaks.toml config then those configs will be used specifically
+	// for those repo audits.
+	AlternativeConfig config.Config
+	config config.Config
+
+	Name    string
+	Manager *manager.Manager
+}
+
+// NewRepo initializes and returns a Repo struct.
+func NewRepo(m *manager.Manager) *Repo {
+	return &Repo{
+		Manager: m,
+		config: m.Config,
+	}
+}
+
+// Clone will clone a repo and return a Repo struct which contains a go-git repo. The clone method
+// is determined by the clone options set in Manager.metadata.cloneOptions
+func (repo *Repo) Clone(cloneOptions ...*git.CloneOptions) error {
+	var (
+		repository  *git.Repository
+		err         error
+		cloneOption *git.CloneOptions
+	)
+	if len(cloneOptions) != 0 {
+		cloneOption = cloneOptions[0]
+	} else {
+		cloneOption = repo.Manager.CloneOptions
+	}
+
+	log.Infof("cloning... %s", cloneOption.URL)
+	start := time.Now()
+
+	if repo.Manager.CloneDir != "" {
+		clonePath := fmt.Sprintf("%s/%x", repo.Manager.CloneDir, md5.Sum([]byte(time.Now().String())))
+		repository, err = git.PlainClone(clonePath, false, cloneOption)
+	} else {
+		repository, err = git.Clone(memory.NewStorage(), nil, cloneOption)
+	}
+	if err != nil {
+		return err
+	}
+	repo.Repository = repository
+	repo.Manager.RecordTime(manager.CloneTime(howLong(start)))
+
+	return nil
+}
+
+// AuditLocal will do a `git diff` and scan changed files that are being tracked. This is useful functionality
+// for a pre-commit hook so you can make sure your code does not have any leaks before committing.
+func (repo *Repo) AuditLocal() error {
+	auditTimeStart := time.Now()
+
+	r, err := repo.Head()
+	if err != nil {
+		return err
+	}
+
+	c, err := repo.CommitObject(r.Hash())
+	if err != nil {
+		return err
+	}
+
+	prevTree, err := c.Tree()
+	if err != nil {
+		return err
+	}
+	wt, err := repo.Worktree()
+	if err != nil {
+		return err
+	}
+
+	status, err := wt.Status()
+	for fn, state := range status {
+		var (
+			prevFileContents string
+			currFileContents string
+			filename         string
+		)
+
+		if state.Staging != git.Untracked {
+			if state.Staging == git.Deleted {
+				// file in staging has been deleted, aka it is not on the filesystem
+				// so the contents of the file are ""
+				currFileContents = ""
+			} else {
+				workTreeBuf := bytes.NewBuffer(nil)
+				workTreeFile, err := wt.Filesystem.Open(fn)
+				if err != nil {
+					continue
+				}
+				if _, err := io.Copy(workTreeBuf, workTreeFile); err != nil {
+					return err
+				}
+				currFileContents = workTreeBuf.String()
+				filename = workTreeFile.Name()
+			}
+
+			// get files at HEAD state
+			prevFile, err := prevTree.File(fn)
+			if err != nil {
+				prevFileContents = ""
+
+			} else {
+				prevFileContents, err = prevFile.Contents()
+				if err != nil {
+					return err
+				}
+				if filename == "" {
+					filename = prevFile.Name
+				}
+			}
+
+			dmp := diffmatchpatch.New()
+			diffs := dmp.DiffMain(prevFileContents, currFileContents, false)
+			var diffContents string
+			for _, d := range diffs {
+				switch d.Type {
+				case diffmatchpatch.DiffInsert:
+					diffContents += fmt.Sprintf("%s\n", d.Text)
+				case diffmatchpatch.DiffDelete:
+					diffContents += fmt.Sprintf("%s\n", d.Text)
+				}
+			}
+			InspectString(diffContents, c, repo, filename)
+		}
+	}
+
+	if err != nil {
+		return err
+	}
+	repo.Manager.RecordTime(manager.AuditTime(howLong(auditTimeStart)))
+	return nil
+}
+
+// Audit is responsible for scanning the entire history (default behavior) of a
+// git repo. Options that can change the behavior of this function include: --commit, --depth, --branch.
+// See options/options.go for an explanation on these options.
+func (repo *Repo) Audit() error {
+	if repo.Repository == nil {
+		return fmt.Errorf("%s repo is empty", repo.Name)
+	}
+
+	// load up alternative config if possible, if not use manager's config
+	if repo.Manager.Opts.RepoConfig {
+		cfg, err := repo.loadRepoConfig()
+		if err != nil {
+			return err
+		}
+		repo.config = cfg
+	}
+
+	auditTimeStart := time.Now()
+
+	// audit single Commit
+	if repo.Manager.Opts.Commit != "" {
+		h := plumbing.NewHash(repo.Manager.Opts.Commit)
+		c, err := repo.CommitObject(h)
+		if err != nil {
+			return err
+		}
+
+		err = inspectCommit(c, repo)
+		if err != nil {
+			return err
+		}
+		return nil
+	}
+
+	logOpts, err := getLogOptions(repo)
+	if err != nil {
+		return err
+	}
+	cIter, err := repo.Log(logOpts)
+	if err != nil {
+		return err
+	}
+
+	//checker := make(map[string]bool)
+	cc := 0
+	semaphore := make(chan bool, howManyThreads(repo.Manager.Opts.Threads))
+	wg := sync.WaitGroup{}
+	err = cIter.ForEach(func(c *object.Commit) error {
+		if c == nil {
+			return storer.ErrStop
+		}
+
+		if len(c.ParentHashes) == 0 {
+			cc++
+			err = inspectCommit(c, repo)
+			if err != nil {
+				return err
+			}
+			return nil
+		}
+
+		// TODO check whitelist Commit
+		if isCommitWhiteListed(c.Hash.String(), repo.config.Whitelist.Commits) {
+			return nil
+		}
+
+		cc++
+		err = c.Parents().ForEach(func(parent *object.Commit) error {
+			start := time.Now()
+			patch, err := c.Patch(parent)
+			if err != nil {
+				return fmt.Errorf("could not generate patch")
+			}
+			repo.Manager.RecordTime(manager.PatchTime(howLong(start)))
+			wg.Add(1)
+			semaphore <- true
+			go func(c *object.Commit, patch *object.Patch) {
+				defer func() {
+					<-semaphore
+					wg.Done()
+				}()
+				inspectPatch(patch, c, repo)
+			}(c, patch)
+
+			return nil
+		})
+		return nil
+	})
+
+	wg.Wait()
+	repo.Manager.RecordTime(manager.AuditTime(howLong(auditTimeStart)))
+	repo.Manager.IncrementCommits(cc)
+	return nil
+}
+
+// Open opens a local repo either from repo-path or $PWD
+func (repo *Repo) Open() error {
+	if repo.Manager.Opts.RepoPath != "" {
+		// open git repo from repo path
+		repository, err := git.PlainOpen(repo.Manager.Opts.RepoPath)
+		if err != nil {
+			return err
+		}
+		repo.Repository = repository
+	} else {
+		// open git repo from PWD
+		dir, err := os.Getwd()
+		if err != nil {
+			return err
+		}
+		repository, err := git.PlainOpen(dir)
+		if err != nil {
+			return err
+		}
+		repo.Repository = repository
+	}
+	return nil
+}
+
+func (repo *Repo) loadRepoConfig() (config.Config, error) {
+	wt, err := repo.Repository.Worktree()
+	if err != nil {
+		return config.Config{}, err
+	}
+	var f billy.File
+	f, _ = wt.Filesystem.Open(".gitleaks.toml")
+	if f == nil {
+		f, err = wt.Filesystem.Open("gitleaks.toml")
+		if err != nil {
+			return config.Config{}, fmt.Errorf("problem loading repo config: %v", err)
+		}
+	}
+	defer f.Close()
+	var tomlLoader config.TomlLoader
+	_, err = toml.DecodeReader(f, &tomlLoader)
+	return tomlLoader.Parse()
+}
+

audit/util.go

@@ -0,0 +1,310 @@
+package audit
+
+import (
+	"fmt"
+	log "github.com/sirupsen/logrus"
+	"github.com/zricethezav/gitleaks-ng/config"
+	"github.com/zricethezav/gitleaks-ng/manager"
+	"gopkg.in/src-d/go-git.v4"
+	"gopkg.in/src-d/go-git.v4/plumbing"
+	fdiff "gopkg.in/src-d/go-git.v4/plumbing/format/diff"
+	"gopkg.in/src-d/go-git.v4/plumbing/object"
+	"math"
+	"path"
+	"regexp"
+	"runtime"
+	"strings"
+	"time"
+)
+
+const maxLineLen = 200
+
+// Inspect patch accepts a patch, commit, and repo. If the patches contains files that are
+// binary, then gitleaks will skip auditing that file OR if a file is matched on
+// whitelisted files set in the configuration. If a global rule for files is defined and a filename
+// matches said global rule, then a laek is sent to the manager.
+// After that, file chunks are created which are then inspected by InspectString()
+func inspectPatch(patch *object.Patch, c *object.Commit, repo *Repo) {
+	for _, f := range patch.FilePatches() {
+		if f.IsBinary() {
+			continue
+		}
+		if fileMatched(getFileName(f), repo.config.Whitelist.File) {
+			log.Debugf("whitelisted file found, skipping audit of file: %s", getFileName(f))
+			continue
+		}
+		if fileMatched(getFileName(f), repo.config.FileRegex) {
+			repo.Manager.SendLeaks(manager.Leak{
+				Line:     "N/A",
+				Offender: getFileName(f),
+				Commit:   c.Hash.String(),
+				Repo:     repo.Name,
+				Rule:     "file regex matched" + repo.config.FileRegex.String(),
+				Author:   c.Author.Name,
+				Email:    c.Author.Email,
+				Date:     c.Author.When,
+				File:     getFileName(f),
+			})
+		}
+		for _, chunk := range f.Chunks() {
+			if chunk.Type() == fdiff.Delete || chunk.Type() == fdiff.Add {
+				InspectString(chunk.Content(), c, repo, getFileName(f))
+			}
+		}
+	}
+}
+
+// getFileName accepts a file patch and returns the filename
+func getFileName(f fdiff.FilePatch) string {
+	fn := "???"
+	from, to := f.Files()
+	if from != nil {
+		return path.Base(from.Path())
+	} else if to != nil {
+		return path.Base(to.Path())
+	}
+
+	return fn
+}
+
+// getShannonEntropy https://en.wiktionary.org/wiki/Shannon_entropy
+func shannonEntropy(data string) (entropy float64) {
+	if data == "" {
+		return 0
+	}
+
+	charCounts := make(map[rune]int)
+	for _, char := range data {
+		charCounts[char]++
+	}
+
+	invLength := 1.0 / float64(len(data))
+	for _, count := range charCounts {
+		freq := float64(count) * invLength
+		entropy -= freq * math.Log2(freq)
+	}
+
+	return entropy
+}
+
+// trippedEntropy checks if a given line falls in between entropy ranges supplied
+// by a custom gitleaks configuration. Gitleaks do not check entropy by default.
+func trippedEntropy(line string, rule config.Rule) bool {
+	for _, e := range rule.Entropy {
+		entropy := shannonEntropy(line)
+		if entropy > e.P1 && entropy < e.P2 {
+			return true
+		}
+	}
+	return false
+}
+
+// InspectString accepts a string, commit object, repo, and filename. This function iterates over
+// all the rules set by the gitleaks config. If the rule contains entropy checks then entropy will be checked first.
+// Next, if the rule contains a regular expression then that will be checked.
+func InspectString(content string, c *object.Commit, repo *Repo, filename string) {
+	for _, rule := range repo.config.Rules {
+		// check entropy
+		if len(rule.Entropy) != 0 {
+			// TODO
+			// an optimization would be to switch the regex from FindAllIndex to FindString
+			// since we are iterating on the lines if entropy rules exist...
+			for _, line := range strings.Split(content, "\n") {
+				if trippedEntropy(line, rule) {
+					_line := line
+					if len(_line) > maxLineLen {
+						_line = line[0 : maxLineLen-1]
+					}
+					repo.Manager.SendLeaks(manager.Leak{
+						Line:     _line,
+						Offender: fmt.Sprintf("Entropy range %+v", rule.Entropy),
+						Commit:   c.Hash.String(),
+						Repo:     repo.Name,
+						Message:  c.Message,
+						Rule:     rule.Description,
+						Author:   c.Author.Name,
+						Email:    c.Author.Email,
+						Date:     c.Author.When,
+						Tags:     strings.Join(rule.Tags, ", "),
+						File:     filename,
+					})
+				}
+			}
+		}
+		if rule.Regex.String() == "" {
+			continue
+		}
+		start := time.Now()
+		locs := rule.Regex.FindAllIndex([]byte(content), -1)
+		if len(locs) != 0 {
+			// check if any rules are whitelisting this leak
+			if len(rule.Whitelist) != 0 {
+				for _, wl := range rule.Whitelist {
+					if fileMatched(filename, wl.File) {
+						// if matched, go to next rule
+						goto NEXT
+					}
+				}
+			}
+			for _, loc := range locs {
+				start := loc[0]
+				end := loc[1]
+				for start != 0 && content[start] != '\n' {
+					start = start - 1
+				}
+				if start != 0 {
+					// skip newline
+					start = start + 1
+				}
+
+				for end < len(content)-1 && content[end] != '\n' {
+					end = end + 1
+				}
+
+				offender := content[loc[0]:loc[1]]
+				line := content[start:end]
+				if repo.Manager.Opts.Redact {
+					line = strings.ReplaceAll(line, offender, "REDACTED")
+					offender = "REDACTED"
+				}
+
+				repo.Manager.SendLeaks(manager.Leak{
+					Line:     line,
+					Offender: offender,
+					Commit:   c.Hash.String(),
+					Message:  c.Message,
+					Repo:     repo.Name,
+					Rule:     rule.Description,
+					Author:   c.Author.Name,
+					Email:    c.Author.Email,
+					Date:     c.Author.When,
+					Tags:     strings.Join(rule.Tags, ", "),
+					File:     filename,
+				})
+			}
+		}
+		repo.Manager.RecordTime(manager.RegexTime{
+			Time:  time.Now().Sub(start).Nanoseconds(),
+			Regex: rule.Regex.String(),
+		})
+	NEXT:
+	}
+}
+
+// inspectCommit accepts a commit object and a repo. This function is only called when the --commit=
+// option has been set. That option tells gitleaks to look only at a single commit and check the contents
+// of said commit. Similar to inspectPatch(), if the files contained in the commit are a binaries or if they are
+// whitelisted then those files will be skipped.
+func inspectCommit(c *object.Commit, repo *Repo) error {
+	fIter, err := c.Files()
+	if err != nil {
+		return err
+	}
+
+	err = fIter.ForEach(func(f *object.File) error {
+		bin, err := f.IsBinary()
+		if bin {
+			return nil
+		} else if err != nil {
+			return err
+		}
+		if fileMatched(f, repo.config.Whitelist.File) {
+			log.Debugf("whitelisted file found, skipping audit of file: %s", f.Name)
+			return nil
+		}
+		content, err := f.Contents()
+		if err != nil {
+			return err
+		}
+
+		InspectString(content, c, repo, f.Name)
+
+		return nil
+	})
+	return err
+}
+
+// howManyThreads will return a number 1-GOMAXPROCS which is the number
+// of goroutines that will spawn during gitleaks execution
+func howManyThreads(threads int) int {
+	maxThreads := runtime.GOMAXPROCS(0)
+	if threads == 0 {
+		return 1
+	} else if threads > maxThreads {
+		log.Warnf("%d threads set too high, setting to system max, %d", threads, maxThreads)
+		return maxThreads
+	}
+	return threads
+}
+
+func isCommitWhiteListed(commitHash string, whitelistedCommits []string) bool {
+	for _, hash := range whitelistedCommits {
+		if commitHash == hash {
+			return true
+		}
+	}
+	return false
+}
+
+func fileMatched(f interface{}, re *regexp.Regexp) bool {
+	if re == nil {
+		return false
+	}
+	switch f.(type) {
+	case nil:
+		return false
+	case string:
+		if re.FindString(f.(string)) != "" {
+			return true
+		}
+		return false
+	case *object.File:
+		if re.FindString(f.(*object.File).Name) != "" {
+			return true
+		}
+		return false
+	}
+	return false
+}
+
+// getLogOptions determines what log options are used when iterating through commits.
+// It is similar to `git log {branch}`. Default behavior is to log ALL branches so
+// gitleaks gets the full git history.
+func getLogOptions(repo *Repo) (*git.LogOptions, error) {
+	if repo.Manager.Opts.Branch != "" {
+		var logOpts git.LogOptions
+		refs, err := repo.Storer.IterReferences()
+		if err != nil {
+			return nil, err
+		}
+		err = refs.ForEach(func(ref *plumbing.Reference) error {
+			if ref.Name().IsTag() {
+				return nil
+			}
+			// check heads first
+			if ref.Name().String() == "refs/heads/"+repo.Manager.Opts.Branch {
+				logOpts = git.LogOptions{
+					From: ref.Hash(),
+				}
+				return nil
+			} else if ref.Name().String() == "refs/remotes/origin/"+repo.Manager.Opts.Branch {
+				logOpts = git.LogOptions{
+					From: ref.Hash(),
+				}
+				return nil
+			}
+			return nil
+		})
+		if logOpts.From.IsZero() {
+			return nil, fmt.Errorf("could not find branch %s", repo.Manager.Opts.Branch)
+		}
+		return &logOpts, nil
+	}
+	return &git.LogOptions{All: true}, nil
+}
+
+// howLong accepts a time.Time object which is subtracted from time.Now() and
+// converted to nanoseconds which is returned
+func howLong(t time.Time) int64 {
+	return time.Now().Sub(t).Nanoseconds()
+}

config/config.go

@@ -0,0 +1,201 @@
+package config
+
+import (
+	"fmt"
+	"github.com/BurntSushi/toml"
+	"github.com/zricethezav/gitleaks-ng/options"
+	"regexp"
+	"strconv"
+	"strings"
+)
+
+// WhiteList is struct containing items that if encountered will whitelist
+// a commit/line of code that would be considered a leak.
+type Whitelist struct {
+	Description string
+	Regex       *regexp.Regexp
+	File        *regexp.Regexp
+}
+
+// entropy represents an entropy range
+type entropy struct {
+	P1, P2 float64
+}
+
+// Rule is a struct that contains information that is loaded from a gitleaks config.
+// This struct is used in the Config struct as an array of Rules and is iterated
+// over during an audit. Each rule will be checked. If a regex match is found AND
+// that match is not whitelisted (globally or locally), then a leak will be appended
+// to the final audit report.
+type Rule struct {
+	Description string
+	Regex       *regexp.Regexp
+	Tags        []string
+	Whitelist   []Whitelist
+	Entropy     []entropy
+}
+
+// Config is a composite struct of Rules and Whitelists
+// Each Rule contains a description, regular expression, tags, and whitelists if available
+type Config struct {
+	FileRegex *regexp.Regexp
+	Message   *regexp.Regexp
+	Rules     []Rule
+	Whitelist struct {
+		Description string
+		Commits     []string
+		File        *regexp.Regexp
+	}
+}
+
+// TomlLoader gets loaded with the values from a gitleaks toml config
+// see the config in config/defaults.go for an example. TomlLoader is used
+// to generate Config values (compiling regexes, etc).
+type TomlLoader struct {
+	Global struct {
+		File    string
+		Message string
+	}
+	Whitelist struct {
+		Description string
+		Commits     []string
+		File        string
+	}
+	Rules []struct {
+		Description string
+		Regex       string
+		Tags        []string
+		Entropies   []string
+		Whitelist   []struct {
+			Description string
+			Regex       string
+			File        string
+		}
+	}
+}
+
+// NewConfig will create a new config struct which contains
+// rules on how gitleaks will proceed with its audit.
+// If no options are passed via cli then NewConfig will return
+// a default config which can be seen in config.go
+func NewConfig(options options.Options) (Config, error) {
+	var cfg Config
+	tomlLoader := TomlLoader{}
+
+	var err error
+	if options.Config != "" {
+		_, err = toml.DecodeFile(options.Config, &tomlLoader)
+	} else {
+		_, err = toml.Decode(DefaultConfig, &tomlLoader)
+	}
+	if err != nil {
+		return cfg, err
+	}
+
+	cfg, err = tomlLoader.Parse()
+	if err != nil {
+		return cfg, err
+	}
+
+	return cfg, nil
+}
+
+// Parse will parse the values set in a TomlLoader and use those values
+// to create compiled regular expressions and rules used in audits
+func (tomlLoader TomlLoader) Parse() (Config, error) {
+	var cfg Config
+	for _, rule := range tomlLoader.Rules {
+		re, err := regexp.Compile(rule.Regex)
+		if err != nil {
+			return cfg, fmt.Errorf("problem loading config: %v", err)
+		}
+
+		// rule specific whitelists
+		var whitelists []Whitelist
+		for _, wl := range rule.Whitelist {
+			re, err := regexp.Compile(wl.Regex)
+			if err != nil {
+				return cfg, fmt.Errorf("problem loading config: %v", err)
+			}
+			fileRe, err := regexp.Compile(wl.File)
+			if err != nil {
+				return cfg, fmt.Errorf("problem loading config: %v", err)
+			}
+			if err != nil {
+				return cfg, fmt.Errorf("problem loading config: %v", err)
+			}
+			whitelists = append(whitelists, Whitelist{
+				Description: wl.Description,
+				File:        fileRe,
+				Regex:       re,
+			})
+		}
+
+		entropies, err := getEntropy(rule.Entropies)
+		if err != nil {
+			return cfg, err
+		}
+
+		cfg.Rules = append(cfg.Rules, Rule{
+			Description: rule.Description,
+			Regex:       re,
+			Tags:        rule.Tags,
+			Whitelist:   whitelists,
+			Entropy:     entropies,
+		})
+	}
+
+	// global leaks
+	if tomlLoader.Global.File != "" {
+		re, err := regexp.Compile(tomlLoader.Global.File)
+		if err != nil {
+			return cfg, fmt.Errorf("problem loading config: %v", err)
+		}
+		cfg.FileRegex = re
+	}
+	if tomlLoader.Global.Message != "" {
+		re, err := regexp.Compile(tomlLoader.Global.Message)
+		if err != nil {
+			return cfg, fmt.Errorf("problem loading config: %v", err)
+		}
+		cfg.Message = re
+	}
+
+	// global whitelists
+	if tomlLoader.Whitelist.File != "" {
+		re, err := regexp.Compile(tomlLoader.Whitelist.File)
+		if err != nil {
+			return cfg, fmt.Errorf("problem loading config: %v", err)
+		}
+		cfg.Whitelist.File = re
+	}
+	cfg.Whitelist.Commits = tomlLoader.Whitelist.Commits
+	cfg.Whitelist.Description = tomlLoader.Whitelist.Description
+
+	return cfg, nil
+}
+
+// getEntropy
+func getEntropy(entropyStr []string) ([]entropy, error) {
+	var ranges []entropy
+	for _, span := range entropyStr {
+		split := strings.Split(span, "-")
+		v1, err := strconv.ParseFloat(split[0], 64)
+		if err != nil {
+			return nil, err
+		}
+		v2, err := strconv.ParseFloat(split[1], 64)
+		if err != nil {
+			return nil, err
+		}
+		if v1 > v2 {
+			return nil, fmt.Errorf("entropy range must be ascending")
+		}
+		r := entropy{P1: v1, P2: v2}
+		if r.P1 > 8.0 || r.P1 < 0.0 || r.P2 > 8.0 || r.P2 < 0.0 {
+			return nil, fmt.Errorf("invalid entropy ranges, must be within 0.0-8.0")
+		}
+		ranges = append(ranges, r)
+	}
+	return ranges, nil
+}

config/config_test.go

@@ -0,0 +1,116 @@
+package config
+
+import (
+	"fmt"
+	"github.com/zricethezav/gitleaks-ng/options"
+	"regexp"
+	"testing"
+)
+
+func TestParse(t *testing.T) {
+	tests := []struct {
+		description   string
+		opts          options.Options
+		wantErr       error
+		wantFileRegex *regexp.Regexp
+		wantMessages  *regexp.Regexp
+		wantWhitelist Whitelist
+	}{
+		{
+			description: "default config",
+			opts: options.Options{},
+		},
+		{
+			description: "test successful load",
+			opts: options.Options{
+				Config: "../test_data/test_configs/aws_key.toml",
+			},
+		},
+		{
+			description: "test bad toml",
+			opts: options.Options{
+				Config: "../test_data/test_configs/bad_aws_key.toml",
+			},
+			wantErr: fmt.Errorf("Near line 7 (last key parsed 'rules.description'): expected value but found \"AWS\" instead"),
+		},
+		{
+			description: "test bad regex",
+			opts: options.Options{
+				Config: "../test_data/test_configs/bad_regex_aws_key.toml",
+			},
+			wantErr: fmt.Errorf("problem loading config: error parsing regexp: invalid nested repetition operator: `???`"),
+		},
+		{
+			description: "test bad global whitelist file regex",
+			opts: options.Options{
+				Config: "../test_data/test_configs/bad_aws_key_global_whitelist_file.toml",
+			},
+			wantErr: fmt.Errorf("problem loading config: error parsing regexp: missing argument to repetition operator: `??`"),
+		},
+		{
+			description: "test bad global file regex",
+			opts: options.Options{
+				Config: "../test_data/test_configs/bad_aws_key_file_regex.toml",
+			},
+			wantErr: fmt.Errorf("problem loading config: error parsing regexp: missing argument to repetition operator: `??`"),
+		},
+		{
+			description: "test bad global message regex",
+			opts: options.Options{
+				Config: "../test_data/test_configs/bad_aws_key_message_regex.toml",
+			},
+			wantErr: fmt.Errorf("problem loading config: error parsing regexp: missing argument to repetition operator: `??`"),
+		},
+		{
+			description: "test successful load big ol thing",
+			opts: options.Options{
+				Config: "../test_data/test_configs/large.toml",
+			},
+		},
+		{
+			description: "test load entropy",
+			opts: options.Options{
+				Config: "../test_data/test_configs/entropy.toml",
+			},
+		},
+		{
+			description: "test entropy bad range",
+			opts: options.Options{
+				Config: "../test_data/test_configs/bad_entropy_1.toml",
+			},
+			wantErr: fmt.Errorf("entropy range must be ascending"),
+		},
+		{
+			description: "test entropy value p2",
+			opts: options.Options{
+				Config: "../test_data/test_configs/bad_entropy_2.toml",
+			},
+			wantErr: fmt.Errorf("strconv.ParseFloat: parsing \"x\": invalid syntax"),
+		},
+		{
+			description: "test entropy value p1",
+			opts: options.Options{
+				Config: "../test_data/test_configs/bad_entropy_3.toml",
+			},
+			wantErr: fmt.Errorf("strconv.ParseFloat: parsing \"x\": invalid syntax"),
+		},
+		{
+			description: "test entropy value p1",
+			opts: options.Options{
+				Config: "../test_data/test_configs/bad_entropy_4.toml",
+			},
+			wantErr: fmt.Errorf("invalid entropy ranges, must be within 0.0-8.0"),
+		},
+	}
+
+	for _, test := range tests {
+		_, err := NewConfig(test.opts)
+		if err != nil {
+			if test.wantErr == nil {
+				t.Error(err)
+			} else if test.wantErr.Error() != err.Error() {
+				t.Errorf("expected err: %s, got %s", test.wantErr, err)
+			}
+		}
+	}
+}

config/default.go

@@ -0,0 +1,136 @@
+package config
+
+const DefaultConfig = `
+title = "gitleaks config"
+
+[[rules]]
+	description = "AWS Secret Key"
+	regex = '''(?i)aws(.{0,20})?(?-i)['\"][0-9a-zA-Z\/+]{40}['\"]'''
+	tags = ["key", "AWS"]
+
+[[rules]]
+	description = "AWS MWS key"
+	regex = '''amzn\.mws\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'''
+	tags = ["key", "AWS", "MWS"]
+
+[[rules]]
+	description = "Facebook Secret Key"
+	regex = '''(?i)(facebook|fb)(.{0,20})?(?-i)['\"][0-9a-f]{32}['\"]'''
+	tags = ["key", "Facebook"]
+
+[[rules]]
+	description = "Facebook Client ID"
+	regex = '''(?i)(facebook|fb)(.{0,20})?['\"][0-9]{13,17}['\"]'''
+	tags = ["key", "Facebook"]
+
+[[rules]]
+	description = "Twitter Secret Key"
+	regex = '''(?i)twitter(.{0,20})?['\"][0-9a-z]{35,44}['\"]'''
+	tags = ["key", "Twitter"]
+
+[[rules]]
+	description = "Twitter Client ID"
+	regex = '''(?i)twitter(.{0,20})?['\"][0-9a-z]{18,25}['\"]'''
+	tags = ["client", "Twitter"]
+
+[[rules]]
+	description = "Github"
+	regex = '''(?i)github(.{0,20})?(?-i)['\"][0-9a-zA-Z]{35,40}['\"]'''
+	tags = ["key", "Github"]
+
+[[rules]]
+	description = "LinkedIn Client ID"
+	regex = '''(?i)linkedin(.{0,20})?(?-i)['\"][0-9a-z]{12}['\"]'''
+	tags = ["client", "LinkedIn"]
+
+[[rules]]
+	description = "LinkedIn Secret Key"
+	regex = '''(?i)linkedin(.{0,20})?['\"][0-9a-z]{16}['\"]'''
+	tags = ["secret", "LinkedIn"]
+
+[[rules]]
+	description = "Slack"
+	regex = '''xox[baprs]-([0-9a-zA-Z]{10,48})?'''
+	tags = ["key", "Slack"]
+
+[[rules]]
+	description = "EC"
+	regex = '''-----BEGIN EC PRIVATE KEY-----'''
+	tags = ["key", "EC"]
+
+[[rules]]
+	description = "Generic API key"
+	regex = '''(?i)(api_key|apikey)(.{0,20})?['|"][0-9a-zA-Z]{32,45}['|"]'''
+	tags = ["key", "API", "generic"]
+
+[[rules]]
+	description = "Generic Secret"
+	regex = '''(?i)secret(.{0,20})?['|"][0-9a-zA-Z]{32,45}['|"]'''
+	tags = ["key", "Secret", "generic"]
+
+[[rules]]
+	description = "Google API key"
+	regex = '''AIza[0-9A-Za-z\\-_]{35}'''
+	tags = ["key", "Google"]
+
+
+[[rules]]
+	description = "Heroku API key"
+	regex = '''(?i)heroku(.{0,20})?['"][0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}['"]'''
+	tags = ["key", "Heroku"]
+
+[[rules]]
+	description = "MailChimp API key"
+	regex = '''(?i)(mailchimp|mc)(.{0,20})?['"][0-9a-f]{32}-us[0-9]{1,2}['"]'''
+	tags = ["key", "Mailchimp"]
+
+[[rules]]
+	description = "Mailgun API key"
+	regex = '''(?i)(mailgun|mg)(.{0,20})?['"][0-9a-z]{32}['"]'''
+	tags = ["key", "Mailgun"]
+
+[[rules]]
+	description = "PayPal Braintree access token"
+	regex = '''access_token\$production\$[0-9a-z]{16}\$[0-9a-f]{32}'''
+	tags = ["key", "Paypal"]
+
+[[rules]]
+	description = "Picatic API key"
+	regex = '''sk_live_[0-9a-z]{32}'''
+	tags = ["key", "Picatic"]
+
+[[rules]]
+	description = "Slack Webhook"
+	regex = '''https://hooks.slack.com/services/T[a-zA-Z0-9_]{8}/B[a-zA-Z0-9_]{8}/[a-zA-Z0-9_]{24}'''
+	tags = ["key", "slack"]
+
+[[rules]]
+	description = "Stripe API key"
+	regex = '''(?i)stripe(.{0,20})?['\"][sk|rk]_live_[0-9a-zA-Z]{24}'''
+	tags = ["key", "Stripe"]
+
+[[rules]]
+	description = "Square access token"
+	regex = '''sq0atp-[0-9A-Za-z\-_]{22}'''
+	tags = ["key", "square"]
+
+[[rules]]
+	description = "Square OAuth secret"
+	regex = '''sq0csp-[0-9A-Za-z\\-_]{43}'''
+	tags = ["key", "square"]
+
+[[rules]]
+	description = "Twilio API key"
+	regex = '''(?i)twilio(.{0,20})?['\"][0-9a-f]{32}['\"]'''
+	tags = ["key", "twilio"]
+
+[[rules]]
+	description = "AWS Manager ID"
+	regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
+	tags = ["key", "AWS"]
+
+[whitelist]
+	description = "image whitelists"
+	file = '''(.*?)(jpg|gif|doc|pdf|bin)$'''
+`
+

gitleaks.toml

@@ -1,222 +0,0 @@
-title = "sample gitleaks config"
-
-# This is a sample config file for gitleaks. You can configure gitleaks what to search for and what to whitelist.
-# The output you are seeing here is the default gitleaks config. If GITLEAKS_CONFIG environment variable
-# is set, gitleaks will load configurations from that path. If option --config-path is set, gitleaks will load
-# configurations from that path. Gitleaks does not whitelist anything by default.
-# - https://www.ndss-symposium.org/wp-content/uploads/2019/02/ndss2019_04B-3_Meli_paper.pdf
-# - https://github.com/dxa4481/truffleHogRegexes/blob/master/truffleHogRegexes/regexes.json
-[[rules]]
-description = "AWS Client ID"
-regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
-tags = ["key", "AWS"]
-
-[[rules]]
-description = "AWS Secret Key"
-regex = '''(?i)aws(.{0,20})?(?-i)['\"][0-9a-zA-Z\/+]{40}['\"]'''
-tags = ["key", "AWS"]
-
-[[rules]]
-description = "AWS MWS key"
-regex = '''amzn\.mws\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'''
-tags = ["key", "AWS", "MWS"]
-
-[[rules]]
-description = "PKCS8"
-regex = '''-----BEGIN PRIVATE KEY-----'''
-tags = ["key", "PKCS8"]
-
-[[rules]]
-description = "RSA"
-regex = '''-----BEGIN RSA PRIVATE KEY-----'''
-tags = ["key", "RSA"]
-
-[[rules]]
-description = "SSH"
-regex = '''-----BEGIN OPENSSH PRIVATE KEY-----'''
-tags = ["key", "SSH"]
-
-[[rules]]
-description = "PGP"
-regex = '''-----BEGIN PGP PRIVATE KEY BLOCK-----'''
-tags = ["key", "PGP"]
-
-[[rules]]
-description = "Facebook Secret Key"
-regex = '''(?i)(facebook|fb)(.{0,20})?(?-i)['\"][0-9a-f]{32}['\"]'''
-tags = ["key", "Facebook"]
-
-[[rules]]
-description = "Facebook Client ID"
-regex = '''(?i)(facebook|fb)(.{0,20})?['\"][0-9]{13,17}['\"]'''
-tags = ["key", "Facebook"]
-
-[[rules]]
-description = "Facebook access token"
-regex = '''EAACEdEose0cBA[0-9A-Za-z]+'''
-tags = ["key", "Facebook"]
-
-[[rules]]
-description = "Twitter Secret Key"
-regex = '''(?i)twitter(.{0,20})?['\"][0-9a-z]{35,44}['\"]'''
-tags = ["key", "Twitter"]
-
-[[rules]]
-description = "Twitter Client ID"
-regex = '''(?i)twitter(.{0,20})?['\"][0-9a-z]{18,25}['\"]'''
-tags = ["client", "Twitter"]
-
-[[rules]]
-description = "Github"
-regex = '''(?i)github(.{0,20})?(?-i)['\"][0-9a-zA-Z]{35,40}['\"]'''
-tags = ["key", "Github"]
-
-[[rules]]
-description = "LinkedIn Client ID"
-regex = '''(?i)linkedin(.{0,20})?(?-i)['\"][0-9a-z]{12}['\"]'''
-tags = ["client", "LinkedIn"]
-
-[[rules]]
-description = "LinkedIn Secret Key"
-regex = '''(?i)linkedin(.{0,20})?['\"][0-9a-z]{16}['\"]'''
-tags = ["secret", "LinkedIn"]
-
-[[rules]]
-description = "Slack"
-regex = '''xox[baprs]-([0-9a-zA-Z]{10,48})?'''
-tags = ["key", "Slack"]
-
-[[rules]]
-description = "EC"
-regex = '''-----BEGIN EC PRIVATE KEY-----'''
-tags = ["key", "EC"]
-
-[[rules]]
-description = "Generic API key"
-regex = '''(?i)(api_key|apikey)(.{0,20})?['|"][0-9a-zA-Z]{32,45}['|"]'''
-tags = ["key", "API", "generic"]
-
-[[rules]]
-description = "Generic Secret"
-regex = '''(?i)secret(.{0,20})?['|"][0-9a-zA-Z]{32,45}['|"]'''
-tags = ["key", "Secret", "generic"]
-
-[[rules]]
-description = "Google API key"
-regex = '''AIza[0-9A-Za-z\\-_]{35}'''
-tags = ["key", "Google"]
-
-[[rules]]
-description = "Google Cloud Platform API key"
-regex = '''(?i)(google|gcp|youtube|drive|yt)(.{0,20})?['\"][AIza[0-9a-z\\-_]{35}]['\"]'''
-tags = ["key", "Google", "GCP"]
-
-[[rules]]
-description = "Google OAuth"
-regex = '''(?i)(google|gcp|auth)(.{0,20})?['"][0-9]+-[0-9a-z_]{32}\.apps\.googleusercontent\.com['"]'''
-tags = ["key", "Google", "OAuth"]
-
-[[rules]]
-description = "Google OAuth access token"
-regex = '''ya29\.[0-9A-Za-z\-_]+'''
-tags = ["key", "Google", "OAuth"]
-
-[[rules]]
-description = "Heroku API key"
-regex = '''(?i)heroku(.{0,20})?['"][0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}['"]'''
-tags = ["key", "Heroku"]
-
-[[rules]]
-description = "MailChimp API key"
-regex = '''(?i)(mailchimp|mc)(.{0,20})?['"][0-9a-f]{32}-us[0-9]{1,2}['"]'''
-tags = ["key", "Mailchimp"]
-
-[[rules]]
-description = "Mailgun API key"
-regex = '''(?i)(mailgun|mg)(.{0,20})?['"][0-9a-z]{32}['"]'''
-tags = ["key", "Mailgun"]
-
-[[rules]]
-description = "Password in URL"
-regex = '''[a-zA-Z]{3,10}:\/\/[^\/\s:@]{3,20}:[^\/\s:@]{3,20}@.{1,100}\/?.?'''
-tags = ["key", "URL", "generic"]
-
-[[rules]]
-description = "PayPal Braintree access token"
-regex = '''access_token\$production\$[0-9a-z]{16}\$[0-9a-f]{32}'''
-tags = ["key", "Paypal"]
-
-[[rules]]
-description = "Picatic API key"
-regex = '''sk_live_[0-9a-z]{32}'''
-tags = ["key", "Picatic"]
-
-[[rules]]
-description = "Slack Webhook"
-regex = '''https://hooks.slack.com/services/T[a-zA-Z0-9_]{8}/B[a-zA-Z0-9_]{8}/[a-zA-Z0-9_]{24}'''
-tags = ["key", "slack"]
-
-[[rules]]
-description = "Stripe API key"
-regex = '''(?i)stripe(.{0,20})?['\"][sk|rk]_live_[0-9a-zA-Z]{24}'''
-tags = ["key", "Stripe"]
-
-[[rules]]
-description = "Square access token"
-regex = '''sq0atp-[0-9A-Za-z\-_]{22}'''
-tags = ["key", "square"]
-
-[[rules]]
-description = "Square OAuth secret"
-regex = '''sq0csp-[0-9A-Za-z\\-_]{43}'''
-tags = ["key", "square"]
-
-[[rules]]
-description = "Twilio API key"
-regex = '''(?i)twilio(.{0,20})?['\"][0-9a-f]{32}['\"]'''
-tags = ["key", "twilio"]
-
-[whitelist]
-files = [
-  "(.*?)(jpg|gif|doc|pdf|bin)$"
-]
-#regexes = [
-#  "AKIAIOSFODNN7EXAMPLE"
-#]
-#commits = [
-#  "whitelisted-commit1",
-#  "whitelisted-commit2",
-#]
-#repos = [
-#	"whitelisted-repo"
-#]
-
-# Additional Examples
-
-# [[rules]]
-# description = "Generic Key"
-# regex = '''(?i)key(.{0,6})?(:|=|=>|:=)'''
-# entropies = [
-#     "4.1-4.3",
-#     "5.5-6.3",
-# ]
-# entropyROI = "line"
-# filetypes = [".go", ".py", ".c"]
-# tags = ["key"]
-# severity = "8"
-#
-#
-# [[rules]]
-# description = "Generic Key"
-# regex = '''(?i)key(.{0,6})?(:|=|=>|:=)'''
-# entropies = ["4.1-4.3"]
-# filetypes = [".gee"]
-# entropyROI = "line"
-# tags = ["key"]
-# severity = "medium"
-
-# [[rules]]
-# description = "Any pem file"
-# filetypes = [".key"]
-# tags = ["pem"]
-# severity = "high"

go.mod

@@ -1,25 +1,18 @@
-module github.com/zricethezav/gitleaks
+module github.com/zricethezav/gitleaks-ng
+
+go 1.12
 
 require (
 	github.com/BurntSushi/toml v0.3.1
-	github.com/emirpasic/gods v1.12.0 // indirect
-	github.com/franela/goblin v0.0.0-20181003173013-ead4ad1d2727
-	github.com/google/go-github v15.0.0+incompatible
-	github.com/google/go-querystring v1.0.0 // indirect
-	github.com/hako/durafmt v0.0.0-20180520121703-7b7ae1e72ead
-	github.com/ipfs/go-ipfs v0.4.19 // indirect
+	github.com/google/go-github v17.0.0+incompatible
+	github.com/hako/durafmt v0.0.0-20191009132224-3f39dc1ed9f4
 	github.com/jessevdk/go-flags v1.4.0
 	github.com/mattn/go-colorable v0.1.2
-	github.com/onsi/ginkgo v1.8.0 // indirect
-	github.com/onsi/gomega v1.5.0 // indirect
-	github.com/sirupsen/logrus v1.0.6
-	github.com/xanzy/go-gitlab v0.11.3
-	golang.org/x/lint v0.0.0-20190409202823-959b441ac422 // indirect
-	golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be
-	golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6 // indirect
-	google.golang.org/appengine v1.2.0 // indirect
-	gopkg.in/airbrake/gobrake.v2 v2.0.9 // indirect
-	gopkg.in/gemnasium/logrus-airbrake-hook.v2 v2.1.2 // indirect
-	gopkg.in/src-d/go-billy.v4 v4.3.0 // indirect
-	gopkg.in/src-d/go-git.v4 v4.9.1
+	github.com/sergi/go-diff v1.0.0
+	github.com/sirupsen/logrus v1.4.2
+	github.com/xanzy/go-gitlab v0.21.0
+	golang.org/x/lint v0.0.0-20190930215403-16217165b5de // indirect
+	golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45
+	gopkg.in/src-d/go-billy.v4 v4.3.2
+	gopkg.in/src-d/go-git.v4 v4.13.1
 )

go.sum

@@ -1,126 +1,123 @@
+cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
 github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
 github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
 github.com/alcortesm/tgz v0.0.0-20161220082320-9c5fe88206d7 h1:uSoVVbwJiQipAclBbw+8quDsfcvFjOpI5iCf4p/cqCs=
 github.com/alcortesm/tgz v0.0.0-20161220082320-9c5fe88206d7/go.mod h1:6zEj6s6u/ghQa61ZWa/C2Aw3RkjiTBOix7dkqa1VLIs=
 github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239 h1:kFOfPq6dUM1hTo4JG6LR5AXSUEsOjtdm0kw0FtQtMJA=
 github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239/go.mod h1:2FmKhYUyUczH0OGQWaF5ceTx0UBShxjsH6f8oGKYe2c=
+github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio=
+github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs=
+github.com/creack/pty v1.1.7/go.mod h1:lj5s0c3V2DBrqTV7llrYr5NG6My20zk30Fl46Y7DoTY=
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/emirpasic/gods v1.9.0/go.mod h1:YfzfFFoVP/catgzJb4IKIqXjX78Ha8FMSDh3ymbK86o=
 github.com/emirpasic/gods v1.12.0 h1:QAUIPSaCu4G+POclxeqb3F+WPpdKqFGlw36+yOzGlrg=
 github.com/emirpasic/gods v1.12.0/go.mod h1:YfzfFFoVP/catgzJb4IKIqXjX78Ha8FMSDh3ymbK86o=
 github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568 h1:BHsljHzVlRcyQhjrss6TZTdY2VfCqZPbv5k3iBFa2ZQ=
 github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc=
-github.com/franela/goblin v0.0.0-20181003173013-ead4ad1d2727 h1:eouy4stZdUKn7n98c1+rdUTxWMg+jvhP+oHt0K8fiug=
-github.com/franela/goblin v0.0.0-20181003173013-ead4ad1d2727/go.mod h1:7dvUGVsVBjqR7JHJk0brhHOZYGmfBYOrK0ZhYMEtBr4=
-github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
-github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
-github.com/gliderlabs/ssh v0.1.1 h1:j3L6gSLQalDETeEg/Jg0mGY0/y/N6zI2xX1978P0Uqw=
-github.com/gliderlabs/ssh v0.1.1/go.mod h1:U7qILu1NlMHj9FlMhZLlkCdDnU1DBEAqr0aevW3Awn0=
+github.com/gliderlabs/ssh v0.2.2 h1:6zsha5zo/TWhRhwqCD3+EarCAgZ2yN28ipRnGPnwkI0=
+github.com/gliderlabs/ssh v0.2.2/go.mod h1:U7qILu1NlMHj9FlMhZLlkCdDnU1DBEAqr0aevW3Awn0=
 github.com/golang/protobuf v1.2.0 h1:P3YflyNX/ehuJFLhxviNdFxQPkGK5cDcApsge1SqnvM=
 github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
-github.com/google/go-cmp v0.2.0 h1:+dTQ8DZQJz0Mb/HjFlkptS1FeQ4cWSnN941F8aEG4SQ=
-github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
-github.com/google/go-github v15.0.0+incompatible h1:jlPg2Cpsxb/FyEV/MFiIE9tW/2RAevQNZDPeHbf5a94=
-github.com/google/go-github v15.0.0+incompatible/go.mod h1:zLgOLi98H3fifZn+44m+umXrS52loVEgC2AApnigrVQ=
+github.com/google/go-cmp v0.3.0 h1:crn/baboCvb5fXaQ0IJ1SGTsTVrWpDsCWC8EGETZijY=
+github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
+github.com/google/go-github v17.0.0+incompatible h1:N0LgJ1j65A7kfXrZnUDaYCs/Sf4rEjNlfyDHW9dolSY=
+github.com/google/go-github v17.0.0+incompatible/go.mod h1:zLgOLi98H3fifZn+44m+umXrS52loVEgC2AApnigrVQ=
 github.com/google/go-querystring v1.0.0 h1:Xkwi/a1rcvNg1PPYe5vI8GbeBY/jrVuDX5ASuANWTrk=
 github.com/google/go-querystring v1.0.0/go.mod h1:odCYkC5MyYFN7vkCjXpyrEuKhc/BUO6wN/zVPAxq5ck=
-github.com/hako/durafmt v0.0.0-20180520121703-7b7ae1e72ead h1:Y9WOGZY2nw5ksbEf5AIpk+vK52Tdg/VN/rHFRfEeeGQ=
-github.com/hako/durafmt v0.0.0-20180520121703-7b7ae1e72ead/go.mod h1:5Scbynm8dF1XAPwIwkGPqzkM/shndPm79Jd1003hTjE=
-github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI=
-github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
-github.com/ipfs/go-ipfs v0.4.19 h1:ioUvuv1L3Zb9XgPi6S8selFMMlTa5AntQzVGvaoYHXM=
-github.com/ipfs/go-ipfs v0.4.19/go.mod h1:iXzbK+Wa6eePj3jQg/uY6Uoq5iOwY+GToD/bgaRadto=
+github.com/hako/durafmt v0.0.0-20191009132224-3f39dc1ed9f4 h1:60gBOooTSmNtrqNaRvrDbi8VAne0REaek2agjnITKSw=
+github.com/hako/durafmt v0.0.0-20191009132224-3f39dc1ed9f4/go.mod h1:5Scbynm8dF1XAPwIwkGPqzkM/shndPm79Jd1003hTjE=
 github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 h1:BQSFePA1RWJOlocH6Fxy8MmwDt+yVQYULKfN0RoTN8A=
 github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo=
 github.com/jessevdk/go-flags v1.4.0 h1:4IU2WS7AumrZ/40jfhf4QVDMsQwqA7VEHozFRrGARJA=
 github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
-github.com/kevinburke/ssh_config v0.0.0-20180830205328-81db2a75821e h1:RgQk53JHp/Cjunrr1WlsXSZpqXn+uREuHvUVcK82CV8=
-github.com/kevinburke/ssh_config v0.0.0-20180830205328-81db2a75821e/go.mod h1:CT57kijsi8u/K/BOFA39wgDQJ9CxiF4nAY/ojJ6r6mM=
+github.com/kevinburke/ssh_config v0.0.0-20190725054713-01f96b0aa0cd h1:Coekwdh0v2wtGp9Gmz1Ze3eVRAWJMLokvN3QjdzCHLY=
+github.com/kevinburke/ssh_config v0.0.0-20190725054713-01f96b0aa0cd/go.mod h1:CT57kijsi8u/K/BOFA39wgDQJ9CxiF4nAY/ojJ6r6mM=
+github.com/konsorten/go-windows-terminal-sequences v1.0.1 h1:mweAR1A6xJ3oS2pRaGiHgQ4OO8tzTaLawm8vnODuwDk=
+github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
 github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
 github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
+github.com/kr/pty v1.1.8/go.mod h1:O1sed60cT9XZ5uDucP5qwvh+TE3NnUj51EiZO/lmSfw=
 github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/mattn/go-colorable v0.1.2 h1:/bC9yWikZXAL9uJdulbSfyVNIR3n3trXl+v8+1sx8mU=
 github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
 github.com/mattn/go-isatty v0.0.8 h1:HLtExJ+uU2HOZ+wI0Tt5DtUDrx8yhUqDcp7fYERX4CE=
 github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
-github.com/mitchellh/go-homedir v1.0.0 h1:vKb8ShqSby24Yrqr/yDYkuFz8d0WUjys40rvnGC8aR0=
-github.com/mitchellh/go-homedir v1.0.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
-github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
-github.com/onsi/ginkgo v1.8.0 h1:VkHVNpR4iVnU8XQR6DBm8BqYjN7CRzw+xKUbVVbbW9w=
-github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
-github.com/onsi/gomega v1.5.0 h1:izbySO9zDPmjJ8rDjLvkA2zJHIo+HkYXHnf7eN7SSyo=
-github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
-github.com/pelletier/go-buffruneio v0.2.0 h1:U4t4R6YkofJ5xHm3dJzuRpPZ0mr5MMCoAWooScCR7aA=
+github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
+github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
 github.com/pelletier/go-buffruneio v0.2.0/go.mod h1:JkE26KsDizTr40EUHkXVtNPvgGtbSNq5BcowyYOWdKo=
-github.com/pkg/errors v0.8.0 h1:WdK/asTD0HN+q6hsWO3/vpuAkAr+tw6aNJNDFFf0+qw=
-github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
+github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/sergi/go-diff v1.0.0 h1:Kpca3qRNrduNnOQeazBd0ysaKrUJiIuISHxogkT9RPQ=
 github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
-github.com/sirupsen/logrus v1.0.6 h1:hcP1GmhGigz/O7h1WVUM5KklBp1JoNS9FggWKdj/j3s=
-github.com/sirupsen/logrus v1.0.6/go.mod h1:pMByvHTf9Beacp5x1UXfOR9xyW/9antXMhjMPG0dEzc=
+github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4=
+github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
 github.com/src-d/gcfg v1.4.0 h1:xXbNR5AlLSA315x2UO+fTSSAXCDf+Ar38/6oyGbDKQ4=
 github.com/src-d/gcfg v1.4.0/go.mod h1:p/UMsR43ujA89BJY9duynAwIpvqEujIH/jFlfL7jWoI=
-github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/objx v0.2.0/go.mod h1:qt09Ya8vawLte6SNmTgCsAVtYtaKzEcn8ATUoHMkEqE=
 github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
-github.com/xanzy/go-gitlab v0.11.3 h1:gSYcSb+pCx3fco6/O3w784/omQVTcrgxRzyf14SBvUQ=
-github.com/xanzy/go-gitlab v0.11.3/go.mod h1:CRKHkvFWNU6C3AEfqLWjnCNnAs4nj8Zk95rX2S3X6Mw=
-github.com/xanzy/ssh-agent v0.2.0 h1:Adglfbi5p9Z0BmK2oKU9nTG+zKfniSfnaMYB+ULd+Ro=
-github.com/xanzy/ssh-agent v0.2.0/go.mod h1:0NyE30eGUDliuLEHJgYte/zncp2zdTStcOnWhgSqHD8=
-golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
-golang.org/x/crypto v0.0.0-20180910181607-0e37d006457b h1:2b9XGzhjiYsYPnKXoEfL7klWZQIt8IfyRCz62gCqqlQ=
-golang.org/x/crypto v0.0.0-20180910181607-0e37d006457b/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
-golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2 h1:VklqNMn3ovrHsnt90PveolxSbWFaJdECFbxSq0Mqo2M=
+github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
+github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
+github.com/xanzy/go-gitlab v0.21.0 h1:Ru55sR4TBoDNsAKwCOpzeaGtbiWj7xTksVmzBJbLu6c=
+github.com/xanzy/go-gitlab v0.21.0/go.mod h1:t4Bmvnxj7k37S4Y17lfLx+nLqkf/oQwT2HagfWKv5Og=
+github.com/xanzy/ssh-agent v0.2.1 h1:TCbipTQL2JiiCprBWx9frJ2eJlCYT00NmctrHxVAr70=
+github.com/xanzy/ssh-agent v0.2.1/go.mod h1:mLlQY/MoOhWBj+gOGMQkOeiEvkx+8pJSI+0Bx9h2kr4=
+golang.org/x/crypto v0.0.0-20190219172222-a4c6cb3142f2/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
-golang.org/x/lint v0.0.0-20190409202823-959b441ac422 h1:QzoH/1pFpZguR8NrRHLcO6jKqfv2zpuSqZLgdm7ZmjI=
-golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
+golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4 h1:HuIa8hRrWRSrqYzx1qI49NNxhdi2PrY7gxVSq1JjLDc=
+golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/lint v0.0.0-20190930215403-16217165b5de h1:5hukYrvBGR8/eNkX5mdUezrA6JiaEZDtJb9Ei+1LlBs=
+golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
 golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
-golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
-golang.org/x/net v0.0.0-20180925072008-f04abc6bdfa7 h1:zKzVgSQ8WOSHzD7I4k8LQjrHUUCNOlBsgc0PcYLVNnY=
-golang.org/x/net v0.0.0-20180925072008-f04abc6bdfa7/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
-golang.org/x/net v0.0.0-20190311183353-d8887717615a h1:oWX7TPOiFAMXLq8o0ikBYfCJVlRHBcsciT5bXOrH628=
+golang.org/x/net v0.0.0-20181108082009-03003ca0c849/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
-golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be h1:vEDujvNQGv4jgYKudGeI/+DAX4Jffq6hpD55MmoEvKs=
-golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
-golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6 h1:bjcUS9ztw9kFmmIxJInhon/0Is3p+EHBKNgquIzo1OI=
-golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sys v0.0.0-20180903190138-2b024373dcd9/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
-golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
-golang.org/x/sys v0.0.0-20180925112736-b09afc3d579e h1:LSlw/Dbj0MkNvPYAAkGinYmGliq+aqS7eKPYlE4oWC4=
-golang.org/x/sys v0.0.0-20180925112736-b09afc3d579e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20190724013045-ca1201d0de80 h1:Ao/3l156eZf2AW5wK8a7/smtodRU+gha3+BeqJ69lRk=
+golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/oauth2 v0.0.0-20181106182150-f42d05182288/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
+golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45 h1:SVwTIAaPC2U/AvvLNZ2a7OVsmBpC8L5BlwK1whH3hm0=
+golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
+golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58 h1:8gQV6CLnAEikrhgkHFbMAEhagSSnXWGV915qUMm9mrU=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
-golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223 h1:DH4skfRX4EBpamg7iV4ZlCpblAHI6s6TDM39bFZumv8=
+golang.org/x/sys v0.0.0-20190221075227-b4e8571b14e0/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
-golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e h1:D5TXcfTk7xF7hvieo4QErS3qqCB4teTffacDWr7CI+0=
+golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
-golang.org/x/tools v0.0.0-20190311212946-11955173bddd h1:/e+gpKk9r3dJobndpTytxS2gOy6m5uvpg+ISQoEcusQ=
+golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
+golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
-google.golang.org/appengine v1.2.0 h1:S0iUepdCWODXRvtE+gcRDd15L+k+k1AiHlMiMjefH24=
-google.golang.org/appengine v1.2.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
-gopkg.in/airbrake/gobrake.v2 v2.0.9 h1:7z2uVWwn7oVeeugY1DtlPAy5H+KYgB1KeKTnqjNatLo=
-gopkg.in/airbrake/gobrake.v2 v2.0.9/go.mod h1:/h5ZAUhDkGaJfjzjKLSjv6zCL6O0LLBxU4K+aSYdM/U=
+golang.org/x/tools v0.0.0-20190729092621-ff9f1409240a h1:mEQZbbaBjWyLNy0tmZmgEuQAR8XOQ3hL8GYi3J/NG64=
+golang.org/x/tools v0.0.0-20190729092621-ff9f1409240a/go.mod h1:jcCCGcm9btYwXyDqrUWc6MKQKKGJCWEQ3AfLSRIbEuI=
+google.golang.org/appengine v1.3.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
+google.golang.org/appengine v1.4.0 h1:/wp5JvzpHIxhs/dumFmF7BXTf3Z+dd4uXta4kVyO508=
+google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
-gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4=
-gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
-gopkg.in/gemnasium/logrus-airbrake-hook.v2 v2.1.2 h1:OAj3g0cR6Dx/R07QgQe8wkA9RNjB2u4i700xBkIT4e0=
-gopkg.in/gemnasium/logrus-airbrake-hook.v2 v2.1.2/go.mod h1:Xk6kEKp8OKb+X14hQBKWaSkCsqBpgog8nAV2xsGOxlo=
-gopkg.in/src-d/go-billy.v4 v4.2.1/go.mod h1:tm33zBoOwxjYHZIE+OV8bxTWFMJLrconzFMd38aARFk=
-gopkg.in/src-d/go-billy.v4 v4.3.0 h1:KtlZ4c1OWbIs4jCv5ZXrTqG8EQocr0g/d4DjNg70aek=
-gopkg.in/src-d/go-billy.v4 v4.3.0/go.mod h1:tm33zBoOwxjYHZIE+OV8bxTWFMJLrconzFMd38aARFk=
-gopkg.in/src-d/go-git-fixtures.v3 v3.1.1 h1:XWW/s5W18RaJpmo1l0IYGqXKuJITWRFuA45iOf1dKJs=
-gopkg.in/src-d/go-git-fixtures.v3 v3.1.1/go.mod h1:dLBcvytrw/TYZsNTWCnkNF2DSIlzWYqTe3rJR56Ac7g=
-gopkg.in/src-d/go-git.v4 v4.9.1 h1:0oKHJZY8tM7B71378cfTg2c5jmWyNlXvestTT6WfY+4=
-gopkg.in/src-d/go-git.v4 v4.9.1/go.mod h1:Vtut8izDyrM8BUVQnzJ+YvmNcem2J89EmfZYCkLokZk=
-gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
-gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
+gopkg.in/src-d/go-billy.v4 v4.3.2 h1:0SQA1pRztfTFx2miS8sA97XvooFeNOmvUenF4o0EcVg=
+gopkg.in/src-d/go-billy.v4 v4.3.2/go.mod h1:nDjArDMp+XMs1aFAESLRjfGSgfvoYN0hDfzEk0GjC98=
+gopkg.in/src-d/go-git-fixtures.v3 v3.5.0 h1:ivZFOIltbce2Mo8IjzUHAFoq/IylO9WHhNOAJK+LsJg=
+gopkg.in/src-d/go-git-fixtures.v3 v3.5.0/go.mod h1:dLBcvytrw/TYZsNTWCnkNF2DSIlzWYqTe3rJR56Ac7g=
+gopkg.in/src-d/go-git.v4 v4.13.1 h1:SRtFyV8Kxc0UP7aCHcijOMQGPxHSmMOPrzulQWolkYE=
+gopkg.in/src-d/go-git.v4 v4.13.1/go.mod h1:nx5NYcxdKxq5fpltdHnPa2Exj4Sx0EclMWZQbYDu2z8=
 gopkg.in/warnings.v0 v0.1.2 h1:wFXVbFY8DY5/xOe1ECiWdKCzZlxgshcYVNkBHstARME=
 gopkg.in/warnings.v0 v0.1.2/go.mod h1:jksf8JmL6Qr/oQM2OXTHunEvvTAsrWBLb6OOjuVWRNI=
-gopkg.in/yaml.v2 v2.2.1 h1:mUhvW9EsL+naU5Q3cakzfE91YhliOondGd6ZrsDBHQE=
-gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
+gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=

hosts/github.go

@@ -0,0 +1,147 @@
+package hosts
+
+import (
+	"context"
+	"fmt"
+	"github.com/google/go-github/github"
+	log "github.com/sirupsen/logrus"
+	"github.com/zricethezav/gitleaks-ng/audit"
+	"github.com/zricethezav/gitleaks-ng/manager"
+	"github.com/zricethezav/gitleaks-ng/options"
+	"golang.org/x/oauth2"
+	"gopkg.in/src-d/go-git.v4"
+	"gopkg.in/src-d/go-git.v4/plumbing"
+	"gopkg.in/src-d/go-git.v4/plumbing/object"
+	"strconv"
+	"strings"
+	"sync"
+)
+
+type GithubError struct {
+	Err    string
+	Repo   string
+	Commit string
+}
+
+func (githubError *GithubError) Error() string {
+	return fmt.Sprintf("repo: %s, err: %s",
+		githubError.Repo, githubError.Err)
+}
+
+type Github struct {
+	client  *github.Client
+	errChan chan GithubError
+	manager manager.Manager
+	wg      sync.WaitGroup
+}
+
+func NewGithubClient(m manager.Manager) *Github {
+	ctx := context.Background()
+	token := oauth2.StaticTokenSource(
+		&oauth2.Token{AccessToken: options.GetAccessToken(m.Opts)},
+	)
+
+	return &Github{
+		manager: m,
+		client:  github.NewClient(oauth2.NewClient(ctx, token)),
+		errChan: make(chan GithubError),
+	}
+}
+
+// Audit will audit a github user or organization's repos.
+func (g *Github) Audit() {
+	ctx := context.Background()
+	listOptions := github.ListOptions{
+		PerPage: 100,
+	}
+
+	var githubRepos []*github.Repository
+
+	for {
+		var (
+			_githubRepos []*github.Repository
+			resp         *github.Response
+			err          error
+		)
+		if g.manager.Opts.User != "" {
+			_githubRepos, resp, err = g.client.Repositories.List(ctx, g.manager.Opts.User,
+				&github.RepositoryListOptions{ListOptions: listOptions})
+		} else if g.manager.Opts.Organization != "" {
+			_githubRepos, resp, err = g.client.Repositories.ListByOrg(ctx, g.manager.Opts.Organization,
+				&github.RepositoryListByOrgOptions{ListOptions: listOptions})
+		}
+
+		githubRepos = append(githubRepos, _githubRepos...)
+
+		if resp == nil {
+			break
+		}
+		listOptions.Page = resp.NextPage
+		if err != nil || listOptions.Page == 0 {
+			break
+		}
+	}
+
+	for _, repo := range githubRepos {
+		r := audit.NewRepo(&g.manager)
+		err := r.Clone(&git.CloneOptions{
+			URL: *repo.CloneURL,
+		})
+		r.Name = *repo.Name
+		if err != nil {
+			log.Warn(err)
+		}
+
+		if err = r.Audit(); err != nil {
+			log.Warn(err)
+		}
+	}
+}
+
+// AuditPR audits a single github PR
+func (g *Github) AuditPR() {
+	ctx := context.Background()
+	splits := strings.Split(g.manager.Opts.PullRequest, "/")
+	owner := splits[len(splits)-4]
+	repoName := splits[len(splits)-3]
+	prNum, err := strconv.Atoi(splits[len(splits)-1])
+	repo := audit.NewRepo(&g.manager)
+	repo.Name = repoName
+	log.Infof("auditing pr %s\n", g.manager.Opts.PullRequest)
+
+	if err != nil {
+		return
+	}
+	page := 1
+	for {
+		commits, resp, err := g.client.PullRequests.ListCommits(ctx, owner, repoName, prNum, &github.ListOptions{
+			PerPage: 100, Page: page})
+		if err != nil {
+			return
+		}
+		for _, c := range commits {
+			c, _, err := g.client.Repositories.GetCommit(ctx, owner, repo.Name, *c.SHA)
+			if err != nil {
+				continue
+			}
+			commitObj := object.Commit{
+				Hash: plumbing.NewHash(*c.SHA),
+				Author: object.Signature{
+					Name:  *c.Commit.Author.Name,
+					Email: *c.Commit.Author.Email,
+					When:  *c.Commit.Author.Date,
+				},
+			}
+			for _, f := range c.Files {
+				if f.Patch == nil {
+					continue
+				}
+				audit.InspectString(*f.Patch, &commitObj, repo, *f.Filename)
+			}
+		}
+		page = resp.NextPage
+		if resp.LastPage == 0 {
+			break
+		}
+	}
+}

hosts/gitlab.go

@@ -0,0 +1,101 @@
+package hosts
+
+import (
+	"context"
+	"fmt"
+	log "github.com/sirupsen/logrus"
+	"github.com/xanzy/go-gitlab"
+	"github.com/zricethezav/gitleaks-ng/audit"
+	"github.com/zricethezav/gitleaks-ng/manager"
+	"github.com/zricethezav/gitleaks-ng/options"
+	"sync"
+)
+
+type GitlabError struct {
+	Err    string
+	Repo   string
+	Commit string
+}
+
+func (gitlabError *GitlabError) Error() string {
+	return fmt.Sprintf("repo: %s, err: %s",
+		gitlabError.Repo, gitlabError.Err)
+}
+
+type Gitlab struct {
+	client  *gitlab.Client
+	errChan chan GitlabError
+	manager manager.Manager
+	ctx     context.Context
+	wg      sync.WaitGroup
+}
+
+func NewGitlabClient(m manager.Manager) *Gitlab {
+	return &Gitlab{
+		manager: m,
+		ctx:     context.Background(),
+		client:  gitlab.NewClient(nil, options.GetAccessToken(m.Opts)),
+		errChan: make(chan GitlabError),
+	}
+}
+
+// Audit will audit a github user or organization's repos.
+func (g *Gitlab) Audit() {
+	var (
+		projects []*gitlab.Project
+		resp     *gitlab.Response
+		err      error
+	)
+
+	page := 1
+	listOpts := gitlab.ListOptions{
+		PerPage: 100,
+		Page:    page,
+	}
+	for {
+		var _projects []*gitlab.Project
+		if g.manager.Opts.User != "" {
+			glOpts := &gitlab.ListProjectsOptions{
+				ListOptions: listOpts,
+			}
+			projects, resp, err = g.client.Projects.ListUserProjects(g.manager.Opts.User, glOpts)
+
+		} else if g.manager.Opts.Organization != "" {
+			glOpts := &gitlab.ListGroupProjectsOptions{
+				ListOptions: listOpts,
+			}
+			projects, resp, err = g.client.Groups.ListGroupProjects(g.manager.Opts.Organization, glOpts)
+		}
+		if err != nil {
+			log.Error(err)
+		}
+
+		projects = append(projects, _projects...)
+		if resp == nil {
+			break
+		}
+		if page >= resp.TotalPages {
+			// exit when we've seen all pages
+			break
+		}
+		page = resp.NextPage
+	}
+
+	// iterate of gitlab projects
+	for _, p := range projects {
+		r := audit.NewRepo(&g.manager)
+		cloneOpts := g.manager.CloneOptions
+		cloneOpts.URL = p.HTTPURLToRepo
+		err := r.Clone(cloneOpts)
+		r.Name = p.Name
+
+		if err = r.Audit(); err != nil {
+			log.Error(err)
+		}
+	}
+}
+
+// Audit(MR)PR TODO
+func (g *Gitlab) AuditPR() {
+	log.Error("AuditPR is not implemented in Gitlab host yet...")
+}

hosts/host.go

@@ -0,0 +1,44 @@
+package hosts
+
+import (
+	"github.com/zricethezav/gitleaks-ng/manager"
+	"strings"
+)
+
+const (
+	_github int = iota + 1
+	_gitlab
+)
+
+type Host interface {
+	Audit()
+	AuditPR()
+}
+
+func Run(m *manager.Manager) error {
+	var host Host
+	switch getHost(m.Opts.Host) {
+	case _github:
+		host = NewGithubClient(*m)
+	case _gitlab:
+		host = NewGitlabClient(*m)
+	default:
+		return nil
+	}
+
+	if m.Opts.PullRequest != "" {
+		host.AuditPR()
+	} else {
+		host.Audit()
+	}
+	return nil
+}
+
+func getHost(host string) int {
+	if strings.ToLower(host) == "github" {
+		return _github
+	} else if strings.ToLower(host) == "gitlab" {
+		return _gitlab
+	}
+	return -1
+}

hosts/hosts_test.go

@@ -0,0 +1,115 @@
+package hosts
+
+import (
+	"flag"
+	"fmt"
+	"github.com/zricethezav/gitleaks-ng/config"
+	"github.com/zricethezav/gitleaks-ng/manager"
+	"github.com/zricethezav/gitleaks-ng/options"
+	"os"
+	"testing"
+)
+
+var (
+	integration = flag.Bool("integration", false, "run github/gitlab integration test")
+)
+
+func TestGithub(t *testing.T) {
+	flag.Parse()
+	if !*integration {
+		fmt.Println("skipping github integration tests")
+		return
+	}
+	if os.Getenv("GITHUB_TOKEN") == "" {
+		t.Log("skipping github integration tests, need env var GITLAB_TOKEN")
+		return
+	}
+
+	tests := []struct {
+		opts         options.Options
+		desiredLeaks int
+	}{
+		{
+			opts: options.Options{
+				Host: "github",
+				User: "gitleakstest",
+				AccessToken: os.Getenv("GITHUB_TOKEN"),
+			},
+			desiredLeaks: 2,
+		},
+		{
+			opts: options.Options{
+				Host: "github",
+				PullRequest: "https://github.com/gitleakstest/gronit/pull/1",
+				AccessToken: os.Getenv("GITHUB_TOKEN"),
+			},
+			desiredLeaks: 4,
+		},
+	}
+
+	for _, test := range tests {
+		cfg, err := config.NewConfig(test.opts)
+		if err != nil {
+			t.Error(err)
+		}
+
+		m, err := manager.NewManager(test.opts, cfg)
+		if err != nil {
+			t.Error(err)
+		}
+		err = Run(m)
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		if test.desiredLeaks != len(m.GetLeaks()) {
+			t.Errorf("got %d leaks, want %d", len(m.GetLeaks()), test.desiredLeaks)
+		}
+	}
+}
+
+func TestGitlab(t *testing.T) {
+	flag.Parse()
+	if !*integration {
+		fmt.Println("skipping gitlab integration tests")
+		return
+	}
+	if os.Getenv("GITLAB_TOKEN") == "" {
+		t.Log("skipping github integration tests, need env var GITLAB_TOKEN")
+		return
+	}
+
+	tests := []struct {
+		opts         options.Options
+		desiredLeaks int
+	}{
+		{
+			opts: options.Options{
+				Host: "gitlab",
+				User: "gitleakstest",
+				AccessToken: os.Getenv("GITLAB_TOKEN"),
+			},
+			desiredLeaks: 2,
+		},
+	}
+
+	for _, test := range tests {
+		cfg, err := config.NewConfig(test.opts)
+		if err != nil {
+			t.Error(err)
+		}
+
+		m, err := manager.NewManager(test.opts, cfg)
+		if err != nil {
+			t.Error(err)
+		}
+		err = Run(m)
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		if test.desiredLeaks != len(m.GetLeaks()) {
+			t.Errorf("got %d leaks, want %d", len(m.GetLeaks()), test.desiredLeaks)
+		}
+	}
+}

main.go

@@ -1,27 +1,96 @@
 package main
 
 import (
-	"os"
-	"strings"
-
+	"github.com/hako/durafmt"
 	log "github.com/sirupsen/logrus"
-	"github.com/zricethezav/gitleaks/src"
+	"github.com/zricethezav/gitleaks-ng/audit"
+	"github.com/zricethezav/gitleaks-ng/config"
+	"github.com/zricethezav/gitleaks-ng/hosts"
+	"github.com/zricethezav/gitleaks-ng/manager"
+	"github.com/zricethezav/gitleaks-ng/options"
+	"io/ioutil"
+	"os"
+	"time"
 )
 
+// TODO documentation for
+// 1. ./gitleaks-ng --repo=https://github.com/gitleakstest/gronit -v | jq -R 'fromjson?'
+// 2. Dockerfile
+// 3. need to add tests for --repo-config
+// 4. look over comments and code
+// 5. prepare release
+
 func main() {
-	leakCount, err := gitleaks.Run(gitleaks.ParseOpts())
+	opts, err := options.ParseOptions()
+	if err != nil {
+		log.Error(err)
+		os.Exit(options.ErrorEncountered)
+	}
+
+	err = opts.Guard()
+	if err != nil {
+		log.Error(err)
+		os.Exit(options.ErrorEncountered)
+	}
+
+	cfg, err := config.NewConfig(opts)
+	if err != nil {
+		log.Error(err)
+		os.Exit(options.ErrorEncountered)
+	}
+
+	m, err := manager.NewManager(opts, cfg)
 	if err != nil {
-		if strings.Contains(err.Error(), "whitelisted") {
-			log.Info(err.Error())
-			os.Exit(0)
-		}
 		log.Error(err)
-		os.Exit(gitleaks.ErrExit)
+		os.Exit(options.ErrorEncountered)
 	}
 
-	if leakCount == 0 {
-		os.Exit(gitleaks.NoLeaks)
+	err = Run(m)
+	if err != nil {
+		log.Error(err)
+		os.Exit(options.ErrorEncountered)
+	}
+
+	leaks := m.GetLeaks()
+	metadata := m.GetMetadata()
+
+	if len(m.GetLeaks()) != 0 {
+		log.Warnf("%d leaks detected. %d commits audited in %s", len(leaks),
+			metadata.Commits, durafmt.Parse(time.Duration(metadata.AuditTime)*time.Nanosecond))
+		os.Exit(options.LeaksPresent)
+	} else {
+		log.Infof("No leaks detected. %d commits audited in %s",
+			metadata.Commits, durafmt.Parse(time.Duration(metadata.AuditTime)*time.Nanosecond))
+	}
+
+	os.Exit(options.Success)
+}
+
+// Run begins the program and contains some basic logic on how to continue with the audit. If any external git host
+// options are set (like auditing a gitlab or github user) then a specific host client will be created and
+// then Audit() and Report() will be called. Otherwise, gitleaks will create a new repo and an audit will proceed.
+// If no options or the uncommitted option is set then a pre-commit audit will
+// take place -- this is similar to running `git diff` on all the tracked files.
+// TODO handle errors from errChan
+func Run(m *manager.Manager) error {
+	if m.Opts.Disk {
+		dir, err := ioutil.TempDir("", "gitleaks")
+		defer os.RemoveAll(dir)
+		if err != nil {
+			return err
+		}
+		m.CloneDir = dir
+	}
+
+	var err error
+	if m.Opts.Host != "" {
+		err = hosts.Run(m)
 	} else {
-		os.Exit(gitleaks.LeakExit)
+		err = audit.Run(m)
+	}
+	if err != nil {
+		return err
 	}
+
+	return m.Report()
 }

manager/manager.go

@@ -0,0 +1,246 @@
+package manager
+
+import (
+	"encoding/json"
+	"fmt"
+	"github.com/hako/durafmt"
+	"github.com/mattn/go-colorable"
+	log "github.com/sirupsen/logrus"
+	"github.com/zricethezav/gitleaks-ng/config"
+	"github.com/zricethezav/gitleaks-ng/options"
+	"gopkg.in/src-d/go-git.v4"
+	"os"
+	"os/signal"
+	"runtime"
+	"sync"
+	"text/tabwriter"
+	"time"
+)
+
+// Manager is a struct containing options and configs as well CloneOptions and CloneDir.
+// This struct is passed into each NewRepo so we are not passing around the manager in func params.
+type Manager struct {
+	Opts   options.Options
+	Config config.Config
+
+	CloneOptions *git.CloneOptions
+	CloneDir     string
+
+	leaks    []Leak
+	leakChan chan Leak
+	leakWG   *sync.WaitGroup
+
+	stopChan chan os.Signal
+	metadata Metadata
+}
+
+type Leak struct {
+	Line     string    `json:"line"`
+	Offender string    `json:"offender"`
+	Commit   string    `json:"commit"`
+	Repo     string    `json:"repo"`
+	Rule     string    `json:"rule"`
+	Message  string    `json:"commitMessage"`
+	Author   string    `json:"author"`
+	Email    string    `json:"email"`
+	File     string    `json:"file"`
+	Date     time.Time `json:"date"`
+	Tags     string    `json:"tags"`
+	Severity string    `json:"severity"`
+}
+
+type AuditTime int64
+type PatchTime int64
+type CloneTime int64
+type RegexTime struct {
+	Time  int64
+	Regex string
+}
+
+type Metadata struct {
+	mux  sync.Mutex
+	data map[string]interface{}
+
+	timings chan interface{}
+
+	RegexTime map[string]int64
+	Commits   int
+	AuditTime int64
+	patchTime int64
+	cloneTime int64
+}
+
+func init() {
+	log.SetOutput(os.Stdout)
+	log.SetFormatter(&log.TextFormatter{
+		ForceColors:   true,
+		FullTimestamp: true,
+	})
+	// Fix colors on Windows
+	if runtime.GOOS == "windows" {
+		log.SetOutput(colorable.NewColorableStdout())
+	}
+}
+
+// GetLeaks returns all available leaks
+func (manager *Manager) GetLeaks() []Leak {
+	// need to wait for any straggling leaks
+	manager.leakWG.Wait()
+	return manager.leaks
+}
+
+// SendLeaks accepts a leak and is used by the audit pkg. This is the public function
+// that allows other packages to send leaks to the manager.
+func (manager *Manager) SendLeaks(l Leak) {
+	manager.leakWG.Add(1)
+	manager.leakChan <- l
+}
+
+// receiveLeaks listens to leakChan for incoming leaks. If any are received, they are appended to the
+// manager's leaks for future reporting. If the -v/--verbose option is set the leaks will marshaled into
+// json and printed out.
+func (manager *Manager) receiveLeaks() {
+	for leak := range manager.leakChan {
+		manager.leaks = append(manager.leaks, leak)
+		if manager.Opts.Verbose {
+			b, _ := json.Marshal(leak)
+			fmt.Println(string(b))
+		}
+		manager.leakWG.Done()
+	}
+}
+
+func (manager *Manager) GetMetadata() Metadata {
+	return manager.metadata
+}
+
+// receiveMetadata is where the messages sent to the metadata channel get consumed. You can view metadata
+// by running gitleaks with the --debug option set. This is extremely useful when trying to optimize regular
+// expressions as that what gitleaks spends most of its cycles on.
+func (manager *Manager) receiveMetadata() {
+	for t := range manager.metadata.timings {
+		switch ti := t.(type) {
+		case CloneTime:
+			manager.metadata.cloneTime += int64(ti)
+		case AuditTime:
+			manager.metadata.AuditTime += int64(ti)
+		case PatchTime:
+			manager.metadata.patchTime += int64(ti)
+		case RegexTime:
+			manager.metadata.RegexTime[ti.Regex] = manager.metadata.RegexTime[ti.Regex] + ti.Time
+		}
+	}
+}
+
+func (manager *Manager) IncrementCommits(i int) {
+	manager.metadata.mux.Lock()
+	manager.metadata.Commits += i
+	manager.metadata.mux.Unlock()
+}
+
+// RecordTime accepts an interface and sends it to the manager's time channel
+func (manager *Manager) RecordTime(t interface{}) {
+	manager.metadata.timings <- t
+}
+
+// NewManager accepts options and returns a manager struct. The manager is a container for gitleaks configurations,
+// options and channel receivers.
+func NewManager(opts options.Options, cfg config.Config) (*Manager, error) {
+	cloneOpts, err := opts.CloneOptions()
+	if err != nil {
+		return nil, err
+	}
+
+	m := &Manager{
+		Opts:         opts,
+		Config:       cfg,
+		CloneOptions: cloneOpts,
+
+		stopChan: make(chan os.Signal, 1),
+		leakChan: make(chan Leak),
+		leakWG:   &sync.WaitGroup{},
+		metadata: Metadata{
+			RegexTime: make(map[string]int64),
+			timings:   make(chan interface{}),
+			data:      make(map[string]interface{}),
+		},
+	}
+
+	signal.Notify(m.stopChan, os.Interrupt)
+
+	// start receiving leaks and metadata
+	go m.receiveLeaks()
+	go m.receiveMetadata()
+	go m.receiveInterrupt()
+
+	return m, nil
+}
+
+// DebugOutput logs metadata and other messages that occurred during a gitleaks audit
+func (manager *Manager) DebugOutput() {
+	log.Debugf("-------------------------\n")
+	log.Debugf("| Times and Commit Counts|\n")
+	log.Debugf("-------------------------\n")
+	fmt.Println("totalAuditTime: ", durafmt.Parse(time.Duration(manager.metadata.AuditTime)*time.Nanosecond))
+	fmt.Println("totalPatchTime: ", durafmt.Parse(time.Duration(manager.metadata.patchTime)*time.Nanosecond))
+	fmt.Println("totalCloneTime: ", durafmt.Parse(time.Duration(manager.metadata.cloneTime)*time.Nanosecond))
+	fmt.Println("totalCommits: ", manager.metadata.Commits)
+
+	const padding = 6
+	w := tabwriter.NewWriter(os.Stdout, 0, 0, padding, '.', 0)
+
+	log.Debugf("--------------------------\n")
+	log.Debugf("| Individual Regex Times |\n")
+	log.Debugf("--------------------------\n")
+	for k, v := range manager.metadata.RegexTime {
+		fmt.Fprintf(w, "%s\t%s\n", k, durafmt.Parse(time.Duration(v)*time.Nanosecond))
+	}
+	w.Flush()
+
+}
+
+// Report saves gitleaks leaks to a json specified by --report={report.json}
+func (manager *Manager) Report() error {
+	close(manager.leakChan)
+	close(manager.metadata.timings)
+
+	if log.IsLevelEnabled(log.DebugLevel) {
+		manager.DebugOutput()
+	}
+
+	if manager.Opts.Report != "" {
+		if len(manager.GetLeaks()) == 0 {
+			log.Infof("no leaks found, skipping writing report")
+			return nil
+		}
+		file, err := os.Create(manager.Opts.Report)
+		if err != nil {
+			return err
+		}
+
+		encoder := json.NewEncoder(file)
+		encoder.SetIndent("", " ")
+		err = encoder.Encode(manager.leaks)
+		if err != nil {
+			return err
+		}
+		err = file.Close()
+		if err != nil {
+			return err
+		}
+		log.Infof("report written to %s", manager.Opts.Report)
+	}
+	return nil
+}
+
+func (manager *Manager) receiveInterrupt() {
+	<-manager.stopChan
+	if manager.Opts.Report != "" {
+		err := manager.Report()
+		if err != nil {
+			log.Error(err)
+		}
+	}
+	log.Info("gitleaks received interrupt, stopping audit")
+	os.Exit(options.ErrorEncountered)
+}

manager/manager_test.go

@@ -0,0 +1,89 @@
+package manager
+
+import (
+	"github.com/zricethezav/gitleaks-ng/config"
+	"github.com/zricethezav/gitleaks-ng/options"
+	"testing"
+)
+
+
+// TODO
+// add more substantial tests... but since literally every pkg uses manager
+// these tests are kind of redundant
+func TestSendReceiveLeaks(t *testing.T) {
+
+	tests := []struct {
+		leaksToAdd int
+		goRoutines int
+	}{
+		{
+			leaksToAdd: 10,
+		},
+		{
+			leaksToAdd: 1000,
+		},
+	}
+	for _, test := range tests {
+		opts := options.Options{}
+		cfg, _ := config.NewConfig(opts)
+		m, _ := NewManager(opts, cfg)
+
+		for i := 0; i < test.leaksToAdd; i++ {
+			m.SendLeaks(Leak{})
+		}
+		got := m.GetLeaks()
+		if len(got) != test.leaksToAdd {
+			t.Errorf("got %d, wanted %d leaks", len(got), test.leaksToAdd)
+		}
+	}
+}
+
+func TestSendReceiveMeta(t *testing.T) {
+	tests := []struct {
+		auditTime  int64
+		patchTime  int64
+		cloneTime  int64
+		regexTime  int64
+		iterations int
+	}{
+		{
+			auditTime:  1000,
+			patchTime:  1000,
+			cloneTime:  1000,
+			regexTime:  1000,
+			iterations: 100,
+		},
+	}
+	for _, test := range tests {
+		opts := options.Options{}
+		cfg, _ := config.NewConfig(opts)
+		m, _ := NewManager(opts, cfg)
+
+		for i := 0; i < test.iterations; i++ {
+			m.RecordTime(AuditTime(test.auditTime))
+			m.RecordTime(PatchTime(test.patchTime))
+			m.RecordTime(CloneTime(test.cloneTime))
+			m.RecordTime(RegexTime{
+				Regex: "regex",
+				Time:  test.regexTime,
+			})
+			m.RecordTime(RegexTime{
+				Regex: "regex2",
+				Time:  test.regexTime,
+			})
+		}
+		md := m.GetMetadata()
+		if md.cloneTime != test.cloneTime * int64(test.iterations) {
+			t.Errorf("clone time mismatch, got %d, wanted %d",
+				md.cloneTime, test.cloneTime * int64(test.iterations))
+		}
+		if md.AuditTime != test.auditTime * int64(test.iterations) {
+			t.Errorf("audit time mismatch, got %d, wanted %d",
+				md.AuditTime, test.auditTime * int64(test.iterations))
+		}
+		if md.patchTime != test.patchTime * int64(test.iterations) {
+			t.Errorf("clone time mismatch, got %d, wanted %d",
+				md.patchTime, test.patchTime * int64(test.iterations))
+		}
+	}
+}

options/options.go

@@ -0,0 +1,206 @@
+package options
+
+import (
+	"fmt"
+	"github.com/jessevdk/go-flags"
+	log "github.com/sirupsen/logrus"
+	"github.com/zricethezav/gitleaks-ng/version"
+	"gopkg.in/src-d/go-git.v4"
+	"gopkg.in/src-d/go-git.v4/plumbing/transport/http"
+	"gopkg.in/src-d/go-git.v4/plumbing/transport/ssh"
+	"io/ioutil"
+	"os"
+	"os/user"
+	"strings"
+)
+
+const (
+	Success int = iota + 1
+	LeaksPresent
+	ErrorEncountered
+)
+
+type Options struct {
+	Verbose     bool   `short:"v" long:"verbose" description:"Show verbose output from audit"`
+	Repo        string `short:"r" long:"repo" description:"Target repository"`
+	Config      string `long:"config" description:"config path"`
+	Disk        bool   `long:"disk" description:"Clones repo(s) to disk"`
+	Version     bool   `long:"version" description:"version number"`
+	Timeout     int    `long:"timeout" description:"Timeout (s)"`
+	Username    string `long:"username" description:"Username for git repo"`
+	Password    string `long:"password" description:"Password for git repo"`
+	AccessToken string `long:"access-token" description:"Access token for git repo"`
+	Commit      string `long:"commit" description:"sha of commit to audit"`
+	Threads     int    `long:"threads" description:"Maximum number of threads gitleaks spawns"`
+	SSH         string `long:"ssh-key" description:"path to ssh key used for auth"`
+	Uncommited  bool   `long:"uncommitted" description:"run gitleaks on uncommitted code"`
+	RepoPath    string `long:"repo-path" description:"Path to repo"`
+	OwnerPath   string `long:"owner-path" description:"Path to owner directory (repos discovered)"`
+	Branch      string `long:"branch" description:"Branch to audit"`
+	Report      string `long:"report" description:"path to write json leaks file"`
+	Redact      bool   `long:"redact" description:"redact secrets from log messages and leaks"`
+	Debug       bool   `long:"debug" description:"log debug messages"`
+	RepoConfig   bool   `long:"repo-config" description:"Load config from target repo. Config file must be \".gitleaks.toml\" or \"gitleaks.toml\""`
+
+	// Hosts
+	Host         string `long:"host" description:"git hosting service like gitlab or github. Supported hosts include: Github, Gitlab"`
+	Organization string `long:"org" description:"organization to audit"`
+	User         string `long:"user" description:"user to audit"` //work
+	PullRequest  string `long:"pr" description:"pull/merge request url"`
+}
+
+// ParseOptions is responsible for parsing options passed in by cli. An Options struct
+// is returned if successful. This struct is passed around the program
+// and will determine how the program executes. If err, an err message or help message
+// will be displayed and the program will exit with code 0.
+func ParseOptions() (Options, error) {
+	var opts Options
+	parser := flags.NewParser(&opts, flags.Default)
+	_, err := parser.Parse()
+
+	if err != nil {
+		parser.WriteHelp(os.Stdout)
+		os.Exit(Success)
+	}
+
+	if opts.Version {
+		fmt.Printf("%s\n", version.Version)
+		os.Exit(Success)
+	}
+
+	if opts.Debug {
+		log.SetLevel(log.DebugLevel)
+	}
+
+	return opts, nil
+}
+
+// Guard checks to makes sure there are no invalid options set.
+// If invalid sets of options are present, a descriptive error will return
+// else nil is returned
+func (opts Options) Guard() error {
+	// 1. only one target option set at a time:
+	// repo, owner-path, repo-path
+	return nil
+}
+
+// cloneOptions returns a git.cloneOptions pointer. The authentication method
+// is determined by what is passed in via command-Line options. If No
+// Username/PW or AccessToken is available and the repo target is not using the
+// git protocol then the repo must be a available via no auth.
+func (opts Options) CloneOptions() (*git.CloneOptions, error) {
+	progress := ioutil.Discard
+	if opts.Verbose {
+		progress = os.Stdout
+	}
+
+	if strings.HasPrefix(opts.Repo, "git") {
+		// using git protocol so needs ssh auth
+		auth, err := sshAuth(opts)
+		if err != nil {
+			return nil, err
+		}
+		return &git.CloneOptions{
+			URL:      opts.Repo,
+			Auth:     auth,
+			Progress: progress,
+		}, nil
+	}
+	if opts.Password != "" && opts.Username != "" {
+		// auth using username and password
+		return &git.CloneOptions{
+			URL: opts.Repo,
+			Auth: &http.BasicAuth{
+				Username: opts.Username,
+				Password: opts.Password,
+			},
+			Progress: progress,
+		}, nil
+	}
+	if opts.AccessToken != "" {
+		return &git.CloneOptions{
+			URL: opts.Repo,
+			Auth: &http.BasicAuth{
+				Username: "gitleaks_user",
+				Password: opts.AccessToken,
+			},
+			Progress: progress,
+		}, nil
+	}
+	if os.Getenv("GITLEAKS_ACCESS_TOKEN") != "" {
+		return &git.CloneOptions{
+			URL: opts.Repo,
+			Auth: &http.BasicAuth{
+				Username: "gitleaks_user",
+				Password: os.Getenv("GITLEAKS_ACCESS_TOKEN"),
+			},
+			Progress: progress,
+		}, nil
+	}
+
+	// No Auth, publicly available
+	return &git.CloneOptions{
+		URL:      opts.Repo,
+		Progress: progress,
+	}, nil
+}
+
+// sshAuth tried to generate ssh public keys based on what was passed via cli. If no
+// path was passed via cli then this will attempt to retrieve keys from the default
+// location for ssh keys, $HOME/.ssh/id_rsa. This function is only called if the
+// repo url using the git:// protocol.
+func sshAuth(opts Options) (*ssh.PublicKeys, error) {
+	if opts.SSH != "" {
+		return ssh.NewPublicKeysFromFile("git", opts.SSH, "")
+	}
+	c, err := user.Current()
+	if err != nil {
+		return nil, err
+	}
+	defaultPath := fmt.Sprintf("%s/.ssh/id_rsa", c.HomeDir)
+	return ssh.NewPublicKeysFromFile("git", defaultPath, "")
+}
+
+// openLocal checks what options are set, if no remote targets are set
+// then return true
+func (opts Options) OpenLocal() bool {
+	if opts.Uncommited || opts.RepoPath != "" || opts.Repo == "" {
+		return true
+	}
+	return false
+}
+
+// CheckUncommitted returns a boolean that indicates whether or not gitleaks should check unstaged pre-commit changes
+// or if gitleaks should check the entire git history
+func (opts Options) CheckUncommitted() bool {
+	// check to make sure no remote shit is set
+	if opts.Uncommited {
+		return true
+	}
+	if opts == (Options{}) {
+		return true
+	}
+	if opts.Repo != "" {
+		return false
+	}
+	if opts.RepoPath != "" {
+		return false
+	}
+	if opts.OwnerPath != "" {
+		return false
+	}
+	if opts.Host != "" {
+		return false
+	}
+	return true
+}
+
+// GetAccessToken accepts options and returns a string which is the access token to a git host.
+// Setting this option or environment var is necessary if performing an audit with any of the git hosting providers
+// in the host pkg. The access token set by cli options takes precedence over env vars.
+func GetAccessToken(opts Options) string {
+	if opts.AccessToken != "" {
+		return opts.AccessToken
+	}
+	return os.Getenv("GITLEAKS_ACCESS_TOKEN")
+}

options/options_test.go

@@ -0,0 +1 @@
+package options

src/config.go

@@ -1,234 +0,0 @@
-package gitleaks
-
-import (
-	"fmt"
-	"os"
-	"os/user"
-	"regexp"
-	"strconv"
-	"strings"
-
-	"github.com/BurntSushi/toml"
-	log "github.com/sirupsen/logrus"
-	"gopkg.in/src-d/go-git.v4/plumbing/transport/ssh"
-)
-
-type entropyRange struct {
-	v1 float64
-	v2 float64
-}
-
-// Rule instructs how gitleaks should audit each line of code
-type Rule struct {
-	description string
-	regex       *regexp.Regexp
-	severity    string
-	tags        []string
-	entropies   []*entropyRange
-	entropyROI  string
-	fileTypes   []*regexp.Regexp
-}
-
-// TomlConfig is used for loading gitleaks configs from a toml file
-type TomlConfig struct {
-	Rules []struct {
-		Description string
-		Regex       string
-		Entropies   []string
-		Tags        []string
-		Severity    string
-		EntropyROI  string
-		FileTypes   []string
-	}
-	Whitelist struct {
-		Files   []string
-		Regexes []string
-		Commits []string
-		Repos   []string
-	}
-}
-
-// Config contains gitleaks config
-type Config struct {
-	Rules     []*Rule
-	WhiteList struct {
-		regexes []*regexp.Regexp
-		files   []*regexp.Regexp
-		commits map[string]bool
-		repos   []*regexp.Regexp
-	}
-	FileRules []*Rule
-	sshAuth   *ssh.PublicKeys
-}
-
-// loadToml loads of the toml config containing regexes and whitelists.
-// This function will first look if the configPath is set and load the config
-// from that file. Otherwise will then look for the path set by the GITHLEAKS_CONIFG
-// env var. If that is not set, then gitleaks will continue with the default configs
-// specified by the const var at the top `defaultConfig`
-func newConfig() (*Config, error) {
-	var (
-		tomlConfig TomlConfig
-		configPath string
-		config     Config
-	)
-
-	if opts.ConfigPath != "" {
-		configPath = opts.ConfigPath
-		_, err := os.Stat(configPath)
-		if err != nil {
-			return nil, fmt.Errorf("no gitleaks config at %s", configPath)
-		}
-	} else {
-		configPath = os.Getenv("GITLEAKS_CONFIG")
-	}
-
-	if configPath != "" {
-		if _, err := toml.DecodeFile(configPath, &tomlConfig); err != nil {
-			return nil, fmt.Errorf("problem loading config: %v", err)
-		}
-	} else {
-		_, err := toml.Decode(defaultConfig, &tomlConfig)
-		if err != nil {
-			return nil, fmt.Errorf("problem loading default config: %v", err)
-		}
-	}
-
-	sshAuth, err := getSSHAuth()
-	if err != nil {
-		return nil, err
-	}
-	config.sshAuth = sshAuth
-
-	err = config.update(tomlConfig)
-	if err != nil {
-		return nil, err
-	}
-	return &config, err
-}
-
-// updateConfig will update a the global config values
-func (config *Config) update(tomlConfig TomlConfig) error {
-	for _, rule := range tomlConfig.Rules {
-		re := regexp.MustCompile(rule.Regex)
-		ranges, err := getEntropyRanges(rule.Entropies)
-		var fileTypes = []*regexp.Regexp{}
-		for _, regex := range rule.FileTypes {
-			fileTypes = append(fileTypes, regexp.MustCompile(regex))
-		}
-
-		if err != nil {
-			log.Errorf("could not create entropy range for %s, skipping rule", rule.Description)
-			continue
-		}
-
-		r := &Rule{
-			description: rule.Description,
-			regex:       re,
-			severity:    rule.Severity,
-			tags:        rule.Tags,
-			entropies:   ranges,
-			entropyROI:  rule.EntropyROI,
-			fileTypes:   fileTypes,
-		}
-
-		if len(rule.Entropies) == 0 && rule.Regex == "" && len(fileTypes) != 0 {
-			config.FileRules = append(config.FileRules, r)
-		}
-		config.Rules = append(config.Rules, r)
-
-	}
-
-	// set whitelists
-	config.WhiteList.commits = make(map[string]bool)
-	for _, commit := range tomlConfig.Whitelist.Commits {
-		config.WhiteList.commits[commit] = true
-	}
-	for _, regex := range tomlConfig.Whitelist.Files {
-		config.WhiteList.files = append(config.WhiteList.files, regexp.MustCompile(regex))
-	}
-	for _, regex := range tomlConfig.Whitelist.Regexes {
-		config.WhiteList.regexes = append(config.WhiteList.regexes, regexp.MustCompile(regex))
-	}
-	for _, regex := range tomlConfig.Whitelist.Repos {
-		config.WhiteList.repos = append(config.WhiteList.repos, regexp.MustCompile(regex))
-	}
-
-	return nil
-}
-
-// entropyRanges hydrates entropyRanges which allows for fine tuning entropy checking
-func getEntropyRanges(entropyLimitStr []string) ([]*entropyRange, error) {
-	var ranges []*entropyRange
-	for _, span := range entropyLimitStr {
-		split := strings.Split(span, "-")
-		v1, err := strconv.ParseFloat(split[0], 64)
-		if err != nil {
-			return nil, err
-		}
-		v2, err := strconv.ParseFloat(split[1], 64)
-		if err != nil {
-			return nil, err
-		}
-		if v1 > v2 {
-			return nil, fmt.Errorf("entropy range must be ascending")
-		}
-		r := &entropyRange{
-			v1: v1,
-			v2: v2,
-		}
-		if r.v1 > 8.0 || r.v1 < 0.0 || r.v2 > 8.0 || r.v2 < 0.0 {
-			return nil, fmt.Errorf("invalid entropy ranges, must be within 0.0-8.0")
-		}
-		ranges = append(ranges, r)
-	}
-	return ranges, nil
-}
-
-// externalConfig will attempt to load a pinned ".gitleaks.toml" configuration file
-// from a remote or local repo. Use the --repo-config option to trigger this.
-func (config *Config) updateFromRepo(repo *Repo) error {
-	var tomlConfig TomlConfig
-	wt, err := repo.repository.Worktree()
-	if err != nil {
-		return err
-	}
-	f, err := wt.Filesystem.Open(".gitleaks.toml")
-	if err != nil {
-		return fmt.Errorf("problem loading config: %v", err)
-	}
-	defer f.Close()
-	if _, err := toml.DecodeReader(f, &tomlConfig); err != nil {
-		return fmt.Errorf("problem loading config: %v", err)
-	}
-
-	return config.update(tomlConfig)
-}
-
-// getSSHAuth return an ssh auth use by go-git to clone repos behind authentication.
-// If --ssh-key is set then it will attempt to load the key from that path. If not,
-// gitleaks will use the default $HOME/.ssh/id_rsa key
-func getSSHAuth() (*ssh.PublicKeys, error) {
-	var (
-		sshKeyPath string
-	)
-	if opts.SSHKey != "" {
-		sshKeyPath = opts.SSHKey
-	} else {
-		// try grabbing default
-		c, err := user.Current()
-		if err != nil {
-			return nil, nil
-		}
-		sshKeyPath = fmt.Sprintf("%s/.ssh/id_rsa", c.HomeDir)
-	}
-	sshAuth, err := ssh.NewPublicKeysFromFile("git", sshKeyPath, "")
-	if err != nil {
-		if strings.HasPrefix(opts.Repo, "git") {
-			// if you are attempting to clone a git repo via ssh and supply a bad ssh key,
-			// the clone will fail.
-			return nil, fmt.Errorf("unable to generate ssh key: %v", err)
-		}
-	}
-	return sshAuth, nil
-}

src/constants.go

@@ -1,207 +0,0 @@
-package gitleaks
-
-const version = "2.1.0"
-
-const NoLeaks = 0
-
-const defaultGithubURL = "https://api.github.com/"
-const defaultThreadNum = 1
-
-// ErrExit used to signal an error during gitleaks execution
-const ErrExit = 2
-
-// LeakExit used to signal leaks present in audit
-const LeakExit = 1
-
-const defaultConfig = `
-# This is a sample config file for gitleaks. You can configure gitleaks what to search for and what to whitelist.
-# The output you are seeing here is the default gitleaks config. If GITLEAKS_CONFIG environment variable
-# is set, gitleaks will load configurations from that path. If option --config is set, gitleaks will load
-# configurations from that path. Gitleaks does not whitelist anything by default.
-# - https://www.ndss-symposium.org/wp-content/uploads/2019/02/ndss2019_04B-3_Meli_paper.pdf
-# - https://github.com/dxa4481/truffleHogRegexes/blob/master/truffleHogRegexes/regexes.json
-
-title = "gitleaks config"
-[[rules]]
-description = "AWS Client ID"
-regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
-tags = ["key", "AWS"]
-
-[[rules]]
-description = "AWS Secret Key"
-regex = '''(?i)aws(.{0,20})?(?-i)['\"][0-9a-zA-Z\/+]{40}['\"]'''
-tags = ["key", "AWS"]
-
-[[rules]]
-description = "AWS MWS key"
-regex = '''amzn\.mws\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'''
-tags = ["key", "AWS", "MWS"]
-
-[[rules]]
-description = "PKCS8"
-regex = '''-----BEGIN PRIVATE KEY-----'''
-tags = ["key", "PKCS8"]
-
-[[rules]]
-description = "RSA"
-regex = '''-----BEGIN RSA PRIVATE KEY-----'''
-tags = ["key", "RSA"]
-
-[[rules]]
-description = "SSH"
-regex = '''-----BEGIN OPENSSH PRIVATE KEY-----'''
-tags = ["key", "SSH"]
-
-[[rules]]
-description = "PGP"
-regex = '''-----BEGIN PGP PRIVATE KEY BLOCK-----'''
-tags = ["key", "PGP"]
-
-[[rules]]
-description = "Facebook Secret Key"
-regex = '''(?i)(facebook|fb)(.{0,20})?(?-i)['\"][0-9a-f]{32}['\"]'''
-tags = ["key", "Facebook"]
-
-[[rules]]
-description = "Facebook Client ID"
-regex = '''(?i)(facebook|fb)(.{0,20})?['\"][0-9]{13,17}['\"]'''
-tags = ["key", "Facebook"]
-
-[[rules]]
-description = "Facebook access token"
-regex = '''EAACEdEose0cBA[0-9A-Za-z]+'''
-tags = ["key", "Facebook"]
-
-[[rules]]
-description = "Twitter Secret Key"
-regex = '''(?i)twitter(.{0,20})?['\"][0-9a-z]{35,44}['\"]'''
-tags = ["key", "Twitter"]
-
-[[rules]]
-description = "Twitter Client ID"
-regex = '''(?i)twitter(.{0,20})?['\"][0-9a-z]{18,25}['\"]'''
-tags = ["client", "Twitter"]
-
-[[rules]]
-description = "Github"
-regex = '''(?i)github(.{0,20})?(?-i)['\"][0-9a-zA-Z]{35,40}['\"]'''
-tags = ["key", "Github"]
-
-[[rules]]
-description = "LinkedIn Client ID"
-regex = '''(?i)linkedin(.{0,20})?(?-i)['\"][0-9a-z]{12}['\"]'''
-tags = ["client", "LinkedIn"]
-
-[[rules]]
-description = "LinkedIn Secret Key"
-regex = '''(?i)linkedin(.{0,20})?['\"][0-9a-z]{16}['\"]'''
-tags = ["secret", "LinkedIn"]
-
-[[rules]]
-description = "Slack"
-regex = '''xox[baprs]-([0-9a-zA-Z]{10,48})?'''
-tags = ["key", "Slack"]
-
-[[rules]]
-description = "EC"
-regex = '''-----BEGIN EC PRIVATE KEY-----'''
-tags = ["key", "EC"]
-
-[[rules]]
-description = "Generic API key"
-regex = '''(?i)(api_key|apikey)(.{0,20})?['|"][0-9a-zA-Z]{32,45}['|"]'''
-tags = ["key", "API", "generic"]
-
-[[rules]]
-description = "Generic Secret"
-regex = '''(?i)secret(.{0,20})?['|"][0-9a-zA-Z]{32,45}['|"]'''
-tags = ["key", "Secret", "generic"]
-
-[[rules]]
-description = "Google API key"
-regex = '''AIza[0-9A-Za-z\\-_]{35}'''
-tags = ["key", "Google"]
-
-[[rules]]
-description = "Google Cloud Platform API key"
-regex = '''(?i)(google|gcp|youtube|drive|yt)(.{0,20})?['\"][AIza[0-9a-z\\-_]{35}]['\"]'''
-tags = ["key", "Google", "GCP"]
-
-[[rules]]
-description = "Google OAuth"
-regex = '''(?i)(google|gcp|auth)(.{0,20})?['"][0-9]+-[0-9a-z_]{32}\.apps\.googleusercontent\.com['"]'''
-tags = ["key", "Google", "OAuth"]
-
-[[rules]]
-description = "Google OAuth access token"
-regex = '''ya29\.[0-9A-Za-z\-_]+'''
-tags = ["key", "Google", "OAuth"]
-
-[[rules]]
-description = "Heroku API key"
-regex = '''(?i)heroku(.{0,20})?['"][0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}['"]'''
-tags = ["key", "Heroku"]
-
-[[rules]]
-description = "MailChimp API key"
-regex = '''(?i)(mailchimp|mc)(.{0,20})?['"][0-9a-f]{32}-us[0-9]{1,2}['"]'''
-tags = ["key", "Mailchimp"]
-
-[[rules]]
-description = "Mailgun API key"
-regex = '''(?i)(mailgun|mg)(.{0,20})?['"][0-9a-z]{32}['"]'''
-tags = ["key", "Mailgun"]
-
-[[rules]]
-description = "Password in URL"
-regex = '''[a-zA-Z]{3,10}:\/\/[^\/\s:@]{3,20}:[^\/\s:@]{3,20}@.{1,100}\/?.?'''
-tags = ["key", "URL", "generic"]
-
-[[rules]]
-description = "PayPal Braintree access token"
-regex = '''access_token\$production\$[0-9a-z]{16}\$[0-9a-f]{32}'''
-tags = ["key", "Paypal"]
-
-[[rules]]
-description = "Picatic API key"
-regex = '''sk_live_[0-9a-z]{32}'''
-tags = ["key", "Picatic"]
-
-[[rules]]
-description = "Slack Webhook"
-regex = '''https://hooks.slack.com/services/T[a-zA-Z0-9_]{8}/B[a-zA-Z0-9_]{8}/[a-zA-Z0-9_]{24}'''
-tags = ["key", "slack"]
-
-[[rules]]
-description = "Stripe API key"
-regex = '''(?i)stripe(.{0,20})?['\"][sk|rk]_live_[0-9a-zA-Z]{24}'''
-tags = ["key", "Stripe"]
-
-[[rules]]
-description = "Square access token"
-regex = '''sq0atp-[0-9A-Za-z\-_]{22}'''
-tags = ["key", "square"]
-
-[[rules]]
-description = "Square OAuth secret"
-regex = '''sq0csp-[0-9A-Za-z\\-_]{43}'''
-tags = ["key", "square"]
-
-[[rules]]
-description = "Twilio API key"
-regex = '''(?i)twilio(.{0,20})?['\"][0-9a-f]{32}['\"]'''
-tags = ["key", "twilio"]
-
-[whitelist]
-files = [
-  "(.*?)(jpg|gif|doc|pdf|bin)$"
-]
-
-#commits = [
-#  "whitelisted-commit1",
-#  "whitelisted-commit2",
-#]
-#repos = [
-#	"whitelisted-repo"
-#]
-`

src/constants_test.go

@@ -1,123 +0,0 @@
-package gitleaks
-
-import (
-	"io/ioutil"
-	"path"
-)
-
-const testWhitelistCommit = `
-[[rules]]
-description = "AWS"
-regex = '''AKIA[0-9A-Z]{16}'''
-
-[whitelist]
-commits = [
-  "eaeffdc65b4c73ccb67e75d96bd8743be2c85973",
-]
-`
-const testWhitelistFile = `
-[[rules]]
-description = "AWS"
-regex = '''AKIA[0-9A-Z]{16}'''
-
-[whitelist]
-files = [
-  ".go",
-]
-`
-
-const testWhitelistRegex = `
-[[rules]]
-description = "AWS"
-regex = '''AKIA[0-9A-Z]{16}'''
-
-[whitelist]
-regexes= [
-  "AKIA",
-]
-`
-
-const testWhitelistRepo = `
-[[rules]]
-description = "AWS"
-regex = '''AKIA[0-9A-Z]{16}'''
-
-[whitelist]
-repos = [
-  "gronit",
-]
-`
-
-const testEntropyRange = `
-[[rules]]
-description = "Entropy ranges"
-entropies = [
-  "7.5-8.0",
-  "3.2-3.4",
-]
-`
-const testBadEntropyRange = `
-[[rules]]
-description = "Bad entropy ranges"
-entropies = [
-  "8.0-3.0",
-]
-`
-const testBadEntropyRange2 = `
-[[rules]]
-description = "Bad entropy ranges"
-entropies = [
-  "8.0-8.9",
-]
-`
-
-const testEntropyWordRegexRange = `
-[[rules]]
-description = "test entropy regex ranges"
-regex = '''(?i)key(.{0,6})?(:|=|=>|:=)'''
-entropies = [
-	"4.1-4.3",
-]
-entropyROI="word"
-`
-
-const testEntropyRegexRange = `
-[[rules]]
-description = "test entropy regex ranges"
-regex = '''(?i)key(.{0,6})?(:|=|=>|:=)'''
-entropies = [
-	"4.1-4.3",
-]
-`
-
-const testMDFileType = `
-[[rules]]
-description = "test only markdown"
-filetypes = [".md"]
-`
-
-const testEntropyRegexRangeGoFilter = `
-[[rules]]
-description = "test entropy regex ranges"
-regex = '''(?i)key(.{0,6})?(:|=|=>|:=)'''
-entropies = [
-	"4.1-4.3",
-]
-filetypes = [".go"]
-`
-
-func testTomlLoader() string {
-	tmpDir, _ := ioutil.TempDir("", "whiteListConfigs")
-	ioutil.WriteFile(path.Join(tmpDir, "regex"), []byte(testWhitelistRegex), 0644)
-	ioutil.WriteFile(path.Join(tmpDir, "entropyWordRegex"), []byte(testEntropyWordRegexRange), 0644)
-	ioutil.WriteFile(path.Join(tmpDir, "entropyRegex"), []byte(testEntropyRegexRange), 0644)
-	ioutil.WriteFile(path.Join(tmpDir, "commit"), []byte(testWhitelistCommit), 0644)
-	ioutil.WriteFile(path.Join(tmpDir, "file"), []byte(testWhitelistFile), 0644)
-	ioutil.WriteFile(path.Join(tmpDir, "repo"), []byte(testWhitelistRepo), 0644)
-	ioutil.WriteFile(path.Join(tmpDir, "entropy"), []byte(testEntropyRange), 0644)
-	ioutil.WriteFile(path.Join(tmpDir, "badEntropy"), []byte(testBadEntropyRange), 0644)
-	ioutil.WriteFile(path.Join(tmpDir, "badEntropy2"), []byte(testBadEntropyRange2), 0644)
-	ioutil.WriteFile(path.Join(tmpDir, "mdFiles"), []byte(testMDFileType), 0644)
-	ioutil.WriteFile(path.Join(tmpDir, "entropyRegexGo"), []byte(testEntropyRegexRangeGoFilter), 0644)
-	return tmpDir
-}

src/core.go

@@ -1,107 +0,0 @@
-package gitleaks
-
-import (
-	log "github.com/sirupsen/logrus"
-	"io/ioutil"
-	"os"
-	"sync"
-)
-
-var (
-	opts         *Options
-	config       *Config
-	dir          string
-	threads      int
-	totalCommits int64
-	mutex        = &sync.Mutex{}
-)
-
-func init() {
-	log.SetOutput(os.Stdout)
-	threads = defaultThreadNum
-}
-
-// Report can be exported as a json or csv. Used for logging informationn
-// about the audit, (duration and # of commits)
-type Report struct {
-	Leaks    []Leak
-	Duration string
-	Commits  int64
-}
-
-// Run is the entry point for gitleaks
-func Run(optsL *Options) (int, error) {
-	var (
-		err   error
-		leaks []Leak
-	)
-
-	opts = optsL
-	config, err = newConfig()
-	if err != nil {
-		return NoLeaks, err
-	}
-
-	if opts.Disk {
-		// temporary directory where all the gitleaks plain clones will reside
-		dir, err = ioutil.TempDir("", "gitleaks")
-		defer os.RemoveAll(dir)
-		if err != nil {
-			return NoLeaks, err
-		}
-	}
-
-	// start audits
-	if opts.Repo != "" || opts.RepoPath != "" {
-		var repo *Repo
-		repo, err = newRepo()
-		if err != nil {
-			return NoLeaks, err
-		}
-		err = repo.clone()
-		if err != nil {
-			return NoLeaks, err
-		}
-		err = repo.audit()
-		if err != nil {
-			return NoLeaks, err
-		}
-		repo.report()
-		leaks = repo.leaks
-	} else if opts.OwnerPath != "" {
-		var repos []*Repo
-		repos, err = discoverRepos(opts.OwnerPath)
-		if err != nil {
-			return NoLeaks, err
-		}
-		for _, repo := range repos {
-			err = repo.clone()
-			if err != nil {
-				log.Warnf("error occurred cloning repo: %s, continuing to next repo", repo.name)
-				continue
-			}
-			err = repo.audit()
-			if err != nil {
-				log.Warnf("error occurred auditing repo: %s, continuing to next repo", repo.name)
-				continue
-			}
-			repo.report()
-			leaks = append(leaks, repo.leaks...)
-		}
-	} else if opts.GithubOrg != "" || opts.GithubUser != "" {
-		return auditGithubRepos()
-	} else if opts.GitLabOrg != "" || opts.GitLabUser != "" {
-		return auditGitlabRepos()
-	} else if opts.GithubPR != "" {
-		return auditGithubPR()
-	}
-
-	if opts.Report != "" {
-		err = writeReport(leaks)
-		if err != nil {
-			return NoLeaks, err
-		}
-	}
-
-	return len(leaks), nil
-}

src/entropy.go

@@ -1,25 +0,0 @@
-package gitleaks
-
-import (
-	"math"
-)
-
-// getShannonEntropy https://en.wiktionary.org/wiki/Shannon_entropy
-func getShannonEntropy(data string) (entropy float64) {
-	if data == "" {
-		return 0
-	}
-
-	charCounts := make(map[rune]int)
-	for _, char := range data {
-		charCounts[char]++
-	}
-
-	invLength := 1.0 / float64(len(data))
-	for _, count := range charCounts {
-		freq := float64(count) * invLength
-		entropy -= freq * math.Log2(freq)
-	}
-
-	return entropy
-}

src/github.go

@@ -1,285 +0,0 @@
-package gitleaks
-
-import (
-	"context"
-	"fmt"
-	"io/ioutil"
-	"net/http"
-	"net/url"
-	"os"
-	"strconv"
-	"strings"
-
-	"github.com/google/go-github/github"
-	log "github.com/sirupsen/logrus"
-	"golang.org/x/oauth2"
-	"gopkg.in/src-d/go-git.v4"
-	gitHttp "gopkg.in/src-d/go-git.v4/plumbing/transport/http"
-	"gopkg.in/src-d/go-git.v4/storage/memory"
-)
-
-var githubPages = 100
-
-// auditPR audits a single github PR
-func auditGithubPR() (int, error) {
-	var leaks []Leak
-	ctx := context.Background()
-	githubClient := github.NewClient(githubToken())
-	splits := strings.Split(opts.GithubPR, "/")
-	owner := splits[len(splits)-4]
-	repo := splits[len(splits)-3]
-	prNum, err := strconv.Atoi(splits[len(splits)-1])
-	if err != nil {
-		return NoLeaks, err
-	}
-
-	page := 1
-	for {
-		commits, resp, err := githubClient.PullRequests.ListCommits(ctx, owner, repo, prNum, &github.ListOptions{
-			PerPage: githubPages,
-			Page:    page,
-		})
-		if err != nil {
-			return NoLeaks, err
-		}
-
-		for _, c := range commits {
-			totalCommits = totalCommits + 1
-			c, _, err := githubClient.Repositories.GetCommit(ctx, owner, repo, *c.SHA)
-			if err != nil {
-				continue
-			}
-			files := c.Files
-			for _, f := range files {
-				skipFile := false
-				if f.Patch == nil || f.Filename == nil {
-					continue
-				}
-				for _, re := range config.WhiteList.files {
-					if re.FindString(f.GetFilename()) != "" {
-						log.Infof("skipping whitelisted file (matched regex '%s'): %s", re.String(), f.GetFilename())
-						skipFile = true
-						break
-					}
-				}
-				if skipFile {
-					continue
-				}
-
-				commit := &Commit{
-					sha:      c.GetSHA(),
-					content:  *f.Patch,
-					filePath: *f.Filename,
-					repoName: repo,
-					author:   c.GetCommitter().GetLogin(),
-					message:  *c.Commit.Message,
-					date:     *c.Commit.Committer.Date,
-				}
-				leaks = append(leaks, inspect(commit)...)
-			}
-		}
-		page = resp.NextPage
-		if resp.LastPage == 0 {
-			break
-		}
-	}
-
-	if len(leaks) != 0 {
-		log.Warnf("%d leaks detected. %d commits inspected for PR: %s", len(leaks), totalCommits, opts.GithubPR)
-	}
-
-	if opts.Report != "" {
-		err = writeReport(leaks)
-		if err != nil {
-			return NoLeaks, err
-		}
-	}
-
-	return len(leaks), nil
-}
-
-// auditGithubRepos kicks off audits if --github-user or --github-org options are set.
-// First, we gather all the github repositories from the github api (this doesnt actually clone the repo).
-// After all the repos have been pulled from github's api we proceed to audit the repos by calling auditGithubRepo.
-// If an error occurs during an audit of a repo, that error is logged but won't break the execution cycle.
-func auditGithubRepos() (int, error) {
-	var (
-		err              error
-		githubRepos      []*github.Repository
-		pagedGithubRepos []*github.Repository
-		resp             *github.Response
-		githubOrgOptions *github.RepositoryListByOrgOptions
-		githubOptions    *github.RepositoryListOptions
-		done             bool
-		ownerDir         string
-		leaks            []Leak
-	)
-	ctx := context.Background()
-	githubClient := github.NewClient(githubToken())
-
-	if opts.GithubOrg != "" {
-		if opts.GithubURL != "" && opts.GithubURL != defaultGithubURL {
-			ghURL, _ := url.Parse(opts.GithubURL)
-			githubClient.BaseURL = ghURL
-		}
-		githubOrgOptions = &github.RepositoryListByOrgOptions{
-			ListOptions: github.ListOptions{PerPage: 100},
-		}
-	} else if opts.GithubUser != "" {
-		if opts.GithubURL != "" && opts.GithubURL != defaultGithubURL {
-			ghURL, _ := url.Parse(opts.GithubURL)
-			githubClient.BaseURL = ghURL
-		}
-
-		githubOptions = &github.RepositoryListOptions{
-			Affiliation: "owner",
-			ListOptions: github.ListOptions{
-				PerPage: 100,
-			},
-		}
-	}
-
-	for {
-		if done {
-			break
-		}
-		if opts.GithubUser != "" {
-			pagedGithubRepos, resp, err = githubClient.Repositories.List(ctx, opts.GithubUser, githubOptions)
-			if err != nil {
-				done = true
-			}
-			githubOptions.Page = resp.NextPage
-			githubRepos = append(githubRepos, pagedGithubRepos...)
-			if resp.NextPage == 0 {
-				done = true
-			}
-		} else if opts.GithubOrg != "" {
-			pagedGithubRepos, resp, err = githubClient.Repositories.ListByOrg(ctx, opts.GithubOrg, githubOrgOptions)
-			if err != nil {
-				done = true
-			}
-			githubOrgOptions.Page = resp.NextPage
-			githubRepos = append(githubRepos, pagedGithubRepos...)
-			if resp.NextPage == 0 {
-				done = true
-			}
-		}
-		if opts.Log == "Debug" || opts.Log == "debug" {
-			for _, githubRepo := range pagedGithubRepos {
-				log.Debugf("staging repos %s", *githubRepo.Name)
-			}
-		}
-	}
-	if opts.Disk {
-		ownerDir, _ = ioutil.TempDir(dir, opts.GithubUser)
-	}
-	for _, githubRepo := range githubRepos {
-		repo, err := cloneGithubRepo(githubRepo)
-		if err != nil {
-			log.Warn(err)
-			continue
-		}
-		err = repo.audit()
-		if err != nil {
-			log.Warnf("error occurred during audit of repo: %s, err: %v, continuing github audit", repo.name, err)
-		}
-		if opts.Disk {
-			os.RemoveAll(fmt.Sprintf("%s/%s", ownerDir, *githubRepo.Name))
-		}
-
-		repo.report()
-
-		leaks = append(leaks, repo.leaks...)
-	}
-
-	if opts.Report != "" {
-		err = writeReport(leaks)
-		if err != nil {
-			return NoLeaks, err
-		}
-	}
-
-	return len(leaks), nil
-}
-
-// cloneGithubRepo clones a repo from the url parsed from a github repo. The repo
-// will be cloned to disk if --disk is set.
-func cloneGithubRepo(githubRepo *github.Repository) (*Repo, error) {
-	var (
-		repo *git.Repository
-		err  error
-	)
-	githubToken := os.Getenv("GITHUB_TOKEN")
-	if opts.ExcludeForks && githubRepo.GetFork() {
-		return nil, fmt.Errorf("skipping %s, excluding forks", *githubRepo.Name)
-	}
-	for _, re := range config.WhiteList.repos {
-		if re.FindString(*githubRepo.Name) != "" {
-			return nil, fmt.Errorf("skipping %s, whitelisted", *githubRepo.Name)
-		}
-	}
-	log.Infof("cloning: %s", *githubRepo.Name)
-	if opts.Disk {
-		ownerDir, err := ioutil.TempDir(dir, opts.GithubUser)
-		if err != nil {
-			return nil, fmt.Errorf("unable to generater owner temp dir: %v", err)
-		}
-		if config.sshAuth != nil && githubToken == "" {
-			repo, err = git.PlainClone(fmt.Sprintf("%s/%s", ownerDir, *githubRepo.Name), false, &git.CloneOptions{
-				URL:  *githubRepo.SSHURL,
-				Auth: config.sshAuth,
-			})
-		} else if githubToken != "" {
-			repo, err = git.PlainClone(fmt.Sprintf("%s/%s", ownerDir, *githubRepo.Name), false, &git.CloneOptions{
-				URL: *githubRepo.CloneURL,
-				Auth: &gitHttp.BasicAuth{
-					Username: "fakeUsername", // yes, this can be anything except an empty string
-					Password: githubToken,
-				},
-			})
-		} else {
-			repo, err = git.PlainClone(fmt.Sprintf("%s/%s", ownerDir, *githubRepo.Name), false, &git.CloneOptions{
-				URL: *githubRepo.CloneURL,
-			})
-		}
-	} else {
-		if config.sshAuth != nil && githubToken == "" {
-			repo, err = git.Clone(memory.NewStorage(), nil, &git.CloneOptions{
-				URL:  *githubRepo.SSHURL,
-				Auth: config.sshAuth,
-			})
-		} else if githubToken != "" {
-			repo, err = git.Clone(memory.NewStorage(), nil, &git.CloneOptions{
-				URL: *githubRepo.CloneURL,
-				Auth: &gitHttp.BasicAuth{
-					Username: "fakeUsername", // yes, this can be anything except an empty string
-					Password: githubToken,
-				},
-			})
-		} else {
-			repo, err = git.Clone(memory.NewStorage(), nil, &git.CloneOptions{
-				URL: *githubRepo.CloneURL,
-			})
-		}
-	}
-	if err != nil {
-		return nil, err
-	}
-	return &Repo{
-		repository: repo,
-		name:       *githubRepo.Name,
-	}, nil
-}
-
-// githubToken returns an oauth2 client for the github api to consume. This token is necessary
-// if you are running audits with --github-user or --github-org
-func githubToken() *http.Client {
-	githubToken := os.Getenv("GITHUB_TOKEN")
-	if githubToken == "" {
-		return nil
-	}
-	ts := oauth2.StaticTokenSource(
-		&oauth2.Token{AccessToken: githubToken},
-	)
-	return oauth2.NewClient(context.Background(), ts)
-}

src/gitlab.go

@@ -1,178 +0,0 @@
-package gitleaks
-
-import (
-	"fmt"
-	"io/ioutil"
-	"os"
-
-	log "github.com/sirupsen/logrus"
-	"github.com/xanzy/go-gitlab"
-	"gopkg.in/src-d/go-git.v4"
-	gitHttp "gopkg.in/src-d/go-git.v4/plumbing/transport/http"
-	"gopkg.in/src-d/go-git.v4/storage/memory"
-)
-
-// gitlabPages number of records per request
-const gitlabPages = 100
-
-// auditGitlabRepos kicks off audits if --gitlab-user or --gitlab-org options are set.
-// Getting all repositories from the GitLab API and run audit. If an error occurs during an audit of a repo,
-// that error is logged.
-func auditGitlabRepos() (int, error) {
-	var (
-		ps      []*gitlab.Project
-		resp    *gitlab.Response
-		tempDir string
-		err     error
-		leaks   []Leak
-	)
-
-	repos := make([]*gitlab.Project, 0, gitlabPages)
-	page := 1
-	cl := gitlab.NewClient(nil, os.Getenv("GITLAB_TOKEN"))
-
-	// if self hosted GitLab server
-	if url := os.Getenv("GITLAB_URL"); url != "" {
-		cl.SetBaseURL(url)
-	}
-
-	for {
-		if opts.GitLabOrg != "" {
-			opt := &gitlab.ListGroupProjectsOptions{
-				ListOptions: gitlab.ListOptions{
-					PerPage: gitlabPages,
-					Page:    page,
-				},
-			}
-
-			ps, resp, err = cl.Groups.ListGroupProjects(opts.GitLabOrg, opt)
-		} else if opts.GitLabUser != "" {
-			opt := &gitlab.ListProjectsOptions{
-				ListOptions: gitlab.ListOptions{
-					PerPage: gitlabPages,
-					Page:    page,
-				},
-			}
-
-			ps, resp, err = cl.Projects.ListUserProjects(opts.GitLabUser, opt)
-		}
-
-		if err != nil {
-			// exit when can't make API call
-			log.Fatal("error listing projects: ", err)
-		}
-
-		repos = append(repos, ps...)
-
-		if page >= resp.TotalPages {
-			// exit when we've seen all pages
-			break
-		}
-
-		page = resp.NextPage
-	}
-
-	log.Debugf("found projects: %d", len(repos))
-
-	if opts.Disk {
-		if tempDir, err = createGitlabTempDir(); err != nil {
-			log.Fatal("error creating temp directory: ", err)
-		}
-	}
-
-	for _, p := range repos {
-		repo, err := cloneGitlabRepo(tempDir, p)
-		if err != nil {
-			log.Warn(err)
-			continue
-		}
-
-		err = repo.audit()
-		if err != nil {
-			log.Warn(err)
-			continue
-		}
-
-		if opts.Disk {
-			os.RemoveAll(fmt.Sprintf("%s/%d", tempDir, p.ID))
-		}
-
-		repo.report()
-		leaks = append(leaks, repo.leaks...)
-	}
-
-	if opts.Report != "" {
-		err = writeReport(leaks)
-		if err != nil {
-			return NoLeaks, err
-		}
-	}
-
-	return len(leaks), nil
-}
-
-func createGitlabTempDir() (string, error) {
-	pathName := opts.GitLabUser
-	if opts.GitLabOrg != "" {
-		pathName = opts.GitLabOrg
-	}
-
-	os.RemoveAll(fmt.Sprintf("%s/%s", dir, pathName))
-
-	ownerDir, err := ioutil.TempDir(dir, pathName)
-	if err != nil {
-		return "", err
-	}
-
-	return ownerDir, nil
-}
-
-func cloneGitlabRepo(tempDir string, p *gitlab.Project) (*Repo, error) {
-	var (
-		repo *git.Repository
-		err  error
-	)
-
-	gitLabToken := os.Getenv("GITLAB_TOKEN")
-
-	if opts.ExcludeForks && p.ForkedFromProject != nil {
-		return nil, fmt.Errorf("skipping %s, excluding forks", p.Name)
-	}
-
-	for _, re := range config.WhiteList.repos {
-		if re.FindString(p.Name) != "" {
-			return nil, fmt.Errorf("skipping %s, whitelisted", p.Name)
-		}
-	}
-
-	opt := &git.CloneOptions{
-		URL: p.HTTPURLToRepo,
-	}
-
-	if config.sshAuth != nil && gitLabToken == "" {
-		opt.URL = p.SSHURLToRepo
-		opt.Auth = config.sshAuth
-	} else if gitLabToken != "" {
-		opt.Auth = &gitHttp.BasicAuth{
-			Username: "fakeUsername", // yes, this can be anything except an empty string
-			Password: gitLabToken,
-		}
-	}
-
-	log.Infof("cloning: %s", p.Name)
-
-	if opts.Disk {
-		repo, err = git.PlainClone(fmt.Sprintf("%s/%d", tempDir, p.ID), false, opt)
-	} else {
-		repo, err = git.Clone(memory.NewStorage(), nil, opt)
-	}
-
-	if err != nil {
-		return nil, err
-	}
-
-	return &Repo{
-		repository: repo,
-		name:       p.Name,
-	}, nil
-}

src/gitleaks_test.go

@@ -1,791 +0,0 @@
-package gitleaks
-
-import (
-	"fmt"
-	"io/ioutil"
-	"os"
-	"path"
-	"regexp"
-	"runtime"
-	"strings"
-	"testing"
-	"time"
-
-	"github.com/franela/goblin"
-	log "github.com/sirupsen/logrus"
-	git "gopkg.in/src-d/go-git.v4"
-	"gopkg.in/src-d/go-git.v4/storage/memory"
-)
-
-func TestGetRepo(t *testing.T) {
-	var err error
-	dir, err = ioutil.TempDir("", "gitleaksTestRepo")
-	defer os.RemoveAll(dir)
-	if err != nil {
-		panic(err)
-	}
-	_, err = git.PlainClone(dir, false, &git.CloneOptions{
-		URL: "https://github.com/gitleakstest/gronit",
-	})
-
-	if err != nil {
-		panic(err)
-	}
-
-	var tests = []struct {
-		testOpts       *Options
-		description    string
-		expectedErrMsg string
-	}{
-		{
-			testOpts: &Options{
-				Repo: "https://github.com/gitleakstest/gronit",
-			},
-			description:    "test plain clone remote repo",
-			expectedErrMsg: "",
-		},
-		{
-			testOpts: &Options{
-				Repo: "https://github.com/gitleakstest/gronit",
-				Disk: true,
-			},
-			description:    "test on disk clone remote repo",
-			expectedErrMsg: "",
-		},
-		{
-			testOpts: &Options{
-				RepoPath: dir,
-			},
-			description:    "test local clone repo",
-			expectedErrMsg: "",
-		},
-		{
-			testOpts: &Options{
-				Repo: "https://github.com/gitleakstest/nope",
-			},
-			description:    "test no repo",
-			expectedErrMsg: "repository not found",
-		},
-		{
-			testOpts: &Options{
-				Repo: "https://github.com/gitleakstest/private",
-			},
-			description:    "test private repo",
-			expectedErrMsg: "repository not found",
-		},
-		{
-			testOpts: &Options{
-				Repo: "https://github.com/gitleakstest/private",
-				Disk: true,
-			},
-			description:    "test private repo disk",
-			expectedErrMsg: "repository not found",
-		},
-	}
-	g := goblin.Goblin(t)
-	for _, test := range tests {
-		g.Describe("TestGetRepo", func() {
-			g.It(test.description, func() {
-				opts = test.testOpts
-				config, err = newConfig()
-				if err != nil {
-					log.Fatal(err)
-				}
-				repo, _ := newRepo()
-				err := repo.clone()
-				if err != nil {
-					g.Assert(err.Error()).Equal(test.expectedErrMsg)
-				}
-			})
-		})
-	}
-}
-
-func TestRun(t *testing.T) {
-	var err error
-	configsDir := testTomlLoader()
-
-	dir, err = ioutil.TempDir("", "gitleaksTestOwner")
-	defer os.RemoveAll(dir)
-	if err != nil {
-		panic(err)
-	}
-	git.PlainClone(dir+"/gronit", false, &git.CloneOptions{
-		URL: "https://github.com/gitleakstest/gronit",
-	})
-	git.PlainClone(dir+"/h1domains", false, &git.CloneOptions{
-		URL: "https://github.com/gitleakstest/h1domains",
-	})
-	var tests = []struct {
-		testOpts       *Options
-		description    string
-		expectedErrMsg string
-		whiteListRepos []string
-		whiteListFiles []*regexp.Regexp
-		numLeaks       int
-		configPath     string
-		commitPerPage  int
-	}{
-		{
-			testOpts: &Options{
-				Repo: "https://github.com/gitleakstest/gronit.git",
-			},
-			description:    "test leak",
-			numLeaks:       2,
-			expectedErrMsg: "",
-		},
-		{
-			testOpts: &Options{
-				GitLabUser: "gitleakstest",
-			},
-			description:    "test gitlab user",
-			numLeaks:       2,
-			expectedErrMsg: "",
-		},
-		{
-			testOpts: &Options{
-				GithubUser: "gitleakstest",
-			},
-			description:    "test github user",
-			numLeaks:       2,
-			expectedErrMsg: "",
-		},
-		{
-			testOpts: &Options{
-				GithubUser: "gitleakstest",
-				Disk:       true,
-			},
-			description:    "test github user on disk ",
-			numLeaks:       2,
-			expectedErrMsg: "",
-		},
-		{
-			testOpts: &Options{
-				GithubOrg: "gitleakstestorg",
-			},
-			description:    "test github org",
-			numLeaks:       2,
-			expectedErrMsg: "",
-		},
-		{
-			testOpts: &Options{
-				GithubOrg: "gitleakstestorg",
-				Disk:      true,
-			},
-			description:    "test org on disk",
-			numLeaks:       2,
-			expectedErrMsg: "",
-		},
-		{
-			testOpts: &Options{
-				OwnerPath: dir,
-			},
-			description:    "test owner path",
-			numLeaks:       2,
-			expectedErrMsg: "",
-		},
-		{
-			testOpts: &Options{
-				Repo:   "git@github.com:gitleakstest/gronit.git",
-				SSHKey: "trash",
-			},
-			description:    "test leak",
-			numLeaks:       0,
-			expectedErrMsg: fmt.Sprintf("unable to generate ssh key: open trash: %s", noSuchFileMessage()),
-		},
-		{
-			testOpts: &Options{
-				Repo: "https://github.com/gitleakstest/gronit.git",
-			},
-			description:    "test leak",
-			numLeaks:       2,
-			expectedErrMsg: "",
-		},
-		{
-			testOpts: &Options{
-				Repo: "https://github.com/gitleakstest/h1domains.git",
-			},
-			description:    "test clean",
-			numLeaks:       0,
-			expectedErrMsg: "",
-		},
-		{
-			testOpts: &Options{
-				Repo: "https://github.com/gitleakstest/empty.git",
-			},
-			description:    "test empty",
-			numLeaks:       0,
-			expectedErrMsg: "repository not found",
-		},
-		{
-			testOpts: &Options{
-				GithubOrg: "gitleakstestorg",
-			},
-			description:    "test github org, whitelist repo",
-			numLeaks:       0,
-			expectedErrMsg: "",
-			configPath:     path.Join(configsDir, "repo"),
-		},
-		{
-			testOpts: &Options{
-				GithubOrg:    "gitleakstestorg",
-				ExcludeForks: true,
-			},
-			description:    "test github org, exclude forks",
-			numLeaks:       0,
-			expectedErrMsg: "",
-		},
-		{
-			testOpts: &Options{
-				GithubPR: "https://github.com/gitleakstest/gronit/pull/1",
-			},
-			description:    "test github pr",
-			numLeaks:       4,
-			expectedErrMsg: "",
-		},
-		{
-			testOpts: &Options{
-				GithubPR: "https://github.com/gitleakstest/gronit/pull/1",
-			},
-			description:    "test github pr",
-			numLeaks:       4,
-			expectedErrMsg: "",
-			commitPerPage:  1,
-		},
-		{
-			testOpts: &Options{
-				GithubPR: "https://github.com/gitleakstest/gronit/pull/1",
-			},
-			description:    "test github pr with whitelisted files",
-			numLeaks:       0,
-			expectedErrMsg: "",
-			commitPerPage:  1,
-			configPath:     path.Join(configsDir, "file"),
-		},
-		{
-			testOpts: &Options{
-				GithubPR: "https://github.com/gitleakstest/gronit/pull/2",
-			},
-			description:    "test github pr with commits without patch info",
-			numLeaks:       0,
-			expectedErrMsg: "",
-			commitPerPage:  1,
-		},
-	}
-	g := goblin.Goblin(t)
-	for _, test := range tests {
-		g.Describe("TestRun", func() {
-			g.It(test.description, func() {
-				if test.configPath != "" {
-					os.Setenv("GITLEAKS_CONFIG", test.configPath)
-				}
-				if test.commitPerPage != 0 {
-					githubPages = test.commitPerPage
-				}
-				numLeaks, err := Run(test.testOpts)
-				if err != nil {
-					g.Assert(err.Error()).Equal(test.expectedErrMsg)
-				} else {
-					g.Assert(numLeaks).Equal(test.numLeaks)
-				}
-				githubPages = 100
-			})
-		})
-	}
-}
-
-func TestWriteReport(t *testing.T) {
-	tmpDir, _ := ioutil.TempDir("", "reportDir")
-	reportJSON := path.Join(tmpDir, "report.json")
-	reportJASON := path.Join(tmpDir, "report.jason")
-	reportVOID := path.Join("thereIsNoWay", "thisReportWillGetWritten.json")
-	reportCSV := path.Join(tmpDir, "report.csv")
-	defer os.RemoveAll(tmpDir)
-	leaks := []Leak{
-		{
-			Line:     "eat",
-			Commit:   "your",
-			Offender: "veggies",
-			Rule:     "and",
-			Message:  "get",
-			Author:   "some",
-			File:     "sleep",
-			Date:     time.Now(),
-		},
-	}
-
-	var tests = []struct {
-		leaks          []Leak
-		reportFile     string
-		fileName       string
-		description    string
-		testOpts       Options
-		expectedErrMsg string
-	}{
-		{
-			leaks:       leaks,
-			reportFile:  reportJSON,
-			fileName:    "report.json",
-			description: "can we write a json file",
-			testOpts: Options{
-				Report: reportJSON,
-			},
-		},
-		{
-			leaks:       leaks,
-			reportFile:  reportCSV,
-			fileName:    "report.csv",
-			description: "can we write a csv file",
-			testOpts: Options{
-				Report: reportCSV,
-			},
-		},
-		{
-			leaks:          leaks,
-			reportFile:     reportJASON,
-			fileName:       "report.jason",
-			description:    "bad file",
-			expectedErrMsg: "Report should be a .json or .csv file",
-			testOpts: Options{
-				Report: reportJASON,
-			},
-		},
-		{
-			leaks:          leaks,
-			reportFile:     reportVOID,
-			fileName:       "report.jason",
-			description:    "bad dir",
-			expectedErrMsg: "thereIsNoWay does not exist",
-			testOpts: Options{
-				Report: reportVOID,
-			},
-		},
-	}
-	g := goblin.Goblin(t)
-	for _, test := range tests {
-		g.Describe("TestWriteReport", func() {
-			g.It(test.description, func() {
-				opts = &(test.testOpts)
-				err := opts.guard()
-				if err != nil {
-					g.Assert(err.Error()).Equal(test.expectedErrMsg)
-				} else {
-					writeReport(test.leaks)
-					f, _ := os.Stat(test.reportFile)
-					g.Assert(f.Name()).Equal(test.fileName)
-				}
-			})
-		})
-	}
-
-}
-
-func TestAuditRepo(t *testing.T) {
-	configsDir := testTomlLoader()
-	defer os.RemoveAll(configsDir)
-
-	leaksR, err := git.Clone(memory.NewStorage(), nil, &git.CloneOptions{
-		URL: "https://github.com/gitleakstest/gronit.git",
-	})
-	if err != nil {
-		panic(err)
-	}
-	leaksRepo := &Repo{
-		repository: leaksR,
-		name:       "gronit",
-	}
-
-	cleanR, err := git.Clone(memory.NewStorage(), nil, &git.CloneOptions{
-		URL: "https://github.com/gitleakstest/h1domains.git",
-	})
-	if err != nil {
-		panic(err)
-	}
-	cleanRepo := &Repo{
-		repository: cleanR,
-		name:       "h1domains",
-	}
-
-	var tests = []struct {
-		testOpts         *Options
-		description      string
-		expectedErrMsg   string
-		numLeaks         int
-		repo             *Repo
-		whiteListFiles   []*regexp.Regexp
-		whiteListCommits map[string]bool
-		whiteListRepos   []*regexp.Regexp
-		whiteListRegexes []*regexp.Regexp
-		configPath       string
-	}{
-		{
-			repo:        leaksRepo,
-			description: "pinned config",
-			numLeaks:    0,
-			testOpts: &Options{
-				RepoConfig: true,
-			},
-		},
-		{
-			repo:        leaksRepo,
-			description: "commit depth = 1, one leak",
-			numLeaks:    1,
-			testOpts: &Options{
-				Depth: 1,
-			},
-		},
-		{
-			repo:        leaksRepo,
-			description: "two leaks present",
-			numLeaks:    2,
-			testOpts:    &Options{},
-		},
-		{
-			repo:        leaksRepo,
-			description: "no leaks present on branch",
-			numLeaks:    0,
-			testOpts: &Options{
-				Branch: "dev",
-			},
-		},
-		{
-			repo:        leaksRepo,
-			description: "two leaks present limit goroutines",
-			numLeaks:    2,
-			testOpts: &Options{
-				Threads: 4,
-			},
-		},
-		{
-			repo:        leaksRepo,
-			description: "two leaks present whitelist AWS.. no leaks",
-			testOpts:    &Options{},
-			configPath:  path.Join(configsDir, "regex"),
-			numLeaks:    0,
-		},
-		{
-			repo:        leaksRepo,
-			description: "two leaks present limit goroutines",
-			testOpts: &Options{
-				Threads: 2,
-			},
-			numLeaks: 2,
-		},
-		{
-			repo:        cleanRepo,
-			description: "no leaks present",
-			testOpts:    &Options{},
-			numLeaks:    0,
-		},
-		{
-			repo:        leaksRepo,
-			description: "two leaks present whitelist go files",
-			testOpts:    &Options{},
-			configPath:  path.Join(configsDir, "file"),
-			numLeaks:    0,
-		},
-		{
-			repo:        leaksRepo,
-			description: "two leaks present whitelist bad commit",
-			configPath:  path.Join(configsDir, "commit"),
-			testOpts:    &Options{},
-			numLeaks:    1,
-		},
-		{
-			repo:        leaksRepo,
-			description: "redact",
-			testOpts: &Options{
-				Redact: true,
-			},
-			numLeaks: 2,
-		},
-		{
-			repo:        leaksRepo,
-			description: "Audit a specific commit",
-			numLeaks:    1,
-			testOpts: &Options{
-				Commit: "cb5599aeed261b2c038aa4729e2d53ca050a4988",
-			},
-		},
-		{
-			repo:        leaksRepo,
-			description: "Audit a specific commit no leaks",
-			numLeaks:    0,
-			testOpts: &Options{
-				Commit: "2b033e012eee364fc41b4ab7c5db1497399b8e67",
-			},
-		},
-		{
-			repo:        leaksRepo,
-			description: "toml whitelist regex",
-			configPath:  path.Join(configsDir, "regex"),
-			testOpts:    &Options{},
-			numLeaks:    0,
-		},
-		{
-			repo:        leaksRepo,
-			description: "toml whitelist file",
-			configPath:  path.Join(configsDir, "file"),
-			testOpts:    &Options{},
-			numLeaks:    0,
-		},
-		{
-			repo:        leaksRepo,
-			description: "toml whitelist commit",
-			configPath:  path.Join(configsDir, "commit"),
-			testOpts:    &Options{},
-			numLeaks:    1,
-		},
-		{
-			repo:        leaksRepo,
-			description: "audit whitelist repo",
-			numLeaks:    0,
-			testOpts:    &Options{},
-			configPath:  path.Join(configsDir, "repo"),
-		},
-		{
-			repo:        leaksRepo,
-			description: "toml whitelist repo",
-			numLeaks:    0,
-			testOpts:    &Options{},
-			configPath:  path.Join(configsDir, "repo"),
-		},
-		{
-			repo:        leaksRepo,
-			description: "Audit until specific commit",
-			numLeaks:    2,
-			testOpts: &Options{
-				CommitStop: "f6839959b7bbdcd23008f1fb16f797f35bcd3a0c",
-			},
-		},
-		{
-			repo:        leaksRepo,
-			description: "commit depth = 2, two leaks",
-			numLeaks:    2,
-			testOpts: &Options{
-				Depth: 2,
-			},
-		},
-		{
-			repo:        leaksRepo,
-			description: "toml entropy range from opts",
-			numLeaks:    266,
-			testOpts: &Options{
-				ConfigPath: path.Join(configsDir, "entropy"),
-			},
-		},
-		{
-			repo:        leaksRepo,
-			description: "toml entropy regex word range",
-			numLeaks:    0,
-			testOpts:    &Options{},
-			configPath:  path.Join(configsDir, "entropyWordRegex"),
-		},
-		{
-			repo:        leaksRepo,
-			description: "toml entropy regex range",
-			numLeaks:    2,
-			testOpts:    &Options{},
-			configPath:  path.Join(configsDir, "entropyRegex"),
-		},
-		{
-			repo:           leaksRepo,
-			description:    "toml bad entropy range",
-			numLeaks:       0,
-			testOpts:       &Options{},
-			configPath:     path.Join(configsDir, "badEntropy"),
-			expectedErrMsg: "entropy range must be ascending",
-		},
-		{
-			repo:           leaksRepo,
-			description:    "toml bad entropy2 range",
-			numLeaks:       0,
-			testOpts:       &Options{},
-			configPath:     path.Join(configsDir, "badEntropy2"),
-			expectedErrMsg: "invalid entropy ranges, must be within 0.0-8.0",
-		},
-		{
-			repo:        leaksRepo,
-			description: "toml md files",
-			numLeaks:    5,
-			testOpts:    &Options{},
-			configPath:  path.Join(configsDir, "mdFiles"),
-		},
-		{
-			repo:        leaksRepo,
-			description: "toml entropys line regex go",
-			numLeaks:    2,
-			testOpts:    &Options{},
-			configPath:  path.Join(configsDir, "entropyRegexGo"),
-		},
-	}
-	g := goblin.Goblin(t)
-	for _, test := range tests {
-		g.Describe("TestAuditRepo", func() {
-			g.It(test.description, func() {
-				opts = test.testOpts
-
-				config, err = newConfig()
-				// config paths
-				if test.configPath != "" {
-					os.Setenv("GITLEAKS_CONFIG", test.configPath)
-					config, err = newConfig()
-					if err != nil {
-						g.Assert(err.Error()).Equal(test.expectedErrMsg)
-						goto next
-					}
-				}
-				err = test.repo.audit()
-				if opts.Redact {
-					g.Assert(test.repo.leaks[0].Offender).Equal("REDACTED")
-				}
-				g.Assert(len(test.repo.leaks)).Equal(test.numLeaks)
-			next:
-				os.Setenv("GITLEAKS_CONFIG", "")
-				test.repo.leaks = []Leak{}
-			})
-		})
-	}
-}
-
-func TestOptionGuard(t *testing.T) {
-	var tests = []struct {
-		testOpts            *Options
-		githubToken         bool
-		description         string
-		expectedErrMsg      string
-		expectedErrMsgFuzzy string
-	}{
-		{
-			testOpts:       &Options{},
-			description:    "default no opts",
-			expectedErrMsg: "",
-		},
-		{
-			testOpts: &Options{
-				GithubUser: "fakeUser",
-				GithubOrg:  "fakeOrg",
-			},
-			description:    "double owner",
-			expectedErrMsg: "github user and organization set",
-		},
-		{
-			testOpts: &Options{
-				GithubOrg: "fakeOrg",
-				OwnerPath: "/dev/null",
-			},
-			description:    "local and remote target",
-			expectedErrMsg: "github organization set and local owner path",
-		},
-		{
-			testOpts: &Options{
-				GithubUser: "fakeUser",
-				OwnerPath:  "/dev/null",
-			},
-			description:    "local and remote target",
-			expectedErrMsg: "github user set and local owner path",
-		},
-	}
-	g := goblin.Goblin(t)
-	for _, test := range tests {
-		g.Describe("Test Option Gaurd", func() {
-			g.It(test.description, func() {
-				os.Clearenv()
-				opts = test.testOpts
-				if test.githubToken {
-					os.Setenv("GITHUB_TOKEN", "fakeToken")
-				}
-				err := opts.guard()
-				if err != nil {
-					if test.expectedErrMsgFuzzy != "" {
-						g.Assert(strings.Contains(err.Error(), test.expectedErrMsgFuzzy)).Equal(true)
-					} else {
-						g.Assert(err.Error()).Equal(test.expectedErrMsg)
-					}
-				} else {
-					g.Assert("").Equal(test.expectedErrMsg)
-				}
-
-			})
-		})
-	}
-}
-
-func TestLoadToml(t *testing.T) {
-	tmpDir, _ := ioutil.TempDir("", "gitleaksTestConfigDir")
-	defer os.RemoveAll(tmpDir)
-	err := ioutil.WriteFile(path.Join(tmpDir, "gitleaksConfig"), []byte(defaultConfig), 0644)
-	if err != nil {
-		panic(err)
-	}
-
-	configPath := path.Join(tmpDir, "gitleaksConfig")
-	noConfigPath := path.Join(tmpDir, "gitleaksConfigNope")
-
-	var tests = []struct {
-		testOpts       *Options
-		description    string
-		configPath     string
-		expectedErrMsg string
-		singleSearch   bool
-	}{
-		{
-			testOpts: &Options{
-				ConfigPath: configPath,
-			},
-			description: "path to config",
-		},
-		{
-			testOpts:     &Options{},
-			description:  "env var path to no config",
-			singleSearch: true,
-		},
-		{
-			testOpts: &Options{
-				ConfigPath: noConfigPath,
-			},
-			description:    "no path to config",
-			expectedErrMsg: fmt.Sprintf("no gitleaks config at %s", noConfigPath),
-		},
-		{
-			testOpts:       &Options{},
-			description:    "env var path to config",
-			configPath:     configPath,
-			expectedErrMsg: "",
-		},
-		{
-			testOpts:       &Options{},
-			description:    "env var path to no config",
-			configPath:     noConfigPath,
-			expectedErrMsg: fmt.Sprintf("problem loading config: open %s: %s", noConfigPath, noSuchFileMessage()),
-		},
-	}
-
-	g := goblin.Goblin(t)
-	for _, test := range tests {
-		g.Describe("TestLoadToml", func() {
-			g.It(test.description, func() {
-				opts = test.testOpts
-				if test.configPath != "" {
-					os.Setenv("GITLEAKS_CONFIG", test.configPath)
-				} else {
-					os.Clearenv()
-				}
-				_, err = newConfig()
-				if err != nil {
-					g.Assert(err.Error()).Equal(test.expectedErrMsg)
-				} else {
-					g.Assert("").Equal(test.expectedErrMsg)
-				}
-			})
-		})
-	}
-}
-
-func noSuchFileMessage() string {
-	if runtime.GOOS == "windows" {
-		// Adapt to Windws
-		return "The system cannot find the file specified."
-	}
-	return "no such file or directory"
-}

src/options.go

@@ -1,161 +0,0 @@
-package gitleaks
-
-import (
-	"fmt"
-	"net"
-	"net/url"
-	"os"
-	"path/filepath"
-	"runtime"
-	"strings"
-	"time"
-
-	"github.com/jessevdk/go-flags"
-	colorable "github.com/mattn/go-colorable"
-	log "github.com/sirupsen/logrus"
-)
-
-// Options for gitleaks
-type Options struct {
-	// remote target options
-	Repo       string `short:"r" long:"repo" description:"Repo url to audit"`
-	GithubUser string `long:"github-user" description:"Github user to audit"`
-	GithubOrg  string `long:"github-org" description:"Github organization to audit"`
-	GithubURL  string `long:"github-url" default:"https://api.github.com/" description:"GitHub API Base URL, use for GitHub Enterprise. Example: https://github.example.com/api/v3/"`
-	GithubPR   string `long:"github-pr" description:"Github PR url to audit. This does not clone the repo. GITHUB_TOKEN must be set"`
-
-	GitLabUser string `long:"gitlab-user" description:"GitLab user ID to audit"`
-	GitLabOrg  string `long:"gitlab-org" description:"GitLab group ID to audit"`
-
-	CommitStop string `long:"commit-stop" description:"sha of commit to stop at"`
-	Commit     string `long:"commit" description:"sha of commit to audit"`
-	Depth      int64  `long:"depth" description:"maximum commit depth"`
-
-	// local target option
-	RepoPath  string `long:"repo-path" description:"Path to repo"`
-	OwnerPath string `long:"owner-path" description:"Path to owner directory (repos discovered)"`
-
-	// Process options
-	Threads      int    `long:"threads" description:"Maximum number of threads gitleaks spawns"`
-	Disk         bool   `long:"disk" description:"Clones repo(s) to disk"`
-	ConfigPath   string `long:"config" description:"path to gitleaks config"`
-	SSHKey       string `long:"ssh-key" description:"path to ssh key"`
-	ExcludeForks bool   `long:"exclude-forks" description:"exclude forks for organization/user audits"`
-	RepoConfig   bool   `long:"repo-config" description:"Load config from target repo. Config file must be \".gitleaks.toml\""`
-	Branch       string `long:"branch" description:"Branch to audit"`
-	// TODO: IncludeMessages  string `long:"messages" description:"include commit messages in audit"`
-
-	// Output options
-	Log          string `short:"l" long:"log" description:"log level"`
-	Verbose      bool   `short:"v" long:"verbose" description:"Show verbose output from gitleaks audit"`
-	Report       string `long:"report" description:"path to write report file. Needs to be csv or json"`
-	Redact       bool   `long:"redact" description:"redact secrets from log messages and report"`
-	Version      bool   `long:"version" description:"version number"`
-	SampleConfig bool   `long:"sample-config" description:"prints a sample config file"`
-}
-
-// ParseOpts parses the options
-func ParseOpts() *Options {
-	var opts Options
-	parser := flags.NewParser(&opts, flags.Default)
-	_, err := parser.Parse()
-
-	if err != nil {
-		if flagsErr, ok := err.(*flags.Error); ok && flagsErr.Type != flags.ErrHelp {
-			parser.WriteHelp(os.Stdout)
-		}
-		os.Exit(0)
-	}
-
-	if len(os.Args) == 1 {
-		parser.WriteHelp(os.Stdout)
-		os.Exit(0)
-	}
-
-	if opts.Version {
-		fmt.Println(version)
-		os.Exit(0)
-	}
-	if opts.SampleConfig {
-		fmt.Println(defaultConfig)
-		os.Exit(0)
-	}
-
-	opts.setLogs()
-
-	err = opts.guard()
-	if err != nil {
-		log.Fatal(err)
-	}
-	return &opts
-}
-
-// optsGuard prevents invalid options
-func (opts *Options) guard() error {
-	if opts.GithubOrg != "" && opts.GithubUser != "" {
-		return fmt.Errorf("github user and organization set")
-	} else if opts.GithubOrg != "" && opts.OwnerPath != "" {
-		return fmt.Errorf("github organization set and local owner path")
-	} else if opts.GithubUser != "" && opts.OwnerPath != "" {
-		return fmt.Errorf("github user set and local owner path")
-	}
-
-	if opts.Threads > runtime.GOMAXPROCS(0) {
-		return fmt.Errorf("%d available threads", runtime.GOMAXPROCS(0))
-	}
-
-	// do the URL Parse and error checking here, so we can skip it later
-	// empty string is OK, it will default to the public github URL.
-	if opts.GithubURL != "" && opts.GithubURL != defaultGithubURL {
-		if !strings.HasSuffix(opts.GithubURL, "/") {
-			opts.GithubURL += "/"
-		}
-		ghURL, err := url.Parse(opts.GithubURL)
-		if err != nil {
-			return err
-		}
-		tcpPort := "443"
-		if ghURL.Scheme == "http" {
-			tcpPort = "80"
-		}
-		timeout := time.Duration(1 * time.Second)
-		_, err = net.DialTimeout("tcp", ghURL.Host+":"+tcpPort, timeout)
-		if err != nil {
-			return fmt.Errorf("%s unreachable, error: %s", ghURL.Host, err)
-		}
-	}
-
-	if opts.Report != "" {
-		if !strings.HasSuffix(opts.Report, ".json") && !strings.HasSuffix(opts.Report, ".csv") {
-			return fmt.Errorf("Report should be a .json or .csv file")
-		}
-		dirPath := filepath.Dir(opts.Report)
-		if _, err := os.Stat(dirPath); os.IsNotExist(err) {
-			return fmt.Errorf("%s does not exist", dirPath)
-		}
-	}
-
-	return nil
-}
-
-// setLogLevel sets log level for gitleaks. Default is Warning
-func (opts *Options) setLogs() {
-	switch opts.Log {
-	case "info":
-		log.SetLevel(log.InfoLevel)
-	case "debug":
-		log.SetLevel(log.DebugLevel)
-	case "warn":
-		log.SetLevel(log.WarnLevel)
-	default:
-		log.SetLevel(log.InfoLevel)
-	}
-	log.SetFormatter(&log.TextFormatter{
-		ForceColors:   true,
-		FullTimestamp: true,
-	})
-	// Fix colors on Windows
-	if runtime.GOOS == "windows" {
-		log.SetOutput(colorable.NewColorableStdout())
-	}
-}

src/repo.go

@@ -1,481 +0,0 @@
-package gitleaks
-
-import (
-	"crypto/md5"
-	"fmt"
-	"github.com/hako/durafmt"
-	"os"
-	"path/filepath"
-	"strings"
-	"sync"
-	"time"
-
-	log "github.com/sirupsen/logrus"
-	"gopkg.in/src-d/go-git.v4"
-	"gopkg.in/src-d/go-git.v4/plumbing"
-	diffType "gopkg.in/src-d/go-git.v4/plumbing/format/diff"
-	"gopkg.in/src-d/go-git.v4/plumbing/object"
-	"gopkg.in/src-d/go-git.v4/plumbing/storer"
-	gitHttp "gopkg.in/src-d/go-git.v4/plumbing/transport/http"
-	"gopkg.in/src-d/go-git.v4/storage/memory"
-	"gopkg.in/src-d/go-git.v4/utils/merkletrie"
-)
-
-// Commit represents a git commit
-type Commit struct {
-	content  string
-	commit   *object.Commit
-	filePath string
-	repoName string
-	sha      string
-	message  string
-	author   string
-	email    string
-	date     time.Time
-}
-
-
-// Leak represents a leaked secret or regex match.
-type Leak struct {
-	Line     string    `json:"line"`
-	Commit   string    `json:"commit"`
-	Offender string    `json:"offender"`
-	Rule     string    `json:"rule"`
-	Info     string    `json:"info"`
-	Message  string    `json:"commitMsg"`
-	Author   string    `json:"author"`
-	Email    string    `json:"email"`
-	File     string    `json:"file"`
-	Repo     string    `json:"repo"`
-	Date     time.Time `json:"date"`
-	Tags     string    `json:"tags"`
-	Severity string    `json:"severity"`
-}
-
-// Repo contains a src-d git repository and other data about the repo
-type Repo struct {
-	leaks         []Leak
-	path          string
-	url           string
-	name          string
-	repository    *git.Repository
-	err           error
-	auditDuration string
-	numCommits    int64
-}
-
-func newRepo() (*Repo, error) {
-	for _, re := range config.WhiteList.repos {
-		if re.FindString(opts.Repo) != "" {
-			return nil, fmt.Errorf("skipping %s, whitelisted", opts.Repo)
-		}
-	}
-	return &Repo{
-		path: opts.RepoPath,
-		url:  opts.Repo,
-		name: filepath.Base(opts.Repo),
-	}, nil
-}
-
-// clone will clone a repo
-func (repo *Repo) clone() error {
-	var (
-		err        error
-		repository *git.Repository
-	)
-
-	// check if cloning to disk
-	if opts.Disk {
-		log.Infof("cloning %s to disk", opts.Repo)
-		cloneTarget := fmt.Sprintf("%s/%x", dir, md5.Sum([]byte(fmt.Sprintf("%s%s", opts.GithubUser, opts.Repo))))
-		if strings.HasPrefix(opts.Repo, "git") {
-			// private
-			repository, err = git.PlainClone(cloneTarget, false, &git.CloneOptions{
-				URL:      opts.Repo,
-				Progress: os.Stdout,
-				Auth:     config.sshAuth,
-			})
-		} else {
-			// public
-			options := &git.CloneOptions{
-				URL:      opts.Repo,
-				Progress: os.Stdout,
-			}
-			if os.Getenv("GITHUB_TOKEN") != "" {
-				options.Auth = &gitHttp.BasicAuth{
-					Username: "fakeUsername", // yes, this can be anything except an empty string
-					Password: os.Getenv("GITHUB_TOKEN"),
-				}
-			}
-			repository, err = git.PlainClone(cloneTarget, false, options)
-		}
-	} else if repo.path != "" {
-		log.Infof("opening %s", repo.path)
-		repository, err = git.PlainOpen(repo.path)
-		if err != nil {
-			log.Errorf("unable to open %s", repo.path)
-		}
-	} else {
-		// cloning to memory
-		log.Infof("cloning %s", opts.Repo)
-		if strings.HasPrefix(opts.Repo, "git") {
-			repository, err = git.Clone(memory.NewStorage(), nil, &git.CloneOptions{
-				URL:      opts.Repo,
-				Progress: os.Stdout,
-				Auth:     config.sshAuth,
-			})
-		} else {
-			options := &git.CloneOptions{
-				URL:      opts.Repo,
-				Progress: os.Stdout,
-			}
-			if os.Getenv("GITHUB_TOKEN") != "" {
-				options.Auth = &gitHttp.BasicAuth{
-					Username: "fakeUsername", // yes, this can be anything except an empty string
-					Password: os.Getenv("GITHUB_TOKEN"),
-				}
-			}
-			repository, err = git.Clone(memory.NewStorage(), nil, options)
-		}
-	}
-	repo.repository = repository
-	repo.err = err
-	return err
-}
-
-// audit performs an audit
-func (repo *Repo) audit() error {
-	var (
-		err         error
-		commitCount int64
-		commitWg    sync.WaitGroup
-		semaphore   chan bool
-		logOpts     git.LogOptions
-	)
-	for _, re := range config.WhiteList.repos {
-		if re.FindString(repo.name) != "" {
-			return fmt.Errorf("skipping %s, whitelisted", repo.name)
-		}
-	}
-
-	start := time.Now()
-
-	// check if target contains an external gitleaks toml
-	if opts.RepoConfig {
-		err := config.updateFromRepo(repo)
-		if err != nil {
-			log.Warn(err)
-		}
-	}
-
-	if opts.Commit != "" {
-		h := plumbing.NewHash(opts.Commit)
-		c, err := repo.repository.CommitObject(h)
-		if err != nil {
-			return err
-		}
-
-		totalCommits = totalCommits + 1
-		repo.numCommits = 1
-		return repo.auditSingleCommit(c)
-	} else if opts.Branch != "" {
-		refs, err := repo.repository.Storer.IterReferences()
-		if err != nil {
-			return err
-		}
-		err = refs.ForEach(func(ref *plumbing.Reference) error {
-			if ref.Name().IsTag() {
-				return nil
-			}
-			// check heads first
-			if ref.Name().String() == "refs/heads/"+opts.Branch {
-				logOpts = git.LogOptions{
-					From: ref.Hash(),
-				}
-				return nil
-			} else if ref.Name().String() == "refs/remotes/origin/"+opts.Branch {
-				logOpts = git.LogOptions{
-					From: ref.Hash(),
-				}
-				return nil
-			}
-			return nil
-		})
-	} else {
-		logOpts = git.LogOptions{
-			All: true,
-		}
-	}
-
-	// iterate all through commits
-	cIter, err := repo.repository.Log(&logOpts)
-	if err != nil {
-		return err
-	}
-
-	if opts.Threads != 0 {
-		threads = opts.Threads
-	}
-	if opts.RepoPath != "" {
-		threads = 1
-	}
-	semaphore = make(chan bool, threads)
-
-	err = cIter.ForEach(func(c *object.Commit) error {
-		if c == nil || (opts.Depth != 0 && commitCount == opts.Depth) {
-			return storer.ErrStop
-		}
-
-		if config.WhiteList.commits[c.Hash.String()] {
-			log.Infof("skipping commit: %s\n", c.Hash.String())
-			return nil
-		}
-
-		// commits w/o parent (root of git the git ref)
-		if len(c.ParentHashes) == 0 {
-			commitCount = commitCount + 1
-			totalCommits = totalCommits + 1
-			err := repo.auditSingleCommit(c)
-			if err != nil {
-				return err
-			}
-			return nil
-		}
-
-		commitCount = commitCount + 1
-		totalCommits = totalCommits + 1
-
-		// regular commit audit
-		err = c.Parents().ForEach(func(parent *object.Commit) error {
-			commitWg.Add(1)
-			semaphore <- true
-			go func(c *object.Commit, parent *object.Commit) {
-				var (
-					filePath string
-					skipFile bool
-				)
-				defer func() {
-					commitWg.Done()
-					<-semaphore
-					if r := recover(); r != nil {
-						log.Warnf("recovering from panic on commit %s, likely large diff causing panic", c.Hash.String())
-					}
-				}()
-				patch, err := c.Patch(parent)
-				if err != nil {
-					log.Warnf("problem generating patch for commit: %s\n", c.Hash.String())
-					return
-				}
-				for _, f := range patch.FilePatches() {
-					if f.IsBinary() {
-						continue
-					}
-					skipFile = false
-					from, to := f.Files()
-					filePath = "???"
-					if from != nil {
-						filePath = from.Path()
-					} else if to != nil {
-						filePath = to.Path()
-					}
-
-					for _, fr := range config.FileRules {
-						for _, r := range fr.fileTypes {
-							if r.FindString(filePath) != "" {
-								commitInfo := &Commit{
-									repoName: repo.name,
-									filePath: filePath,
-									sha:      c.Hash.String(),
-									author:   c.Author.Name,
-									email:    c.Author.Email,
-									message:  strings.Replace(c.Message, "\n", " ", -1),
-									date:     c.Author.When,
-								}
-								leak := *newLeak("N/A", fmt.Sprintf("filetype %s found", r.String()), r.String(), fr, commitInfo)
-								mutex.Lock()
-								repo.leaks = append(repo.leaks, leak)
-								mutex.Unlock()
-							}
-						}
-					}
-
-					for _, re := range config.WhiteList.files {
-						if re.FindString(filePath) != "" {
-							log.Debugf("skipping whitelisted file (matched regex '%s'): %s", re.String(), filePath)
-							skipFile = true
-							break
-						}
-					}
-					if skipFile {
-						continue
-					}
-					chunks := f.Chunks()
-					for _, chunk := range chunks {
-						if chunk.Type() == diffType.Add || chunk.Type() == diffType.Delete {
-							diff := &Commit{
-								repoName: repo.name,
-								filePath: filePath,
-								content:  chunk.Content(),
-								sha:      c.Hash.String(),
-								author:   c.Author.Name,
-								email:    c.Author.Email,
-								message:  strings.Replace(c.Message, "\n", " ", -1),
-								date:     c.Author.When,
-							}
-							chunkLeaks := inspect(diff)
-							for _, leak := range chunkLeaks {
-								mutex.Lock()
-								repo.leaks = append(repo.leaks, leak)
-								mutex.Unlock()
-							}
-						}
-					}
-				}
-			}(c, parent)
-
-			return nil
-		})
-
-		return nil
-	})
-
-	commitWg.Wait()
-	repo.numCommits = commitCount
-	repo.auditDuration = durafmt.Parse(time.Now().Sub(start)).String()
-
-	return nil
-}
-
-func (repo *Repo) auditSingleCommit(c *object.Commit) error {
-	fIter, err := c.Files()
-	if err != nil {
-		return err
-	}
-
-	// If current commit has parents then search for leaks in tree change,
-	// that means scan in changed/modified files from one commit to another.
-	if len(c.ParentHashes) > 0 {
-		prevCommitObject, err := c.Parents().Next()
-		if err != nil {
-			return err
-		}
-		return repo.auditTreeChange(prevCommitObject, c)
-	}
-
-	// Scan for leaks in files related to current commit
-	err = fIter.ForEach(func(f *object.File) error {
-		bin, err := f.IsBinary()
-		if bin || err != nil {
-			return nil
-		}
-		for _, re := range config.WhiteList.files {
-			if re.FindString(f.Name) != "" {
-				log.Debugf("skipping whitelisted file (matched regex '%s'): %s", re.String(), f.Name)
-				return nil
-			}
-		}
-		content, err := f.Contents()
-		if err != nil {
-			return nil
-		}
-		diff := &Commit{
-			repoName: repo.name,
-			filePath: f.Name,
-			content:  content,
-			sha:      c.Hash.String(),
-			author:   c.Author.Name,
-			email:    c.Author.Email,
-			message:  strings.Replace(c.Message, "\n", " ", -1),
-			date:     c.Author.When,
-		}
-		fileLeaks := inspect(diff)
-		mutex.Lock()
-		repo.leaks = append(repo.leaks, fileLeaks...)
-		mutex.Unlock()
-		return nil
-	})
-	return err
-}
-
-func (repo *Repo) report() {
-	if len(repo.leaks) != 0 {
-		log.Warnf("%d leaks detected. %d commits inspected in %s", len(repo.leaks), repo.numCommits, repo.auditDuration)
-	} else {
-		log.Infof("No leaks detected. %d commits inspected in %s", repo.numCommits, repo.auditDuration)
-	}
-}
-
-// auditTreeChange will search for leaks in changed/modified files from one
-// commit to another
-func (repo *Repo) auditTreeChange(src, dst *object.Commit) error {
-	var (
-		skip bool
-	)
-
-	// Get state of src commit
-	srcState, err := src.Tree()
-	if err != nil {
-		return err
-	}
-
-	// Get state of destination commit
-	dstState, err := dst.Tree()
-	if err != nil {
-		return err
-	}
-	changes, err := srcState.Diff(dstState)
-
-	// Run through each change
-	for _, change := range changes {
-
-		// Ignore deleted files
-		action, err := change.Action()
-		if err != nil {
-			return err
-		}
-		if action == merkletrie.Delete {
-			continue
-		}
-
-		// Get list of involved files
-		_, to, err := change.Files()
-		bin, err := to.IsBinary()
-		if bin || err != nil {
-			continue
-		}
-
-		for _, re := range config.WhiteList.files {
-			if re.FindString(to.Name) != "" {
-				log.Debugf("skipping whitelisted file (matched regex '%s'): %s", re.String(), to.Name)
-				skip = true
-			}
-		}
-
-		if skip {
-			skip = false
-			continue
-		}
-
-		content, err := to.Contents()
-		if err != nil {
-			return err
-		}
-
-		diff := &Commit{
-			repoName: repo.name,
-			filePath: to.Name,
-			content:  content,
-			sha:      dst.Hash.String(),
-			author:   dst.Author.Name,
-			email:    dst.Author.Email,
-			message:  strings.Replace(dst.Message, "\n", " ", -1),
-			date:     dst.Author.When,
-		}
-		fileLeaks := inspect(diff)
-		mutex.Lock()
-		repo.leaks = append(repo.leaks, fileLeaks...)
-		mutex.Unlock()
-	}
-	return nil
-
-}

src/utils.go

@@ -1,226 +0,0 @@
-package gitleaks
-
-import (
-	"encoding/csv"
-	"encoding/json"
-	"fmt"
-	"io/ioutil"
-	"os"
-	"path"
-	"strings"
-	"time"
-
-	log "github.com/sirupsen/logrus"
-)
-
-// writeReport writes a report to a file specified in the --report= option.
-// Default format for report is JSON. You can use the --csv option to write the report as a csv
-func writeReport(leaks []Leak) error {
-	if len(leaks) == 0 {
-		return nil
-	}
-
-	log.Infof("writing report to %s", opts.Report)
-	if strings.HasSuffix(opts.Report, ".csv") {
-		f, err := os.Create(opts.Report)
-		if err != nil {
-			return err
-		}
-		defer f.Close()
-		w := csv.NewWriter(f)
-		w.Write([]string{"repo", "line", "commit", "offender", "rule", "info", "tags", "severity", "commitMsg", "author", "email", "file", "date"})
-		for _, leak := range leaks {
-			w.Write([]string{leak.Repo, leak.Line, leak.Commit, leak.Offender, leak.Rule, leak.Info, leak.Tags, leak.Severity, leak.Message, leak.Author, leak.Email, leak.File, leak.Date.Format(time.RFC3339)})
-		}
-		w.Flush()
-	} else {
-		f, err := os.Create(opts.Report)
-		if err != nil {
-			return err
-		}
-		defer f.Close()
-		encoder := json.NewEncoder(f)
-		encoder.SetIndent("", "\t")
-		if _, err := f.WriteString("[\n"); err != nil {
-			return err
-		}
-		for i := 0; i < len(leaks); i++ {
-			if err := encoder.Encode(leaks[i]); err != nil {
-				return err
-			}
-			// for all but the last leak, seek back and overwrite the newline appended by Encode() with comma & newline
-			if i+1 < len(leaks) {
-				if _, err := f.Seek(-1, 1); err != nil {
-					return err
-				}
-				if _, err := f.WriteString(",\n"); err != nil {
-					return err
-				}
-			}
-		}
-		if _, err := f.WriteString("]"); err != nil {
-			return err
-		}
-		if err := f.Sync(); err != nil {
-			log.Error(err)
-			return err
-		}
-	}
-	return nil
-}
-
-// check rule will inspect a single line and return a leak if it encounters one
-func (rule *Rule) check(line string, commit *Commit) (*Leak, error) {
-	var (
-		match       string
-		fileMatch   string
-		entropy     float64
-		entropyWord string
-	)
-
-	for _, f := range rule.fileTypes {
-		fileMatch = f.FindString(commit.filePath)
-		if fileMatch != "" {
-			break
-		}
-	}
-
-	if fileMatch == "" && len(rule.fileTypes) != 0 {
-		return nil, nil
-	}
-
-	if rule.entropies != nil {
-		if rule.entropyROI == "word" {
-			words := strings.Fields(line)
-			for _, word := range words {
-				_entropy := getShannonEntropy(word)
-				for _, e := range rule.entropies {
-					if _entropy > e.v1 && _entropy < e.v2 {
-						entropy = _entropy
-						entropyWord = word
-						goto postEntropy
-					}
-				}
-			}
-		} else {
-			_entropy := getShannonEntropy(line)
-			for _, e := range rule.entropies {
-				if _entropy > e.v1 && _entropy < e.v2 {
-					entropy = _entropy
-					entropyWord = line
-					goto postEntropy
-				}
-			}
-		}
-	}
-
-postEntropy:
-	if rule.regex != nil {
-		match = rule.regex.FindString(line)
-	}
-
-	if match != "" && entropy != 0.0 {
-		return newLeak(line, fmt.Sprintf("%s regex match and entropy met at %.2f", rule.regex.String(), entropy), entropyWord, rule, commit), nil
-	} else if match != "" && rule.entropies == nil {
-		return newLeak(line, fmt.Sprintf("%s regex match", rule.regex.String()), match, rule, commit), nil
-	} else if entropy != 0.0 && rule.regex.String() == "" {
-		return newLeak(line, fmt.Sprintf("entropy met at %.2f", entropy), entropyWord, rule, commit), nil
-	}
-	return nil, nil
-}
-
-// inspect will parse each line of the git diff's content against a set of regexes or
-// a set of regexes set by the config (see gitleaks.toml for example). This function
-// will skip lines that include a whitelisted regex. A list of leaks is returned.
-// If verbose mode (-v/--verbose) is set, then checkDiff will log leaks as they are discovered.
-func inspect(commit *Commit) []Leak {
-	var leaks []Leak
-	lines := strings.Split(commit.content, "\n")
-
-	for _, line := range lines {
-		for _, rule := range config.Rules {
-			if isLineWhitelisted(line) {
-				break
-			}
-			leak, err := rule.check(line, commit)
-			if err != nil || leak == nil {
-				continue
-			}
-			leaks = append(leaks, *leak)
-		}
-	}
-	return leaks
-}
-
-// isLineWhitelisted returns true iff the line is matched by at least one of the whiteListRegexes.
-func isLineWhitelisted(line string) bool {
-	for _, wRe := range config.WhiteList.regexes {
-		whitelistMatch := wRe.FindString(line)
-		if whitelistMatch != "" {
-			return true
-		}
-	}
-	return false
-}
-
-func newLeak(line string, info string, offender string, rule *Rule, commit *Commit) *Leak {
-	leak := &Leak{
-		Line:     line,
-		Commit:   commit.sha,
-		Offender: offender,
-		Rule:     rule.description,
-		Info:     info,
-		Author:   commit.author,
-		Email:    commit.email,
-		File:     commit.filePath,
-		Repo:     commit.repoName,
-		Message:  commit.message,
-		Date:     commit.date,
-		Tags:     strings.Join(rule.tags, ", "),
-		Severity: rule.severity,
-	}
-	if opts.Redact {
-		leak.Offender = "REDACTED"
-		leak.Line = strings.Replace(line, offender, "REDACTED", -1)
-	}
-
-	if opts.Verbose {
-		leak.log()
-	}
-	return leak
-}
-
-// discoverRepos walks all the children of `path`. If a child directory
-// contain a .git subdirectory then that repo will be added to the list of repos returned
-func discoverRepos(ownerPath string) ([]*Repo, error) {
-	var (
-		err    error
-		repoDs []*Repo
-	)
-	files, err := ioutil.ReadDir(ownerPath)
-	if err != nil {
-		return repoDs, err
-	}
-	for _, f := range files {
-		repoPath := path.Join(ownerPath, f.Name())
-		if f.IsDir() && containsGit(repoPath) {
-			repoDs = append(repoDs, &Repo{
-				name: f.Name(),
-				path: repoPath,
-			})
-		}
-	}
-	return repoDs, err
-}
-
-func (leak Leak) log() {
-	b, _ := json.MarshalIndent(leak, "", "   ")
-	fmt.Println(string(b))
-}
-
-func containsGit(repoPath string) bool {
-	if _, err := os.Stat(repoPath); os.IsNotExist(err) {
-		return false
-	}
-	return true
-}

test_data/test_configs/aws_key.toml

@@ -0,0 +1,9 @@
+[[rules]]
+	description = "AWS Secret Key"
+	regex = '''(?i)aws(.{0,20})?(?-i)['\"][0-9a-zA-Z\/+]{40}['\"]'''
+	tags = ["key", "AWS"]
+
+[[rules]]
+    description = "AWS Manager ID"
+    regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
+    tags = ["key", "AWS"]

test_data/test_configs/aws_key_file_regex.toml

@@ -0,0 +1,13 @@
+[[rules]]
+	description = "AWS Secret Key"
+	regex = '''(?i)aws(.{0,20})?(?-i)['\"][0-9a-zA-Z\/+]{40}['\"]'''
+	tags = ["key", "AWS"]
+
+[[rules]]
+    description = "AWS Manager ID"
+    regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
+    tags = ["key", "AWS"]
+
+[Global]
+    file = '''(.*)?py$'''
+

test_data/test_configs/aws_key_global_whitelist_file.toml

@@ -0,0 +1,8 @@
+[[rules]]
+    description = "AWS Manager ID"
+    regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
+    tags = ["key", "AWS"]
+
+[whitelist]
+    description = "ignore md files"
+    file = '''(.*)?md$'''

test_data/test_configs/aws_key_whitelist_files.toml

@@ -0,0 +1,14 @@
+[[rules]]
+    description = "AWS Manager ID"
+    regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
+    tags = ["key", "AWS"]
+        [[rules.whitelist]]
+            description = "ignore md files"
+            file = '''(.*)?md$'''
+        [[rules.whitelist]]
+            description = "ignore this regex"
+            regex = '''ignore$'''
+        [[rules.whitelist]]
+            description = "ignore regex and md files"
+            regex = '''ignore$'''
+            file = '''(.*)?md$'''

test_data/test_configs/aws_key_whitelist_python_files.toml

@@ -0,0 +1,7 @@
+[[rules]]
+    description = "AWS Manager ID"
+    regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
+    tags = ["key", "AWS"]
+        [[rules.whitelist]]
+            description = "ignore python files"
+            file = '''(.*)?py$'''

test_data/test_configs/bad_aws_key.toml

@@ -0,0 +1,9 @@
+[[rules]]
+	description = "AWS Secret Key"
+	regex = '''(?i)aws(.{0,20})?(?-i)['\"][0-9a-zA-Z\/+]{40}['\"]'''
+	tags = ["key", "AWS"]
+
+[[rules]]
+    description = AWS Manager ID"
+    regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
+    tags = ["key", "AWS"]

test_data/test_configs/bad_aws_key_file_regex.toml

@@ -0,0 +1,13 @@
+[[rules]]
+	description = "AWS Secret Key"
+	regex = '''(?i)aws(.{0,20})?(?-i)['\"][0-9a-zA-Z\/+]{40}['\"]'''
+	tags = ["key", "AWS"]
+
+[[rules]]
+    description = "AWS Manager ID"
+    regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
+    tags = ["key", "AWS"]
+
+[Global]
+    file = '''?????????????'''
+

test_data/test_configs/bad_aws_key_global_whitelist_file.toml

@@ -0,0 +1,8 @@
+[[rules]]
+    description = "AWS Manager ID"
+    regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
+    tags = ["key", "AWS"]
+
+[whitelist]
+    description = "ignore md files"
+    file = '''???????'''

test_data/test_configs/bad_aws_key_message_regex.toml

@@ -0,0 +1,13 @@
+[[rules]]
+	description = "AWS Secret Key"
+	regex = '''(?i)aws(.{0,20})?(?-i)['\"][0-9a-zA-Z\/+]{40}['\"]'''
+	tags = ["key", "AWS"]
+
+[[rules]]
+    description = "AWS Manager ID"
+    regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
+    tags = ["key", "AWS"]
+
+[Global]
+    message = '''?????????????'''
+

test_data/test_configs/bad_entropy_1.toml

@@ -0,0 +1,7 @@
+[[rules]]
+	description = "entropy"
+    entropies = [
+        "4.3-4.1",
+    ]
+	tags = ["entropy"]
+

test_data/test_configs/bad_entropy_2.toml

@@ -0,0 +1,7 @@
+[[rules]]
+	description = "entropy"
+    entropies = [
+        "4.3-x",
+    ]
+	tags = ["entropy"]
+

test_data/test_configs/bad_entropy_3.toml

@@ -0,0 +1,7 @@
+[[rules]]
+	description = "entropy"
+    entropies = [
+        "x-4.3",
+    ]
+	tags = ["entropy"]
+

test_data/test_configs/bad_entropy_4.toml

@@ -0,0 +1,7 @@
+[[rules]]
+	description = "entropy"
+    entropies = [
+        "1-8.9",
+    ]
+	tags = ["entropy"]
+

test_data/test_configs/bad_regex_aws_key.toml

@@ -0,0 +1,9 @@
+[[rules]]
+	description = "AWS Secret Key"
+	regex = '''$$$???$$$??$?$?$'''
+	tags = ["key", "AWS"]
+
+[[rules]]
+    description = "AWS Manager ID"
+    regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
+    tags = ["key", "AWS"]

test_data/test_configs/entropy.toml

@@ -0,0 +1,8 @@
+[[rules]]
+	description = "entropy"
+    entropies = [
+        "4.5-4.7",
+        "5.5-6.3",
+    ]
+	tags = ["entropy"]
+

test_data/test_configs/large.toml

@@ -0,0 +1,145 @@
+title = "gitleaks config"
+
+[[rules]]
+	description = "AWS Secret Key"
+	regex = '''(?i)aws(.{0,20})?(?-i)['\"][0-9a-zA-Z\/+]{40}['\"]'''
+	tags = ["key", "AWS"]
+
+[[rules]]
+	description = "AWS MWS key"
+	regex = '''amzn\.mws\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'''
+	tags = ["key", "AWS", "MWS"]
+
+[[rules]]
+	description = "Facebook Secret Key"
+	regex = '''(?i)(facebook|fb)(.{0,20})?(?-i)['\"][0-9a-f]{32}['\"]'''
+	tags = ["key", "Facebook"]
+
+[[rules]]
+	description = "Facebook Client ID"
+	regex = '''(?i)(facebook|fb)(.{0,20})?['\"][0-9]{13,17}['\"]'''
+	tags = ["key", "Facebook"]
+
+[[rules]]
+	description = "Twitter Secret Key"
+	regex = '''(?i)twitter(.{0,20})?['\"][0-9a-z]{35,44}['\"]'''
+	tags = ["key", "Twitter"]
+
+[[rules]]
+	description = "Twitter Client ID"
+	regex = '''(?i)twitter(.{0,20})?['\"][0-9a-z]{18,25}['\"]'''
+	tags = ["client", "Twitter"]
+
+[[rules]]
+	description = "Github"
+	regex = '''(?i)github(.{0,20})?(?-i)['\"][0-9a-zA-Z]{35,40}['\"]'''
+	tags = ["key", "Github"]
+
+[[rules]]
+	description = "LinkedIn Client ID"
+	regex = '''(?i)linkedin(.{0,20})?(?-i)['\"][0-9a-z]{12}['\"]'''
+	tags = ["client", "LinkedIn"]
+
+[[rules]]
+	description = "LinkedIn Secret Key"
+	regex = '''(?i)linkedin(.{0,20})?['\"][0-9a-z]{16}['\"]'''
+	tags = ["secret", "LinkedIn"]
+
+[[rules]]
+	description = "Slack"
+	regex = '''xox[baprs]-([0-9a-zA-Z]{10,48})?'''
+	tags = ["key", "Slack"]
+
+[[rules]]
+	description = "EC"
+	regex = '''-----BEGIN EC PRIVATE KEY-----'''
+	tags = ["key", "EC"]
+
+[[rules]]
+	description = "Generic API key"
+	regex = '''(?i)(api_key|apikey)(.{0,20})?['|"][0-9a-zA-Z]{32,45}['|"]'''
+	tags = ["key", "API", "generic"]
+
+[[rules]]
+	description = "Generic Secret"
+	regex = '''(?i)secret(.{0,20})?['|"][0-9a-zA-Z]{32,45}['|"]'''
+	tags = ["key", "Secret", "generic"]
+
+[[rules]]
+	description = "Google API key"
+	regex = '''AIza[0-9A-Za-z\\-_]{35}'''
+	tags = ["key", "Google"]
+
+
+[[rules]]
+	description = "Heroku API key"
+	regex = '''(?i)heroku(.{0,20})?['"][0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}['"]'''
+	tags = ["key", "Heroku"]
+
+[[rules]]
+	description = "MailChimp API key"
+	regex = '''(?i)(mailchimp|mc)(.{0,20})?['"][0-9a-f]{32}-us[0-9]{1,2}['"]'''
+	tags = ["key", "Mailchimp"]
+
+[[rules]]
+	description = "Mailgun API key"
+	regex = '''(?i)(mailgun|mg)(.{0,20})?['"][0-9a-z]{32}['"]'''
+	tags = ["key", "Mailgun"]
+
+[[rules]]
+	description = "PayPal Braintree access token"
+	regex = '''access_token\$production\$[0-9a-z]{16}\$[0-9a-f]{32}'''
+	tags = ["key", "Paypal"]
+
+[[rules]]
+	description = "Picatic API key"
+	regex = '''sk_live_[0-9a-z]{32}'''
+	tags = ["key", "Picatic"]
+
+[[rules]]
+	description = "Slack Webhook"
+	regex = '''https://hooks.slack.com/services/T[a-zA-Z0-9_]{8}/B[a-zA-Z0-9_]{8}/[a-zA-Z0-9_]{24}'''
+	tags = ["key", "slack"]
+
+[[rules]]
+	description = "Stripe API key"
+	regex = '''(?i)stripe(.{0,20})?['\"][sk|rk]_live_[0-9a-zA-Z]{24}'''
+	tags = ["key", "Stripe"]
+
+[[rules]]
+	description = "Square access token"
+	regex = '''sq0atp-[0-9A-Za-z\-_]{22}'''
+	tags = ["key", "square"]
+
+[[rules]]
+	description = "Square OAuth secret"
+	regex = '''sq0csp-[0-9A-Za-z\\-_]{43}'''
+	tags = ["key", "square"]
+
+[[rules]]
+	description = "Twilio API key"
+	regex = '''(?i)twilio(.{0,20})?['\"][0-9a-f]{32}['\"]'''
+	tags = ["key", "twilio"]
+
+[[rules]]
+	description = "AWS Manager ID"
+	regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
+	tags = ["key", "AWS"]
+		[[rules.whitelist]]
+			description = "ignore common jenkins functions"
+			regex = '''whitelistme'''
+			file = '''(.*)?Jenkinsfile$'''
+		[[rules.whitelist]]
+			description = "ignore common jenkins functions2"
+			regex = '''whitelistme'''
+			file = '''(.*)?Jenkinsfile$'''
+
+# Global rules. This instructs gitleaks to ignore all .pem files or if a message with
+[Global]
+    file = '''(.*?)(pem)'''
+    message = '''somethingbad'''
+
+[whitelist]
+	description = "image whitelists"
+	file = '''(.*?)(jpg|gif|doc|pdf|bin)$'''
+

test_data/test_entropy.json

@@ -0,0 +1,16 @@
+[
+ {
+  "line": "    aws_access_key_id='AKIAIO5FODNN7EXAMPLE',",
+  "offender": "Entropy range [{P1:4.5 P2:4.7} {P1:5.5 P2:6.3}]",
+  "commit": "6557c92612d3b35979bd426d429255b3bf9fab74",
+  "repo": "test_repo_1",
+  "rule": "entropy",
+  "commitMessage": "commit 1 with secrets\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "server.test.py",
+  "date": "2019-10-24T09:29:27-04:00",
+  "tags": "entropy",
+  "severity": ""
+ }
+]

test_data/test_local_owner_aws_leak.json

@@ -0,0 +1,240 @@
+[
+ {
+  "line": "    aws_access_key_id='AKIAIO5FODNN7EXAMPLE',",
+  "offender": "AKIAIO5FODNN7EXAMPLE",
+  "commit": "6557c92612d3b35979bd426d429255b3bf9fab74",
+  "repo": "test_repo_1",
+  "rule": "AWS Manager ID",
+  "commitMessage": "commit 1 with secrets\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "server.test.py",
+  "date": "2019-10-24T09:29:27-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "    const AWSKEY = \"AKIALALEMEL33243OLIBE\"",
+  "offender": "AKIALALEMEL33243OLIB",
+  "commit": "f61cd8587b7ac1d75a89a0c9af870a2f24c60263",
+  "repo": "test_repo_2",
+  "rule": "AWS Manager ID",
+  "commitMessage": "rm secrets again\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:12:32-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "    const AWSKEY = \"AKIALALEMEL33243OLIBE\"",
+  "offender": "AKIALALEMEL33243OLIB",
+  "commit": "b2eb34a61c988afd9b4aaa9dd58c8dd7d5f14dba",
+  "repo": "test_repo_2",
+  "rule": "AWS Manager ID",
+  "commitMessage": "adding another one\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:12:08-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "Here's an AWS secret: \"AKIALALEMEL33243OLIAE\"",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "996865bb912f3bc45898a370a13aadb315014b55",
+  "repo": "test_repo_2",
+  "rule": "AWS Manager ID",
+  "commitMessage": "committing pem\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:07:41-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "Here's an AWS secret: \"AKIALALEMEL33243OLIAE\"",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "17471a5fda722a9e423f1a0d3f0d267ea009d41c",
+  "repo": "test_repo_2",
+  "rule": "AWS Manager ID",
+  "commitMessage": "wait this is actually adding an aws secret\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:01:27-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "Here's an AWS secret: AKIALALEMEL33243OLIAE",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "17471a5fda722a9e423f1a0d3f0d267ea009d41c",
+  "repo": "test_repo_2",
+  "rule": "AWS Manager ID",
+  "commitMessage": "wait this is actually adding an aws secret\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:01:27-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "\nHere's an AWS secret: AKIALALEMEL33243OLIAE",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "b10b3e2cb320a8c211fda94c4567299d37de7776",
+  "repo": "test_repo_2",
+  "rule": "AWS Manager ID",
+  "commitMessage": "adding aws key\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T12:58:39-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "const AWSKEY = \"AKIALALEMEL33243OLIAE\"",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "cd5eb8bef855f73c46b97b4c088badffdc40ebe9",
+  "repo": "test_repo_3",
+  "rule": "AWS Manager ID",
+  "commitMessage": "rm secrets\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:54:26-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "const AWSKEY = \"AKIALALEMEL33243OLIAE\"",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "84ac4e80d4dbf2c968b64e9d4005f5079795bb81",
+  "repo": "test_repo_3",
+  "rule": "AWS Manager ID",
+  "commitMessage": "more secrets\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:54:08-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "AWS secret: \"AKIALALEMEL33243OLIAE\"",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "64cfcee9aad1c84581631636bfc54f2050718d1a",
+  "repo": "test_repo_3",
+  "rule": "AWS Manager ID",
+  "commitMessage": "rm secrets\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:36:22-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "AWS secret: \"AKIALALEMEL33243OLIAE\"",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "deea550dd6c7acaf0e59432600593533984a2125",
+  "repo": "test_repo_3",
+  "rule": "AWS Manager ID",
+  "commitMessage": "dev branch\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:35:03-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "    const AWSKEY = \"AKIALALEMEL33243OLIBE\"",
+  "offender": "AKIALALEMEL33243OLIB",
+  "commit": "f61cd8587b7ac1d75a89a0c9af870a2f24c60263",
+  "repo": "test_repo_3",
+  "rule": "AWS Manager ID",
+  "commitMessage": "rm secrets again\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:12:32-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "    const AWSKEY = \"AKIALALEMEL33243OLIBE\"",
+  "offender": "AKIALALEMEL33243OLIB",
+  "commit": "b2eb34a61c988afd9b4aaa9dd58c8dd7d5f14dba",
+  "repo": "test_repo_3",
+  "rule": "AWS Manager ID",
+  "commitMessage": "adding another one\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:12:08-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "Here's an AWS secret: \"AKIALALEMEL33243OLIAE\"",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "996865bb912f3bc45898a370a13aadb315014b55",
+  "repo": "test_repo_3",
+  "rule": "AWS Manager ID",
+  "commitMessage": "committing pem\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:07:41-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "Here's an AWS secret: \"AKIALALEMEL33243OLIAE\"",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "17471a5fda722a9e423f1a0d3f0d267ea009d41c",
+  "repo": "test_repo_3",
+  "rule": "AWS Manager ID",
+  "commitMessage": "wait this is actually adding an aws secret\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:01:27-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "Here's an AWS secret: AKIALALEMEL33243OLIAE",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "17471a5fda722a9e423f1a0d3f0d267ea009d41c",
+  "repo": "test_repo_3",
+  "rule": "AWS Manager ID",
+  "commitMessage": "wait this is actually adding an aws secret\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:01:27-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "\nHere's an AWS secret: AKIALALEMEL33243OLIAE",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "b10b3e2cb320a8c211fda94c4567299d37de7776",
+  "repo": "test_repo_3",
+  "rule": "AWS Manager ID",
+  "commitMessage": "adding aws key\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T12:58:39-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ }
+]

test_data/test_local_repo_one_aws_leak.json

@@ -0,0 +1,16 @@
+[
+ {
+  "line": "    aws_access_key_id='AKIAIO5FODNN7EXAMPLE',",
+  "offender": "AKIAIO5FODNN7EXAMPLE",
+  "commit": "6557c92612d3b35979bd426d429255b3bf9fab74",
+  "repo": "test_repo_1",
+  "rule": "AWS Manager ID",
+  "commitMessage": "commit 1 with secrets\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "server.test.py",
+  "date": "2019-10-24T09:29:27-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ }
+]

test_data/test_local_repo_one_aws_leak_and_file_leak.json

@@ -0,0 +1,30 @@
+[
+ {
+  "line": "N/A",
+  "offender": "server.test.py",
+  "commit": "d274003914c707212cbe84e3e466a00013ccb639",
+  "repo": "test_repo_1",
+  "rule": "file regex matched(.*)?py$",
+  "commitMessage": "",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "server.test.py",
+  "date": "2019-10-24T10:03:38-04:00",
+  "tags": "",
+  "severity": ""
+ },
+ {
+  "line": "    aws_access_key_id='AKIAIO5FODNN7EXAMPLE',",
+  "offender": "AKIAIO5FODNN7EXAMPLE",
+  "commit": "6557c92612d3b35979bd426d429255b3bf9fab74",
+  "repo": "test_repo_1",
+  "rule": "AWS Manager ID",
+  "commitMessage": "commit 1 with secrets\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "server.test.py",
+  "date": "2019-10-24T09:29:27-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ }
+]

test_data/test_local_repo_one_aws_leak_commit.json

@@ -0,0 +1,16 @@
+[
+ {
+  "line": "    aws_access_key_id='AKIAIO5FODNN7EXAMPLE',",
+  "offender": "AKIAIO5FODNN7EXAMPLE",
+  "commit": "6557c92612d3b35979bd426d429255b3bf9fab74",
+  "repo": "test_repo_1",
+  "rule": "AWS Manager ID",
+  "commitMessage": "commit 1 with secrets\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "server.test.py",
+  "date": "2019-10-24T09:29:27-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ }
+]

test_data/test_local_repo_one_aws_leak_uncommitted.json

@@ -0,0 +1,16 @@
+[
+ {
+  "line": " aws_access_key_id='AKIAIO5FODNN7DXAMPLE'",
+  "offender": "AKIAIO5FODNN7DXAMPLE",
+  "commit": "d274003914c707212cbe84e3e466a00013ccb639",
+  "repo": "test_repo_1",
+  "rule": "AWS Manager ID",
+  "commitMessage": "comment\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "server.test.py",
+  "date": "2019-10-24T10:03:38-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ }
+]

test_data/test_local_repo_three_leaks.json

@@ -0,0 +1,114 @@
+[
+ {
+  "line": "AWS secret: \"AKIALALEMEL33243OLIAE\"",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "64cfcee9aad1c84581631636bfc54f2050718d1a",
+  "repo": "test_repo_3",
+  "rule": "AWS Manager ID",
+  "commitMessage": "rm secrets\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:36:22-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "AWS secret: \"AKIALALEMEL33243OLIAE\"",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "deea550dd6c7acaf0e59432600593533984a2125",
+  "repo": "test_repo_3",
+  "rule": "AWS Manager ID",
+  "commitMessage": "dev branch\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:35:03-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "    const AWSKEY = \"AKIALALEMEL33243OLIBE\"",
+  "offender": "AKIALALEMEL33243OLIB",
+  "commit": "f61cd8587b7ac1d75a89a0c9af870a2f24c60263",
+  "repo": "test_repo_3",
+  "rule": "AWS Manager ID",
+  "commitMessage": "rm secrets again\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:12:32-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "    const AWSKEY = \"AKIALALEMEL33243OLIBE\"",
+  "offender": "AKIALALEMEL33243OLIB",
+  "commit": "b2eb34a61c988afd9b4aaa9dd58c8dd7d5f14dba",
+  "repo": "test_repo_3",
+  "rule": "AWS Manager ID",
+  "commitMessage": "adding another one\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:12:08-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "Here's an AWS secret: \"AKIALALEMEL33243OLIAE\"",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "996865bb912f3bc45898a370a13aadb315014b55",
+  "repo": "test_repo_3",
+  "rule": "AWS Manager ID",
+  "commitMessage": "committing pem\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:07:41-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "Here's an AWS secret: \"AKIALALEMEL33243OLIAE\"",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "17471a5fda722a9e423f1a0d3f0d267ea009d41c",
+  "repo": "test_repo_3",
+  "rule": "AWS Manager ID",
+  "commitMessage": "wait this is actually adding an aws secret\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:01:27-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "Here's an AWS secret: AKIALALEMEL33243OLIAE",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "17471a5fda722a9e423f1a0d3f0d267ea009d41c",
+  "repo": "test_repo_3",
+  "rule": "AWS Manager ID",
+  "commitMessage": "wait this is actually adding an aws secret\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:01:27-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "\nHere's an AWS secret: AKIALALEMEL33243OLIAE",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "b10b3e2cb320a8c211fda94c4567299d37de7776",
+  "repo": "test_repo_3",
+  "rule": "AWS Manager ID",
+  "commitMessage": "adding aws key\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T12:58:39-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ }
+]

test_data/test_local_repo_two_leaks.json

@@ -0,0 +1,86 @@
+[
+ {
+  "line": "    const AWSKEY = \"AKIALALEMEL33243OLIBE\"",
+  "offender": "AKIALALEMEL33243OLIB",
+  "commit": "f61cd8587b7ac1d75a89a0c9af870a2f24c60263",
+  "repo": "test_repo_2",
+  "rule": "AWS Manager ID",
+  "commitMessage": "rm secrets again\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:12:32-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "    const AWSKEY = \"AKIALALEMEL33243OLIBE\"",
+  "offender": "AKIALALEMEL33243OLIB",
+  "commit": "b2eb34a61c988afd9b4aaa9dd58c8dd7d5f14dba",
+  "repo": "test_repo_2",
+  "rule": "AWS Manager ID",
+  "commitMessage": "adding another one\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:12:08-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "Here's an AWS secret: \"AKIALALEMEL33243OLIAE\"",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "996865bb912f3bc45898a370a13aadb315014b55",
+  "repo": "test_repo_2",
+  "rule": "AWS Manager ID",
+  "commitMessage": "committing pem\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:07:41-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "Here's an AWS secret: \"AKIALALEMEL33243OLIAE\"",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "17471a5fda722a9e423f1a0d3f0d267ea009d41c",
+  "repo": "test_repo_2",
+  "rule": "AWS Manager ID",
+  "commitMessage": "wait this is actually adding an aws secret\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:01:27-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "Here's an AWS secret: AKIALALEMEL33243OLIAE",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "17471a5fda722a9e423f1a0d3f0d267ea009d41c",
+  "repo": "test_repo_2",
+  "rule": "AWS Manager ID",
+  "commitMessage": "wait this is actually adding an aws secret\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T13:01:27-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ },
+ {
+  "line": "\nHere's an AWS secret: AKIALALEMEL33243OLIAE",
+  "offender": "AKIALALEMEL33243OLIA",
+  "commit": "b10b3e2cb320a8c211fda94c4567299d37de7776",
+  "repo": "test_repo_2",
+  "rule": "AWS Manager ID",
+  "commitMessage": "adding aws key\n",
+  "author": "zach rice",
+  "email": "zricer@protonmail.com",
+  "file": "secrets.md",
+  "date": "2019-10-25T12:58:39-04:00",
+  "tags": "key, AWS",
+  "severity": ""
+ }
+]

test_data/test_repos/test_repo_1/dotGit/COMMIT_EDITMSG

@@ -0,0 +1 @@
+comment

test_data/test_repos/test_repo_1/dotGit/HEAD

@@ -0,0 +1 @@
+ref: refs/heads/master

test_data/test_repos/test_repo_1/dotGit/config

@@ -0,0 +1,7 @@
+[core]
+	repositoryformatversion = 0
+	filemode = true
+	bare = false
+	logallrefupdates = true
+	ignorecase = true
+	precomposeunicode = true

test_data/test_repos/test_repo_1/dotGit/description

@@ -0,0 +1 @@
+Unnamed repository; edit this file 'description' to name the repository.

test_data/test_repos/test_repo_1/dotGit/hooks/applypatch-msg.sample

@@ -0,0 +1,15 @@
+#!/bin/sh
+#
+# An example hook script to check the commit log message taken by
+# applypatch from an e-mail message.
+#
+# The hook should exit with non-zero status after issuing an
+# appropriate message if it wants to stop the commit.  The hook is
+# allowed to edit the commit message file.
+#
+# To enable this hook, rename this file to "applypatch-msg".
+
+. git-sh-setup
+commitmsg="$(git rev-parse --git-path hooks/commit-msg)"
+test -x "$commitmsg" && exec "$commitmsg" ${1+"$@"}
+:

test_data/test_repos/test_repo_1/dotGit/hooks/commit-msg.sample

@@ -0,0 +1,24 @@
+#!/bin/sh
+#
+# An example hook script to check the commit log message.
+# Called by "git commit" with one argument, the name of the file
+# that has the commit message.  The hook should exit with non-zero
+# status after issuing an appropriate message if it wants to stop the
+# commit.  The hook is allowed to edit the commit message file.
+#
+# To enable this hook, rename this file to "commit-msg".
+
+# Uncomment the below to add a Signed-off-by line to the message.
+# Doing this in a hook is a bad idea in general, but the prepare-commit-msg
+# hook is more suited to it.
+#
+# SOB=$(git var GIT_AUTHOR_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p')
+# grep -qs "^$SOB" "$1" || echo "$SOB" >> "$1"
+
+# This example catches duplicate Signed-off-by lines.
+
+test "" = "$(grep '^Signed-off-by: ' "$1" |
+	 sort | uniq -c | sed -e '/^[ 	]*1[ 	]/d')" || {
+	echo >&2 Duplicate Signed-off-by lines.
+	exit 1
+}

test_data/test_repos/test_repo_1/dotGit/hooks/fsmonitor-watchman.sample

@@ -0,0 +1,114 @@
+#!/usr/bin/perl
+
+use strict;
+use warnings;
+use IPC::Open2;
+
+# An example hook script to integrate Watchman
+# (https://facebook.github.io/watchman/) with git to speed up detecting
+# new and modified files.
+#
+# The hook is passed a version (currently 1) and a time in nanoseconds
+# formatted as a string and outputs to stdout all files that have been
+# modified since the given time. Paths must be relative to the root of
+# the working tree and separated by a single NUL.
+#
+# To enable this hook, rename this file to "query-watchman" and set
+# 'git config core.fsmonitor .git/hooks/query-watchman'
+#
+my ($version, $time) = @ARGV;
+
+# Check the hook interface version
+
+if ($version == 1) {
+	# convert nanoseconds to seconds
+	$time = int $time / 1000000000;
+} else {
+	die "Unsupported query-fsmonitor hook version '$version'.\n" .
+	    "Falling back to scanning...\n";
+}
+
+my $git_work_tree;
+if ($^O =~ 'msys' || $^O =~ 'cygwin') {
+	$git_work_tree = Win32::GetCwd();
+	$git_work_tree =~ tr/\\/\//;
+} else {
+	require Cwd;
+	$git_work_tree = Cwd::cwd();
+}
+
+my $retry = 1;
+
+launch_watchman();
+
+sub launch_watchman {
+
+	my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j --no-pretty')
+	    or die "open2() failed: $!\n" .
+	    "Falling back to scanning...\n";
+
+	# In the query expression below we're asking for names of files that
+	# changed since $time but were not transient (ie created after
+	# $time but no longer exist).
+	#
+	# To accomplish this, we're using the "since" generator to use the
+	# recency index to select candidate nodes and "fields" to limit the
+	# output to file names only. Then we're using the "expression" term to
+	# further constrain the results.
+	#
+	# The category of transient files that we want to ignore will have a
+	# creation clock (cclock) newer than $time_t value and will also not
+	# currently exist.
+
+	my $query = <<"	END";
+		["query", "$git_work_tree", {
+			"since": $time,
+			"fields": ["name"],
+			"expression": ["not", ["allof", ["since", $time, "cclock"], ["not", "exists"]]]
+		}]
+	END
+
+	print CHLD_IN $query;
+	close CHLD_IN;
+	my $response = do {local $/; <CHLD_OUT>};
+
+	die "Watchman: command returned no output.\n" .
+	    "Falling back to scanning...\n" if $response eq "";
+	die "Watchman: command returned invalid output: $response\n" .
+	    "Falling back to scanning...\n" unless $response =~ /^\{/;
+
+	my $json_pkg;
+	eval {
+		require JSON::XS;
+		$json_pkg = "JSON::XS";
+		1;
+	} or do {
+		require JSON::PP;
+		$json_pkg = "JSON::PP";
+	};
+
+	my $o = $json_pkg->new->utf8->decode($response);
+
+	if ($retry > 0 and $o->{error} and $o->{error} =~ m/unable to resolve root .* directory (.*) is not watched/) {
+		print STDERR "Adding '$git_work_tree' to watchman's watch list.\n";
+		$retry--;
+		qx/watchman watch "$git_work_tree"/;
+		die "Failed to make watchman watch '$git_work_tree'.\n" .
+		    "Falling back to scanning...\n" if $? != 0;
+
+		# Watchman will always return all files on the first query so
+		# return the fast "everything is dirty" flag to git and do the
+		# Watchman query just to get it over with now so we won't pay
+		# the cost in git to look up each individual file.
+		print "/\0";
+		eval { launch_watchman() };
+		exit 0;
+	}
+
+	die "Watchman: $o->{error}.\n" .
+	    "Falling back to scanning...\n" if $o->{error};
+
+	binmode STDOUT, ":utf8";
+	local $, = "\0";
+	print @{$o->{files}};
+}

test_data/test_repos/test_repo_1/dotGit/hooks/post-update.sample

@@ -0,0 +1,8 @@
+#!/bin/sh
+#
+# An example hook script to prepare a packed repository for use over
+# dumb transports.
+#
+# To enable this hook, rename this file to "post-update".
+
+exec git update-server-info

test_data/test_repos/test_repo_1/dotGit/hooks/pre-applypatch.sample

@@ -0,0 +1,14 @@
+#!/bin/sh
+#
+# An example hook script to verify what is about to be committed
+# by applypatch from an e-mail message.
+#
+# The hook should exit with non-zero status after issuing an
+# appropriate message if it wants to stop the commit.
+#
+# To enable this hook, rename this file to "pre-applypatch".
+
+. git-sh-setup
+precommit="$(git rev-parse --git-path hooks/pre-commit)"
+test -x "$precommit" && exec "$precommit" ${1+"$@"}
+:

test_data/test_repos/test_repo_1/dotGit/hooks/pre-commit.sample

@@ -0,0 +1,49 @@
+#!/bin/sh
+#
+# An example hook script to verify what is about to be committed.
+# Called by "git commit" with no arguments.  The hook should
+# exit with non-zero status after issuing an appropriate message if
+# it wants to stop the commit.
+#
+# To enable this hook, rename this file to "pre-commit".
+
+if git rev-parse --verify HEAD >/dev/null 2>&1
+then
+	against=HEAD
+else
+	# Initial commit: diff against an empty tree object
+	against=$(git hash-object -t tree /dev/null)
+fi
+
+# If you want to allow non-ASCII filenames set this variable to true.
+allownonascii=$(git config --bool hooks.allownonascii)
+
+# Redirect output to stderr.
+exec 1>&2
+
+# Cross platform projects tend to avoid non-ASCII filenames; prevent
+# them from being added to the repository. We exploit the fact that the
+# printable range starts at the space character and ends with tilde.
+if [ "$allownonascii" != "true" ] &&
+	# Note that the use of brackets around a tr range is ok here, (it's
+	# even required, for portability to Solaris 10's /usr/bin/tr), since
+	# the square bracket bytes happen to fall in the designated range.
+	test $(git diff --cached --name-only --diff-filter=A -z $against |
+	  LC_ALL=C tr -d '[ -~]\0' | wc -c) != 0
+then
+	cat <<\EOF
+Error: Attempt to add a non-ASCII file name.
+
+This can cause problems if you want to work with people on other platforms.
+
+To be portable it is advisable to rename the file.
+
+If you know what you are doing you can disable this check using:
+
+  git config hooks.allownonascii true
+EOF
+	exit 1
+fi
+
+# If there are whitespace errors, print the offending file names and fail.
+exec git diff-index --check --cached $against --

test_data/test_repos/test_repo_1/dotGit/hooks/pre-push.sample

@@ -0,0 +1,53 @@
+#!/bin/sh
+
+# An example hook script to verify what is about to be pushed.  Called by "git
+# push" after it has checked the remote status, but before anything has been
+# pushed.  If this script exits with a non-zero status nothing will be pushed.
+#
+# This hook is called with the following parameters:
+#
+# $1 -- Name of the remote to which the push is being done
+# $2 -- URL to which the push is being done
+#
+# If pushing without using a named remote those arguments will be equal.
+#
+# Information about the commits which are being pushed is supplied as lines to
+# the standard input in the form:
+#
+#   <local ref> <local sha1> <remote ref> <remote sha1>
+#
+# This sample shows how to prevent push of commits where the log message starts
+# with "WIP" (work in progress).
+
+remote="$1"
+url="$2"
+
+z40=0000000000000000000000000000000000000000
+
+while read local_ref local_sha remote_ref remote_sha
+do
+	if [ "$local_sha" = $z40 ]
+	then
+		# Handle delete
+		:
+	else
+		if [ "$remote_sha" = $z40 ]
+		then
+			# New branch, examine all commits
+			range="$local_sha"
+		else
+			# Update to existing branch, examine new commits
+			range="$remote_sha..$local_sha"
+		fi
+
+		# Check for WIP commit
+		commit=`git rev-list -n 1 --grep '^WIP' "$range"`
+		if [ -n "$commit" ]
+		then
+			echo >&2 "Found WIP commit in $local_ref, not pushing"
+			exit 1
+		fi
+	fi
+done
+
+exit 0

test_data/test_repos/test_repo_1/dotGit/hooks/pre-rebase.sample

@@ -0,0 +1,169 @@
+#!/bin/sh
+#
+# Copyright (c) 2006, 2008 Junio C Hamano
+#
+# The "pre-rebase" hook is run just before "git rebase" starts doing
+# its job, and can prevent the command from running by exiting with
+# non-zero status.
+#
+# The hook is called with the following parameters:
+#
+# $1 -- the upstream the series was forked from.
+# $2 -- the branch being rebased (or empty when rebasing the current branch).
+#
+# This sample shows how to prevent topic branches that are already
+# merged to 'next' branch from getting rebased, because allowing it
+# would result in rebasing already published history.
+
+publish=next
+basebranch="$1"
+if test "$#" = 2
+then
+	topic="refs/heads/$2"
+else
+	topic=`git symbolic-ref HEAD` ||
+	exit 0 ;# we do not interrupt rebasing detached HEAD
+fi
+
+case "$topic" in
+refs/heads/??/*)
+	;;
+*)
+	exit 0 ;# we do not interrupt others.
+	;;
+esac
+
+# Now we are dealing with a topic branch being rebased
+# on top of master.  Is it OK to rebase it?
+
+# Does the topic really exist?
+git show-ref -q "$topic" || {
+	echo >&2 "No such branch $topic"
+	exit 1
+}
+
+# Is topic fully merged to master?
+not_in_master=`git rev-list --pretty=oneline ^master "$topic"`
+if test -z "$not_in_master"
+then
+	echo >&2 "$topic is fully merged to master; better remove it."
+	exit 1 ;# we could allow it, but there is no point.
+fi
+
+# Is topic ever merged to next?  If so you should not be rebasing it.
+only_next_1=`git rev-list ^master "^$topic" ${publish} | sort`
+only_next_2=`git rev-list ^master           ${publish} | sort`
+if test "$only_next_1" = "$only_next_2"
+then
+	not_in_topic=`git rev-list "^$topic" master`
+	if test -z "$not_in_topic"
+	then
+		echo >&2 "$topic is already up to date with master"
+		exit 1 ;# we could allow it, but there is no point.
+	else
+		exit 0
+	fi
+else
+	not_in_next=`git rev-list --pretty=oneline ^${publish} "$topic"`
+	/usr/bin/perl -e '
+		my $topic = $ARGV[0];
+		my $msg = "* $topic has commits already merged to public branch:\n";
+		my (%not_in_next) = map {
+			/^([0-9a-f]+) /;
+			($1 => 1);
+		} split(/\n/, $ARGV[1]);
+		for my $elem (map {
+				/^([0-9a-f]+) (.*)$/;
+				[$1 => $2];
+			} split(/\n/, $ARGV[2])) {
+			if (!exists $not_in_next{$elem->[0]}) {
+				if ($msg) {
+					print STDERR $msg;
+					undef $msg;
+				}
+				print STDERR " $elem->[1]\n";
+			}
+		}
+	' "$topic" "$not_in_next" "$not_in_master"
+	exit 1
+fi
+
+<<\DOC_END
+
+This sample hook safeguards topic branches that have been
+published from being rewound.
+
+The workflow assumed here is:
+
+ * Once a topic branch forks from "master", "master" is never
+   merged into it again (either directly or indirectly).
+
+ * Once a topic branch is fully cooked and merged into "master",
+   it is deleted.  If you need to build on top of it to correct
+   earlier mistakes, a new topic branch is created by forking at
+   the tip of the "master".  This is not strictly necessary, but
+   it makes it easier to keep your history simple.
+
+ * Whenever you need to test or publish your changes to topic
+   branches, merge them into "next" branch.
+
+The script, being an example, hardcodes the publish branch name
+to be "next", but it is trivial to make it configurable via
+$GIT_DIR/config mechanism.
+
+With this workflow, you would want to know:
+
+(1) ... if a topic branch has ever been merged to "next".  Young
+    topic branches can have stupid mistakes you would rather
+    clean up before publishing, and things that have not been
+    merged into other branches can be easily rebased without
+    affecting other people.  But once it is published, you would
+    not want to rewind it.
+
+(2) ... if a topic branch has been fully merged to "master".
+    Then you can delete it.  More importantly, you should not
+    build on top of it -- other people may already want to
+    change things related to the topic as patches against your
+    "master", so if you need further changes, it is better to
+    fork the topic (perhaps with the same name) afresh from the
+    tip of "master".
+
+Let's look at this example:
+
+		   o---o---o---o---o---o---o---o---o---o "next"
+		  /       /           /           /
+		 /   a---a---b A     /           /
+		/   /               /           /
+	       /   /   c---c---c---c B         /
+	      /   /   /             \         /
+	     /   /   /   b---b C     \       /
+	    /   /   /   /             \     /
+    ---o---o---o---o---o---o---o---o---o---o---o "master"
+
+
+A, B and C are topic branches.
+
+ * A has one fix since it was merged up to "next".
+
+ * B has finished.  It has been fully merged up to "master" and "next",
+   and is ready to be deleted.
+
+ * C has not merged to "next" at all.
+
+We would want to allow C to be rebased, refuse A, and encourage
+B to be deleted.
+
+To compute (1):
+
+	git rev-list ^master ^topic next
+	git rev-list ^master        next
+
+	if these match, topic has not merged in next at all.
+
+To compute (2):
+
+	git rev-list master..topic
+
+	if this is empty, it is fully merged to "master".
+
+DOC_END

test_data/test_repos/test_repo_1/dotGit/hooks/pre-receive.sample

@@ -0,0 +1,24 @@
+#!/bin/sh
+#
+# An example hook script to make use of push options.
+# The example simply echoes all push options that start with 'echoback='
+# and rejects all pushes when the "reject" push option is used.
+#
+# To enable this hook, rename this file to "pre-receive".
+
+if test -n "$GIT_PUSH_OPTION_COUNT"
+then
+	i=0
+	while test "$i" -lt "$GIT_PUSH_OPTION_COUNT"
+	do
+		eval "value=\$GIT_PUSH_OPTION_$i"
+		case "$value" in
+		echoback=*)
+			echo "echo from the pre-receive-hook: ${value#*=}" >&2
+			;;
+		reject)
+			exit 1
+		esac
+		i=$((i + 1))
+	done
+fi

test_data/test_repos/test_repo_1/dotGit/hooks/prepare-commit-msg.sample

@@ -0,0 +1,42 @@
+#!/bin/sh
+#
+# An example hook script to prepare the commit log message.
+# Called by "git commit" with the name of the file that has the
+# commit message, followed by the description of the commit
+# message's source.  The hook's purpose is to edit the commit
+# message file.  If the hook fails with a non-zero status,
+# the commit is aborted.
+#
+# To enable this hook, rename this file to "prepare-commit-msg".
+
+# This hook includes three examples. The first one removes the
+# "# Please enter the commit message..." help message.
+#
+# The second includes the output of "git diff --name-status -r"
+# into the message, just before the "git status" output.  It is
+# commented because it doesn't cope with --amend or with squashed
+# commits.
+#
+# The third example adds a Signed-off-by line to the message, that can
+# still be edited.  This is rarely a good idea.
+
+COMMIT_MSG_FILE=$1
+COMMIT_SOURCE=$2
+SHA1=$3
+
+/usr/bin/perl -i.bak -ne 'print unless(m/^. Please enter the commit message/..m/^#$/)' "$COMMIT_MSG_FILE"
+
+# case "$COMMIT_SOURCE,$SHA1" in
+#  ,|template,)
+#    /usr/bin/perl -i.bak -pe '
+#       print "\n" . `git diff --cached --name-status -r`
+# 	 if /^#/ && $first++ == 0' "$COMMIT_MSG_FILE" ;;
+#  *) ;;
+# esac
+
+# SOB=$(git var GIT_COMMITTER_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p')
+# git interpret-trailers --in-place --trailer "$SOB" "$COMMIT_MSG_FILE"
+# if test -z "$COMMIT_SOURCE"
+# then
+#   /usr/bin/perl -i.bak -pe 'print "\n" if !$first_line++' "$COMMIT_MSG_FILE"
+# fi

test_data/test_repos/test_repo_1/dotGit/hooks/update.sample

@@ -0,0 +1,128 @@
+#!/bin/sh
+#
+# An example hook script to block unannotated tags from entering.
+# Called by "git receive-pack" with arguments: refname sha1-old sha1-new
+#
+# To enable this hook, rename this file to "update".
+#
+# Config
+# ------
+# hooks.allowunannotated
+#   This boolean sets whether unannotated tags will be allowed into the
+#   repository.  By default they won't be.
+# hooks.allowdeletetag
+#   This boolean sets whether deleting tags will be allowed in the
+#   repository.  By default they won't be.
+# hooks.allowmodifytag
+#   This boolean sets whether a tag may be modified after creation. By default
+#   it won't be.
+# hooks.allowdeletebranch
+#   This boolean sets whether deleting branches will be allowed in the
+#   repository.  By default they won't be.
+# hooks.denycreatebranch
+#   This boolean sets whether remotely creating branches will be denied
+#   in the repository.  By default this is allowed.
+#
+
+# --- Command line
+refname="$1"
+oldrev="$2"
+newrev="$3"
+
+# --- Safety check
+if [ -z "$GIT_DIR" ]; then
+	echo "Don't run this script from the command line." >&2
+	echo " (if you want, you could supply GIT_DIR then run" >&2
+	echo "  $0 <ref> <oldrev> <newrev>)" >&2
+	exit 1
+fi
+
+if [ -z "$refname" -o -z "$oldrev" -o -z "$newrev" ]; then
+	echo "usage: $0 <ref> <oldrev> <newrev>" >&2
+	exit 1
+fi
+
+# --- Config
+allowunannotated=$(git config --bool hooks.allowunannotated)
+allowdeletebranch=$(git config --bool hooks.allowdeletebranch)
+denycreatebranch=$(git config --bool hooks.denycreatebranch)
+allowdeletetag=$(git config --bool hooks.allowdeletetag)
+allowmodifytag=$(git config --bool hooks.allowmodifytag)
+
+# check for no description
+projectdesc=$(sed -e '1q' "$GIT_DIR/description")
+case "$projectdesc" in
+"Unnamed repository"* | "")
+	echo "*** Project description file hasn't been set" >&2
+	exit 1
+	;;
+esac
+
+# --- Check types
+# if $newrev is 0000...0000, it's a commit to delete a ref.
+zero="0000000000000000000000000000000000000000"
+if [ "$newrev" = "$zero" ]; then
+	newrev_type=delete
+else
+	newrev_type=$(git cat-file -t $newrev)
+fi
+
+case "$refname","$newrev_type" in
+	refs/tags/*,commit)
+		# un-annotated tag
+		short_refname=${refname##refs/tags/}
+		if [ "$allowunannotated" != "true" ]; then
+			echo "*** The un-annotated tag, $short_refname, is not allowed in this repository" >&2
+			echo "*** Use 'git tag [ -a | -s ]' for tags you want to propagate." >&2
+			exit 1
+		fi
+		;;
+	refs/tags/*,delete)
+		# delete tag
+		if [ "$allowdeletetag" != "true" ]; then
+			echo "*** Deleting a tag is not allowed in this repository" >&2
+			exit 1
+		fi
+		;;
+	refs/tags/*,tag)
+		# annotated tag
+		if [ "$allowmodifytag" != "true" ] && git rev-parse $refname > /dev/null 2>&1
+		then
+			echo "*** Tag '$refname' already exists." >&2
+			echo "*** Modifying a tag is not allowed in this repository." >&2
+			exit 1
+		fi
+		;;
+	refs/heads/*,commit)
+		# branch
+		if [ "$oldrev" = "$zero" -a "$denycreatebranch" = "true" ]; then
+			echo "*** Creating a branch is not allowed in this repository" >&2
+			exit 1
+		fi
+		;;
+	refs/heads/*,delete)
+		# delete branch
+		if [ "$allowdeletebranch" != "true" ]; then
+			echo "*** Deleting a branch is not allowed in this repository" >&2
+			exit 1
+		fi
+		;;
+	refs/remotes/*,commit)
+		# tracking branch
+		;;
+	refs/remotes/*,delete)
+		# delete tracking branch
+		if [ "$allowdeletebranch" != "true" ]; then
+			echo "*** Deleting a tracking branch is not allowed in this repository" >&2
+			exit 1
+		fi
+		;;
+	*)
+		# Anything else (is there anything else?)
+		echo "*** Update hook: unknown type of update to ref $refname of type $newrev_type" >&2
+		exit 1
+		;;
+esac
+
+# --- Finished
+exit 0

test_data/test_repos/test_repo_1/dotGit/info/exclude

@@ -0,0 +1,6 @@
+# git ls-files --others --exclude-from=.git/info/exclude
+# Lines that start with '#' are comments.
+# For a project mostly in C, the following would be a good set of
+# exclude patterns (uncomment them if you want to use them):
+# *.[oa]
+# *~

test_data/test_repos/test_repo_1/dotGit/logs/HEAD

@@ -0,0 +1,2 @@
+0000000000000000000000000000000000000000 6557c92612d3b35979bd426d429255b3bf9fab74 zach rice <zricer@protonmail.com> 1571923767 -0400	commit (initial): commit 1 with secrets
+6557c92612d3b35979bd426d429255b3bf9fab74 d274003914c707212cbe84e3e466a00013ccb639 zach rice <zricer@protonmail.com> 1571925818 -0400	commit: comment

test_data/test_repos/test_repo_1/dotGit/logs/refs/heads/master

@@ -0,0 +1,2 @@
+0000000000000000000000000000000000000000 6557c92612d3b35979bd426d429255b3bf9fab74 zach rice <zricer@protonmail.com> 1571923767 -0400	commit (initial): commit 1 with secrets
+6557c92612d3b35979bd426d429255b3bf9fab74 d274003914c707212cbe84e3e466a00013ccb639 zach rice <zricer@protonmail.com> 1571925818 -0400	commit: comment

test_data/test_repos/test_repo_1/dotGit/objects/9e/523225b31add24e72f2feb0b2645cfb36542dc

@@ -0,0 +1,3 @@
+x
MŽO‚@Ĺ;ď§č`‘%:XjE	˘<Éş;�¤»°ł~űĚ
+z§÷`~ďMVęFţ´Ó…P�ŇnÜHZ"�•-xIL”Eca™¶Ú÷>±Ç ‘Cľă¶®«,�¬)TNŔéżČm7
+]U¨$JŻ%ů“R.Ąw¬ÓBÎś`łÖ‡q|÷űIt
vÇmôÝy_6ďŮ?¨!ć‹0Šb´ťÔW\>ε9ć	…ęr:-†Aň+a}Ć^K§N$

test_data/test_repos/test_repo_1/dotGit/objects/a1/fd29ec14823d8bc4a8d1a2cfe35451580f5118

@@ -0,0 +1,3 @@
+x
uPMOÃ0åÚüŠGV1lc.LEB`†Ä¸!Miç*a]R5!þ;N‡øââÄö{ÏÏÎ+—ãìôt§³;Ì�z-:¢ƒKzQ«º"hç–ðEcê€à°RKÂÚ\‰zí5\Œ³~À¬™¦/š7Ì~Ú‘‡ªª_h­|PMÀÆ�næªXf]Rv�†ž¨ßTVÙh²Ì$ÈmSþÔ„ñÑׂ�D+dUÎþƒæF\âˆ%­Z}VJ{²n¨ßPAæ™ä@S"�è[Èôz2›ß=Üß̧w³Éôv~5}¸�IÁ¬HLv,’�nu"C¦F¢_ÅÏ?¼dáD’гª 9¬){üeÆŠ/,Ó$axZòu ƒgm
+‹(·jÏòc•~Üøé[+Ñ9ÈÞ%.öN"s<æ¸=àVéÅŒâ¯
+~L–îïbÔãþÂY¥eMª$

test_data/test_repos/test_repo_1/dotGit/objects/a5/196d1be8fb59edf8062bef36d3a602e0812139

@@ -0,0 +1 @@
+x
-�±jÃ0„;ç)<¤
µSJ§B¦@¦¾Aé Û¿lI¤_¸Yú쑃·ãø¾»Þ³ÇûÇÛSƒÉ)|n­ó’ѶÔYÒäoðe”Ö&†SW©£‹–Ç­Þ5ør±:EV“‹Óûf“C�¨¹«ä…	×Ä_�Yý
.âüZu�¥÷\\œ°°ø½Tz"GdQÐÖ…íW£*)f<—¸=¬
ÎâÆ‚ÅD…%Ë£ù¬ò¡û¦ùYÃÿîá›SC