Experimental JGit-based GC Executor

James Moger

2012-10-28 e92c6d230b3a350749fdb9fa2150bb1773260b8c

Experimental JGit-based GC Executor

 distrib/gitblit.properties

@@ -108,6 +108,67 @@
# SINCE 1.1.0

git.defaultAuthorizationControl = NAMED



# Enable JGit-based garbage collection. (!!EXPERIMENTAL!!)

#

# If enabled, the garbage collection executor scans all repositories once a day

# at the hour of your choosing.  The GC executor will take each repository "offline",

# one-at-a-time, to check if the repository satisfies it's GC trigger requirements.

#

# While the repository is offline it will be inaccessible from the web UI or from

# any of the other services (git, rpc, rss, etc).

#

# Gitblit's GC Executor MAY NOT PLAY NICE with the other Git kids on the block,

# especially on Windows systems, so if you are using other tools please coordinate

# their usage with your GC Executor schedule or do not use this feature.

#

# Use this feature at your own risk!

#

# The GC algorithm complex and the JGit team advises caution when using their

# young implementation of GC.

#

# http://wiki.eclipse.org/EGit/New_and_Noteworthy/2.1#Garbage_Collector_and_Repository_Storage_Statistics

#

# EXPERIMENTAL

# SINCE 1.2.0

# RESTART REQUIRED

git.enableGarbageCollection = false



# Hour of the day for the GC Executor to scan repositories.

# This value is in 24-hour time.

#

# SINCE 1.2.0

git.garbageCollectionHour = 0



# The default minimum total filesize of loose objects to trigger early garbage

# collection.

#

# You may specify a custom threshold for a repository in the repository's settings.

# Common unit suffixes of k, m, or g are supported.

#

# SINCE 1.2.0

git.defaultGarbageCollectionThreshold = 500k



# The default period between GCs for a repository.  If the total filesize of the

# loose object exceeds *git.garbageCollectionThreshold* or the repository's

# custom threshold, this period will be short-circuited. 
#

# e.g. if a repository collects 100KB of loose objects every day with a 500KB

# threshold and a period of 7 days, it will take 5 days for the loose objects to

# be collected, packed, and pruned.

#

# OR

#

# if a repository collects 10KB of loose objects every day with a 500KB threshold

# and a period of 7 days, it will take the full 7 days for the loose objects to be

# collected, packed, and pruned.

#

# You may specify a custom period for a repository in the repository's settings.

#

# The minimum value is 1 day since the GC Executor only runs once a day.

#

# SINCE 1.2.0

git.defaultGarbageCollectionPeriod = 7 days



# Number of bytes of a pack file to load into memory in a single read operation.

# This is the "page size" of the JGit buffer cache, used for all pack access

# operations. All disk IO occurs as single window reads. Setting this too large


 docs/01_features.mkd

@@ -17,6 +17,7 @@
- Optional feature to allow users to create personal repositories

- Optional feature to fork a repository to a personal repository

- Optional feature to create a repository on push

- Experimental built-in Garbage Collection

- Ability to federate with one or more other Gitblit instances

- RSS/JSON RPC interface

- Java/Swing Gitblit Manager tool 

@@ -66,13 +67,6 @@


## Limitations

- HTTP/HTTPS are the only supported Git protocols

- Built-in access controls are not path-based, they are repository-based.  
- Only Administrators can create, rename or delete repositories

- Only Administrators can create, modify or delete users

- Only Administrators can create, modify or delete teams

- Native Git may be needed to periodically run git-gc as [JGit][jgit] does not fully support the git-gc featureset.



### Caveats

- Gitblit may have security holes.  Patches welcome.  :)

- Built-in access controls are not path-based, they are repository-based.



[jgit]: http://eclipse.org/jgit "Eclipse JGit Site"


 docs/04_releases.mkd

@@ -3,8 +3,7 @@
<div class="alert alert-info">

<h4>Update Note</h4>

The permissions model has changed in this release.

<p>

If you are updating your server, you must also update any Gitblit Manager and Federation Client installs to 1.2.0 as well.  The data model used by the RPC mechanism has changed slightly for the new permissions infrastructure.

<p>If you are updating your server, you must also update any Gitblit Manager and Federation Client installs to 1.2.0 as well.  The data model used by the RPC mechanism has changed slightly for the new permissions infrastructure.</p>

</div>



### Current Release

@@ -33,14 +32,19 @@
    - RW+ (clone and push with ref creation, deletion, rewind)  

While not as sophisticated as Gitolite, this does give finer access controls.  These permissions fit in cleanly with the existing users.conf and users.properties files.  In Gitblit <= 1.1.0, all your existing user accounts have RW+ access.   If you are upgrading to 1.2.0, the RW+ access is *preserved* and you will have to lower/adjust accordingly.

- Implemented *case-insensitive* regex repository permission matching (issue 36)  

This allows you to specify a permission like `RW:mygroup/[a-z0-9-~_\\./]+` to grant push privileges to all repositories within the *mygroup* project/folder.

This allows you to specify a permission like `RW:mygroup/.*` to grant push privileges to all repositories within the *mygroup* project/folder.

- Added DELETE, CREATE, and NON-FAST-FORWARD ref change logging

- Added support for personal repositories.  

Personal repositories can be created by accounts with the *create* permission and are stored in *git.repositoriesFolder/~username*.  Each user with personal repositories will have a user page, something like the GitHub profile page.  Personal repositories have all the same features as common repositories, except personal repositories can be renamed by their owner.

- Added support for server-side forking of a repository to a personal repository (issue 137)  

In order to fork a repository, the user account must have the *fork* permission **and** the repository must *allow forks*.  The clone inherits the access list of its origin.  i.e. if Team A has clone access to the origin repository, then by default Team A also has clone access to the fork.  This is to facilitate collaboration.  The fork owner may change access to the fork and add/remove users/teams, etc as required <u>however</u> it should be noted that all personal forks will be enumerated in the fork network regardless of access view restrictions.  If you really must have an invisible fork, the clone it locally, create a new repository for your invisible fork, and push it back to Gitblit.

- Added optional *create-on-push* support  

    **New:** *git.allowCreateOnPush=true*

    **New:** *git.allowCreateOnPush=true*  
- Added **experimental** JGit-based garbage collection service.  This service is disabled by default.  
    **New:** *git.allowGarbageCollection=false*  
    **New:** *git.garbageCollectionHour = 0*  
    **New:** *git.defaultGarbageCollectionThreshold = 500k*  
    **New:** *git.defaultGarbageCollectionPeriod = 7 days*

- Added simple project pages.  A project is a subfolder off the *git.repositoriesFolder*.

- Added support for X-Forwarded-Context for Apache subdomain proxy configurations (issue 135)

- Delete branch feature (issue 121, Github/ajermakovics)

@@ -50,6 +54,7 @@


#### changes



- Teams can now specify the *admin*, *create*, and *fork* roles to simplify user administration

- Use https Gravatar urls to avoid browser complaints

- Expose ReceivePack to Groovy push hooks (issue 125)

- Redirect to summary page when refreshing the empty repository page on a repository that is not empty (issue 129)


 src/com/gitblit/AccessRestrictionFilter.java

@@ -125,6 +125,12 @@


        String fullUrl = getFullUrl(httpRequest);

        String repository = extractRepositoryName(fullUrl);

		
        if (GitBlit.self().isCollectingGarbage(repository)) {

            logger.info(MessageFormat.format("ARF: Rejecting request for {0}, busy collecting garbage!", repository));

            httpResponse.sendError(HttpServletResponse.SC_FORBIDDEN);

            return;

        }



        // Determine if the request URL is restricted

        String fullSuffix = fullUrl.substring(repository.length());


 src/com/gitblit/Constants.java

@@ -86,6 +86,8 @@
    

    public static final String CONFIG_CUSTOM_FIELDS = "customFields";

    

    public static final String ISO8601 = "yyyy-MM-dd'T'HH:mm:ssZ";

	
    public static String getGitBlitVersion() {

        return NAME + " v" + VERSION;

    }

@@ -384,6 +386,14 @@
        REPOSITORY, USER, TEAM;

    }

    

    public static enum GCStatus {

        READY, COLLECTING;

		
        public boolean exceeds(GCStatus s) {

            return ordinal() > s.ordinal();

        }

    }

	
    @Documented

    @Retention(RetentionPolicy.RUNTIME)

    public @interface Unused {


 src/com/gitblit/DownloadZipServlet.java

@@ -101,11 +101,16 @@
            if (!StringUtils.isEmpty(objectId)) {

                name += "-" + objectId;

            }



						
            Repository r = GitBlit.self().getRepository(repository);

            if (r == null) {

                error(response, MessageFormat.format("# Error\nFailed to find repository {0}", repository));

                return;

                if (GitBlit.self().isCollectingGarbage(repository)) {

                    error(response, MessageFormat.format("# Error\nGitblit is busy collecting garbage in {0}", repository));

                    return;

                } else {

                    error(response, MessageFormat.format("# Error\nFailed to find repository {0}", repository));

                    return;

                }

            }

            RevCommit commit = JGitUtils.getCommit(r, objectId);

            if (commit == null) {


 src/com/gitblit/FederationPullExecutor.java

@@ -189,11 +189,17 @@
                            repositoryName.indexOf(DOT_GIT_EXT));

                }

            }



			
            // confirm that the origin of any pre-existing repository matches

            // the clone url

            String fetchHead = null;

            Repository existingRepository = GitBlit.self().getRepository(repositoryName);

			
            if (existingRepository == null && GitBlit.self().isCollectingGarbage(repositoryName)) {

                logger.warn(MessageFormat.format("Skipping local repository {0}, busy collecting garbage", repositoryName));

                continue;

            }



            if (existingRepository != null) {

                StoredConfig config = existingRepository.getConfig();

                config.load();


 src/com/gitblit/GCExecutor.java

New file
@@ -0,0 +1,229 @@
/*

 * Copyright 2012 gitblit.com.

 *

 * Licensed under the Apache License, Version 2.0 (the "License");

 * you may not use this file except in compliance with the License.

 * You may obtain a copy of the License at

 *

 *     http://www.apache.org/licenses/LICENSE-2.0

 *

 * Unless required by applicable law or agreed to in writing, software

 * distributed under the License is distributed on an "AS IS" BASIS,

 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

 * See the License for the specific language governing permissions and

 * limitations under the License.

 */

package com.gitblit;



import java.lang.reflect.Field;

import java.text.MessageFormat;

import java.util.Calendar;

import java.util.Date;

import java.util.Map;

import java.util.concurrent.ConcurrentHashMap;

import java.util.concurrent.atomic.AtomicBoolean;

import java.util.concurrent.atomic.AtomicInteger;



import org.eclipse.jgit.lib.Repository;

import org.eclipse.jgit.storage.file.FileRepository;

import org.eclipse.jgit.storage.file.GC;

import org.eclipse.jgit.storage.file.GC.RepoStatistics;

import org.slf4j.Logger;

import org.slf4j.LoggerFactory;



import com.gitblit.models.RepositoryModel;

import com.gitblit.utils.FileUtils;

import com.gitblit.utils.TimeUtils;



/**

 * The GC executor handles periodic garbage collection in repositories.

 * 
 * @author James Moger

 * 
 */

public class GCExecutor implements Runnable {



    public static enum GCStatus {

        READY, COLLECTING;

		
        public boolean exceeds(GCStatus s) {

            return ordinal() > s.ordinal();

        }

    }

    private final Logger logger = LoggerFactory.getLogger(GCExecutor.class);



    private final IStoredSettings settings;

	
    private AtomicBoolean forceClose = new AtomicBoolean(false);

	
    private final Map<String, GCStatus> gcCache = new ConcurrentHashMap<String, GCStatus>();



    public GCExecutor(IStoredSettings settings) {

        this.settings = settings;

    }



    /**

     * Indicates if the GC executor is ready to process repositories.

     * 
     * @return true if the GC executor is ready to process repositories

     */

    public boolean isReady() {

        return settings.getBoolean(Keys.git.enableGarbageCollection, false);

    }

	
    public boolean lock(String repositoryName) {

        return setGCStatus(repositoryName, GCStatus.COLLECTING);

    }



    /**

     * Tries to set a GCStatus for the specified repository.

     * 
     * @param repositoryName

     * @return true if the status has been set

     */

    private boolean setGCStatus(String repositoryName, GCStatus status) {

        String key = repositoryName.toLowerCase();

        if (gcCache.containsKey(key)) {

            if (gcCache.get(key).exceeds(GCStatus.READY)) {

                // already collecting or blocked

                return false;

            }

        }

        gcCache.put(key, status);

        return true;

    }



    /**

     * Returns true if Gitblit is actively collecting garbage in this repository.

     * 
     * @param repositoryName

     * @return true if actively collecting garbage

     */

    public boolean isCollectingGarbage(String repositoryName) {

        String key = repositoryName.toLowerCase();

        return gcCache.containsKey(key) && GCStatus.COLLECTING.equals(gcCache.get(key));

    }



    /**

     * Resets the GC status to ready.

     * 
     * @param repositoryName

     */

    public void releaseLock(String repositoryName) {

        gcCache.put(repositoryName.toLowerCase(), GCStatus.READY);

    }

	
    public void close() {

        forceClose.set(true);

    }



    @Override

    public void run() {

        if (!isReady()) {

            return;

        }

        Date now = new Date();



        for (String repositoryName : GitBlit.self().getRepositoryList()) {

            if (forceClose.get()) {

                break;

            }

            if (isCollectingGarbage(repositoryName)) {

                logger.warn(MessageFormat.format("Already collecting garbage from {0}?!?", repositoryName));

                continue;

            }

            boolean garbageCollected = false;

            RepositoryModel model = null;

            FileRepository repository = null;

            try {

                model = GitBlit.self().getRepositoryModel(repositoryName);

                repository = (FileRepository) GitBlit.self().getRepository(repositoryName);

                if (repository == null) {

                    logger.warn(MessageFormat.format("GCExecutor is missing repository {0}?!?", repositoryName));

                    continue;

                }

				
                if (!isRepositoryIdle(repository)) {

                    logger.debug(MessageFormat.format("GCExecutor is skipping {0} because it is not idle", repositoryName));

                    continue;

                }



                // By setting the GCStatus to COLLECTING we are

                // disabling *all* access to this repository from Gitblit.

                // Think of this as a clutch in a manual transmission vehicle.

                if (!setGCStatus(repositoryName, GCStatus.COLLECTING)) {

                    logger.warn(MessageFormat.format("Can not acquire GC lock for {0}, skipping", repositoryName));

                    continue;

                }

				
                logger.debug(MessageFormat.format("GCExecutor locked idle repository {0}", repositoryName));

				
                GC gc = new GC(repository);

                RepoStatistics stats = gc.getStatistics();

				
                // determine if this is a scheduled GC

                int gcPeriodInDays = TimeUtils.convertFrequencyToMinutes(model.gcPeriod)/(60*24);

                Calendar cal = Calendar.getInstance();

                cal.setTime(model.lastGC);

                cal.set(Calendar.HOUR_OF_DAY, 0);

                cal.set(Calendar.MINUTE, 0);

                cal.set(Calendar.SECOND, 0);

                cal.set(Calendar.MILLISECOND, 0);

                cal.add(Calendar.DATE, gcPeriodInDays);

                Date gcDate = cal.getTime();

                boolean shouldCollectGarbage = now.after(gcDate);



                // determine if filesize triggered GC

                long gcThreshold = FileUtils.convertSizeToLong(model.gcThreshold, 500*1024L);

                boolean hasEnoughGarbage = stats.sizeOfLooseObjects >= gcThreshold;



                // if we satisfy one of the requirements, GC

                boolean hasGarbage = stats.sizeOfLooseObjects > 0;

                if (hasGarbage && (hasEnoughGarbage || shouldCollectGarbage)) {

                    long looseKB = stats.sizeOfLooseObjects/1024L;

                    logger.info(MessageFormat.format("Collecting {1} KB of loose objects from {0}", repositoryName, looseKB));

					
                    // do the deed

                    gc.gc();

					
                    garbageCollected = true;

                }

            } catch (Exception e) {

                logger.error("Error collecting garbage in " + repositoryName, e);

            } finally {

                // cleanup

                if (repository != null) {

                    if (garbageCollected) {

                        // update the last GC date

                        model.lastGC = new Date();

                        GitBlit.self().updateConfiguration(repository, model);

                    }

				
                    repository.close();

                }

				
                // reset the GC lock 
                releaseLock(repositoryName);

                logger.debug(MessageFormat.format("GCExecutor released GC lock for {0}", repositoryName));

            }

        }

    }

	
    private boolean isRepositoryIdle(FileRepository repository) {

        try {

            // Read the use count.

            // An idle use count is 2:

            // +1 for being in the cache

            // +1 for the repository parameter in this method

            Field useCnt = Repository.class.getDeclaredField("useCnt");

            useCnt.setAccessible(true);

            int useCount = ((AtomicInteger) useCnt.get(repository)).get();

            return useCount == 2;

        } catch (Exception e) {

            logger.warn(MessageFormat

                    .format("Failed to reflectively determine use count for repository {0}",

                            repository.getDirectory().getPath()), e);

        }

        return false;

    }

}


 src/com/gitblit/GitBlit.java

@@ -28,6 +28,7 @@
import java.text.SimpleDateFormat;

import java.util.ArrayList;

import java.util.Arrays;

import java.util.Calendar;

import java.util.Collection;

import java.util.Collections;

import java.util.Date;

@@ -103,6 +104,7 @@
import com.gitblit.utils.MetricUtils;

import com.gitblit.utils.ObjectCache;

import com.gitblit.utils.StringUtils;

import com.gitblit.utils.TimeUtils;

import com.gitblit.wicket.WicketUtils;



/**

@@ -159,6 +161,8 @@
    private MailExecutor mailExecutor;

    

    private LuceneExecutor luceneExecutor;

	
    private GCExecutor gcExecutor;

    

    private TimeZone timezone;

    

@@ -250,6 +254,34 @@
     */

    public static int getInteger(String key, int defaultValue) {

        return self().settings.getInteger(key, defaultValue);

    }

	
    /**

     * Returns the value in bytes for the specified key. If the key does not

     * exist or the value for the key can not be interpreted as an integer, the

     * defaultValue is returned.

     * 
     * @see IStoredSettings.getFilesize(String key, int defaultValue)

     * @param key

     * @param defaultValue

     * @return key value or defaultValue

     */

    public static int getFilesize(String key, int defaultValue) {

        return self().settings.getFilesize(key, defaultValue);

    }



    /**

     * Returns the value in bytes for the specified key. If the key does not

     * exist or the value for the key can not be interpreted as a long, the

     * defaultValue is returned.

     * 
     * @see IStoredSettings.getFilesize(String key, long defaultValue)

     * @param key

     * @param defaultValue

     * @return key value or defaultValue

     */

    public static long getFilesize(String key, long defaultValue) {

        return self().settings.getFilesize(key, defaultValue);

    }



    /**

@@ -1018,10 +1050,15 @@
     * @return repository or null

     */

    public Repository getRepository(String repositoryName, boolean logError) {

        if (isCollectingGarbage(repositoryName)) {

            logger.warn(MessageFormat.format("Rejecting request for {0}, busy collecting garbage!", repositoryName));

            return null;

        }



        File dir = FileKey.resolve(new File(repositoriesFolder, repositoryName), FS.DETECTED);

        if (dir == null)

            return null;



		
        Repository r = null;

        try {

            FileKey key = FileKey.exact(dir, FS.DETECTED);

@@ -1115,7 +1152,14 @@
        

        // cached model

        RepositoryModel model = repositoryListCache.get(repositoryName);

		


        if (gcExecutor.isCollectingGarbage(model.name)) {

            // Gitblit is busy collecting garbage, use our cached model

            RepositoryModel rm = DeepCopier.copy(model);

            rm.isCollectingGarbage = true;

            return rm;

        }



        // check for updates

        Repository r = getRepository(repositoryName);

        if (r == null) {

@@ -1180,12 +1224,6 @@
                }

                project.title = projectConfigs.getString("project", name, "title");

                project.description = projectConfigs.getString("project", name, "description");

                // TODO add more interesting metadata

                // project manager?

                // commit message regex?

                // RW+

                // RW

                // R

                configs.put(name.toLowerCase(), project);                

            }

            projectCache.clear();

@@ -1379,6 +1417,13 @@
            model.federationSets = new ArrayList<String>(Arrays.asList(config.getStringList(

                    Constants.CONFIG_GITBLIT, null, "federationSets")));

            model.isFederated = getConfig(config, "isFederated", false);

            model.gcThreshold = getConfig(config, "gcThreshold", settings.getString(Keys.git.defaultGarbageCollectionThreshold, "500KB"));

            model.gcPeriod = getConfig(config, "gcPeriod", settings.getString(Keys.git.defaultGarbageCollectionPeriod, "7 days"));

            try {

                model.lastGC = new SimpleDateFormat(Constants.ISO8601).parse(getConfig(config, "lastGC", "1970-01-01'T'00:00:00Z"));

            } catch (Exception e) {

                model.lastGC = new Date(0);

            }

            model.origin = config.getString("remote", "origin", "url");

            if (model.origin != null) {

                model.origin = model.origin.replace('\\', '/');

@@ -1675,6 +1720,10 @@
     */

    public void updateRepositoryModel(String repositoryName, RepositoryModel repository,

            boolean isCreate) throws GitBlitException {

        if (gcExecutor.isCollectingGarbage(repositoryName)) {

            throw new GitBlitException(MessageFormat.format("sorry, Gitblit is busy collecting garbage in {0}",

                    repositoryName));

        }

        Repository r = null;

        String projectPath = StringUtils.getFirstPathElement(repository.name);

        if (!StringUtils.isEmpty(projectPath)) {

@@ -1819,6 +1868,9 @@
        config.setString(Constants.CONFIG_GITBLIT, null, "federationStrategy",

                repository.federationStrategy.name());

        config.setBoolean(Constants.CONFIG_GITBLIT, null, "isFederated", repository.isFederated);

        config.setString(Constants.CONFIG_GITBLIT, null, "gcThreshold", repository.gcThreshold);

        config.setString(Constants.CONFIG_GITBLIT, null, "gcPeriod", repository.gcPeriod);

        config.setString(Constants.CONFIG_GITBLIT, null, "lastGC", new SimpleDateFormat(Constants.ISO8601).format(repository.lastGC));



        updateList(config, "federationSets", repository.federationSets);

        updateList(config, "preReceiveScript", repository.preReceiveScripts);

@@ -2614,6 +2666,12 @@
    public void configureContext(IStoredSettings settings, boolean startFederation) {

        logger.info("Reading configuration from " + settings.toString());

        this.settings = settings;

		
        // prepare service executors

        mailExecutor = new MailExecutor(settings);

        luceneExecutor = new LuceneExecutor(settings, repositoriesFolder);

        gcExecutor = new GCExecutor(settings);

		
        repositoriesFolder = getRepositoriesFolder();

        logger.info("Git repositories folder " + repositoriesFolder.getAbsolutePath());



@@ -2647,16 +2705,43 @@
        // load and cache the project metadata

        projectConfigs = new FileBasedConfig(getFileOrFolder(Keys.web.projectsFile, "projects.conf"), FS.detect());

        getProjectConfigs();

        mailExecutor = new MailExecutor(settings);

		
        // schedule mail engine

        if (mailExecutor.isReady()) {

            logger.info("Mail executor is scheduled to process the message queue every 2 minutes.");

            scheduledExecutor.scheduleAtFixedRate(mailExecutor, 1, 2, TimeUnit.MINUTES);

        } else {

            logger.warn("Mail server is not properly configured.  Mail services disabled.");

        }

        luceneExecutor = new LuceneExecutor(settings, repositoriesFolder);

		
        // schedule lucene engine

        logger.info("Lucene executor is scheduled to process indexed branches every 2 minutes.");

        scheduledExecutor.scheduleAtFixedRate(luceneExecutor, 1, 2, TimeUnit.MINUTES);

		
        // schedule gc engine

        if (gcExecutor.isReady()) {

            logger.info("GC executor is scheduled to scan repositories every 24 hours.");

            Calendar c = Calendar.getInstance();

            c.set(Calendar.HOUR_OF_DAY, settings.getInteger(Keys.git.garbageCollectionHour, 0));

            c.set(Calendar.MINUTE, 0);

            c.set(Calendar.SECOND, 0);

            c.set(Calendar.MILLISECOND, 0);

            Date cd = c.getTime();

            Date now = new Date();

            int delay = 0;

            if (cd.before(now)) {

                c.add(Calendar.DATE, 1);

                cd = c.getTime();

            }

            delay = (int) ((cd.getTime() - now.getTime())/TimeUtils.MIN);

            String when = delay + " mins";

            if (delay > 60) {

                when = MessageFormat.format("{0,number,0.0} hours", ((float)delay)/60f);

            }

            logger.info(MessageFormat.format("Next scheculed GC scan is in {0}", when));

            scheduledExecutor.scheduleAtFixedRate(gcExecutor, delay, 60*24, TimeUnit.MINUTES);

        }

		
        if (startFederation) {

            configureFederation();

        }

@@ -2758,9 +2843,20 @@
        logger.info("Gitblit context destroyed by servlet container.");

        scheduledExecutor.shutdownNow();

        luceneExecutor.close();

        gcExecutor.close();

    }

    

    /**

     * Returns true if Gitblit is actively collecting garbage in this repository.

     * 
     * @param repositoryName

     * @return true if actively collecting garbage

     */

    public boolean isCollectingGarbage(String repositoryName) {

        return gcExecutor.isCollectingGarbage(repositoryName);

    }



    /**

     * Creates a personal fork of the specified repository. The clone is view

     * restricted by default and the owner of the source repository is given

     * access to the clone. 


 src/com/gitblit/LuceneExecutor.java

@@ -171,6 +171,12 @@
            RepositoryModel model = GitBlit.self().getRepositoryModel(repositoryName);

            if (model.hasCommits && !ArrayUtils.isEmpty(model.indexedBranches)) {

                Repository repository = GitBlit.self().getRepository(model.name);

                if (repository == null) {

                    if (GitBlit.self().isCollectingGarbage(model.name)) {

                        logger.info(MessageFormat.format("Skipping Lucene index of {0}, busy garbage collecting", repositoryName));

                    }

                    continue;

                }

                index(model, repository);                

                repository.close();

                System.gc();


 src/com/gitblit/RpcServlet.java

@@ -110,6 +110,11 @@
                    // skip empty repository

                    continue;

                }

                if (model.isCollectingGarbage) {

                    // skip garbage collecting repository

                    logger.warn(MessageFormat.format("Temporarily excluding {0} from RPC, busy collecting garbage", model.name));

                    continue;

                }

                // get local branches

                Repository repository = GitBlit.self().getRepository(model.name);

                List<RefModel> refs = JGitUtils.getLocalBranches(repository, false, -1);


 src/com/gitblit/SyndicationServlet.java

@@ -210,7 +210,13 @@
        for (String name : repositories) {

            Repository repository = GitBlit.self().getRepository(name);

            RepositoryModel model = GitBlit.self().getRepositoryModel(name);

			


            if (repository == null) {

                if (model.isCollectingGarbage) {

                    logger.warn(MessageFormat.format("Temporarily excluding {0} from feed, busy collecting garbage", name));

                } 
                continue;

            }

            if (!isProjectFeed) {

                // single-repository feed

                feedName = model.name;


 src/com/gitblit/models/RepositoryModel.java

@@ -76,6 +76,11 @@
    public Set<String> forks;

    public String originRepository;

    public boolean verifyCommitter;

    public String gcThreshold;

    public String gcPeriod;

	
    public transient boolean isCollectingGarbage;

    public Date lastGC;

    

    public RepositoryModel() {

        this("", "", "", new Date(0));


 src/com/gitblit/utils/ActivityUtils.java

@@ -82,6 +82,9 @@
        Map<String, Activity> activity = new HashMap<String, Activity>();

        for (RepositoryModel model : models) {

            if (model.hasCommits && model.lastChange.after(thresholdDate)) {

                if (model.isCollectingGarbage) {

                    continue;

                }

                Repository repository = GitBlit.self()

                        .getRepository(model.name);

                List<String> branches = new ArrayList<String>();


 src/com/gitblit/wicket/GitBlitWebApp.properties

@@ -357,4 +357,10 @@
gb.deletePermission = {0} (push, ref creation+deletion)

gb.rewindPermission = {0} (push, ref creation+deletion+rewind)

gb.permission = permission

gb.regexPermission = this permission is set from a regular expression
gb.regexPermission = this permission is set from a regular expression

gb.accessDenied = access denied

gb.busyCollectingGarbage = sorry, Gitblit is busy collecting garbage in {0}

gb.gcPeriod = GC period

gb.gcPeriodDescription = duration between garbage collections

gb.gcThreshold = GC threshold

gb.gcThresholdDescription = minimum total size of loose objects to trigger early garbage collection


 src/com/gitblit/wicket/pages/EditRepositoryPage.html

@@ -28,15 +28,17 @@
                <tr><th><wicket:message key="gb.name"></wicket:message></th><td class="edit"><input class="span4" type="text" wicket:id="name" id="name" size="40" tabindex="1" /> &nbsp;<span class="help-inline"><wicket:message key="gb.nameDescription"></wicket:message></span></td></tr>

                <tr><th><wicket:message key="gb.description"></wicket:message></th><td class="edit"><input class="span4" type="text" wicket:id="description" size="40" tabindex="2" /></td></tr>

                <tr><th><wicket:message key="gb.origin"></wicket:message></th><td class="edit"><input class="span5" type="text" wicket:id="origin" size="80" tabindex="3" /></td></tr>

                <tr><th><wicket:message key="gb.headRef"></wicket:message></th><td class="edit"><select wicket:id="HEAD" tabindex="4" /> &nbsp;<span class="help-inline"><wicket:message key="gb.headRefDescription"></wicket:message></span></td></tr>

                <tr><th><wicket:message key="gb.owner"></wicket:message></th><td class="edit"><select wicket:id="owner" tabindex="5" /> &nbsp;<span class="help-inline"><wicket:message key="gb.ownerDescription"></wicket:message></span></td></tr>

                <tr><th><wicket:message key="gb.enableTickets"></wicket:message></th><td class="edit"><label class="checkbox"><input type="checkbox" wicket:id="useTickets" tabindex="6" /> &nbsp;<span class="help-inline"><wicket:message key="gb.useTicketsDescription"></wicket:message></span></label></td></tr>

                <tr><th><wicket:message key="gb.enableDocs"></wicket:message></th><td class="edit"><label class="checkbox"><input type="checkbox" wicket:id="useDocs" tabindex="7" /> &nbsp;<span class="help-inline"><wicket:message key="gb.useDocsDescription"></wicket:message></span></label></td></tr>

                <tr><th><wicket:message key="gb.showRemoteBranches"></wicket:message></th><td class="edit"><label class="checkbox"><input type="checkbox" wicket:id="showRemoteBranches" tabindex="8" /> &nbsp;<span class="help-inline"><wicket:message key="gb.showRemoteBranchesDescription"></wicket:message></span></label></td></tr>

                <tr><th><wicket:message key="gb.showReadme"></wicket:message></th><td class="edit"><label class="checkbox"><input type="checkbox" wicket:id="showReadme" tabindex="9" /> &nbsp;<span class="help-inline"><wicket:message key="gb.showReadmeDescription"></wicket:message></span></label></td></tr>

                <tr><th><wicket:message key="gb.skipSizeCalculation"></wicket:message></th><td class="edit"><label class="checkbox"><input type="checkbox" wicket:id="skipSizeCalculation" tabindex="10" /> &nbsp;<span class="help-inline"><wicket:message key="gb.skipSizeCalculationDescription"></wicket:message></span></label></td></tr>

                <tr><th><wicket:message key="gb.skipSummaryMetrics"></wicket:message></th><td class="edit"><label class="checkbox"><input type="checkbox" wicket:id="skipSummaryMetrics" tabindex="11" /> &nbsp;<span class="help-inline"><wicket:message key="gb.skipSummaryMetricsDescription"></wicket:message></span></label></td></tr>

                <tr><th><wicket:message key="gb.mailingLists"></wicket:message></th><td class="edit"><input class="span8" type="text" wicket:id="mailingLists" size="40" tabindex="13" /></td></tr>

                <tr><th><wicket:message key="gb.headRef"></wicket:message></th><td class="edit"><select class="span3" wicket:id="HEAD" tabindex="4" /> &nbsp;<span class="help-inline"><wicket:message key="gb.headRefDescription"></wicket:message></span></td></tr>

                <tr><th><wicket:message key="gb.owner"></wicket:message></th><td class="edit"><select class="span2" wicket:id="owner" tabindex="5" /> &nbsp;<span class="help-inline"><wicket:message key="gb.ownerDescription"></wicket:message></span></td></tr>

                <tr><th><wicket:message key="gb.gcPeriod"></wicket:message></th><td class="edit"><select class="span2" wicket:id="gcPeriod" tabindex="6" /> &nbsp;<span class="help-inline"><wicket:message key="gb.gcPeriodDescription"></wicket:message></span></td></tr>

                <tr><th><wicket:message key="gb.gcThreshold"></wicket:message></th><td class="edit"><input class="span1" type="text" wicket:id="gcThreshold" tabindex="7" /> &nbsp;<span class="help-inline"><wicket:message key="gb.gcThresholdDescription"></wicket:message></span></td></tr>

                <tr><th><wicket:message key="gb.enableTickets"></wicket:message></th><td class="edit"><label class="checkbox"><input type="checkbox" wicket:id="useTickets" tabindex="8" /> &nbsp;<span class="help-inline"><wicket:message key="gb.useTicketsDescription"></wicket:message></span></label></td></tr>

                <tr><th><wicket:message key="gb.enableDocs"></wicket:message></th><td class="edit"><label class="checkbox"><input type="checkbox" wicket:id="useDocs" tabindex="9" /> &nbsp;<span class="help-inline"><wicket:message key="gb.useDocsDescription"></wicket:message></span></label></td></tr>

                <tr><th><wicket:message key="gb.showRemoteBranches"></wicket:message></th><td class="edit"><label class="checkbox"><input type="checkbox" wicket:id="showRemoteBranches" tabindex="10" /> &nbsp;<span class="help-inline"><wicket:message key="gb.showRemoteBranchesDescription"></wicket:message></span></label></td></tr>

                <tr><th><wicket:message key="gb.showReadme"></wicket:message></th><td class="edit"><label class="checkbox"><input type="checkbox" wicket:id="showReadme" tabindex="11" /> &nbsp;<span class="help-inline"><wicket:message key="gb.showReadmeDescription"></wicket:message></span></label></td></tr>

                <tr><th><wicket:message key="gb.skipSizeCalculation"></wicket:message></th><td class="edit"><label class="checkbox"><input type="checkbox" wicket:id="skipSizeCalculation" tabindex="12" /> &nbsp;<span class="help-inline"><wicket:message key="gb.skipSizeCalculationDescription"></wicket:message></span></label></td></tr>

                <tr><th><wicket:message key="gb.skipSummaryMetrics"></wicket:message></th><td class="edit"><label class="checkbox"><input type="checkbox" wicket:id="skipSummaryMetrics" tabindex="13" /> &nbsp;<span class="help-inline"><wicket:message key="gb.skipSummaryMetricsDescription"></wicket:message></span></label></td></tr>

                <tr><th><wicket:message key="gb.mailingLists"></wicket:message></th><td class="edit"><input class="span8" type="text" wicket:id="mailingLists" size="40" tabindex="14" /></td></tr>

            </tbody>

        </table>

        </div>

@@ -45,18 +47,18 @@
        <div class="tab-pane" id="permissions">

            <table class="plain">

                <tbody class="settings">

                    <tr><th><wicket:message key="gb.accessRestriction"></wicket:message></th><td class="edit"><select class="span4" wicket:id="accessRestriction" tabindex="13" /></td></tr>

                    <tr><th><wicket:message key="gb.accessRestriction"></wicket:message></th><td class="edit"><select class="span4" wicket:id="accessRestriction" tabindex="15" /></td></tr>

                    <tr><th colspan="2"><hr/></th></tr>

                    <tr><th><wicket:message key="gb.authorizationControl"></wicket:message></th><td style="padding:2px;">

                        <wicket:container wicket:id="authorizationControl">

                            <label class="radio"><input type="radio" wicket:id="allowAuthenticated" tabindex="14" /> &nbsp;<span class="help-inline"><wicket:message key="gb.allowAuthenticatedDescription"></wicket:message></span></label>

                            <label class="radio"><input type="radio" wicket:id="allowNamed" tabindex="15" /> &nbsp;<span class="help-inline"><wicket:message key="gb.allowNamedDescription"></wicket:message></span></label>

                            <label class="radio"><input type="radio" wicket:id="allowAuthenticated" tabindex="16" /> &nbsp;<span class="help-inline"><wicket:message key="gb.allowAuthenticatedDescription"></wicket:message></span></label>

                            <label class="radio"><input type="radio" wicket:id="allowNamed" tabindex="17" /> &nbsp;<span class="help-inline"><wicket:message key="gb.allowNamedDescription"></wicket:message></span></label>

                        </wicket:container>

                    </td></tr>

                    <tr><th colspan="2"><hr/></th></tr>

                    <tr><th><wicket:message key="gb.isFrozen"></wicket:message></th><td class="edit"><label class="checkbox"><input type="checkbox" wicket:id="isFrozen" tabindex="16" /> &nbsp;<span class="help-inline"><wicket:message key="gb.isFrozenDescription"></wicket:message></span></label></td></tr>

                    <tr><th><wicket:message key="gb.allowForks"></wicket:message></th><td class="edit"><label class="checkbox"><input type="checkbox" wicket:id="allowForks" tabindex="17" /> &nbsp;<span class="help-inline"><wicket:message key="gb.allowForksDescription"></wicket:message></span></label></td></tr>

                    <tr><th><wicket:message key="gb.verifyCommitter"></wicket:message></th><td class="edit"><label class="checkbox"><input type="checkbox" wicket:id="verifyCommitter" tabindex="18" /> &nbsp;<span class="help-inline"><wicket:message key="gb.verifyCommitterDescription"></wicket:message></span><br/><span class="help-inline" style="padding-left:10px;"><wicket:message key="gb.verifyCommitterNote"></wicket:message></span></label></td></tr>

                    <tr><th><wicket:message key="gb.isFrozen"></wicket:message></th><td class="edit"><label class="checkbox"><input type="checkbox" wicket:id="isFrozen" tabindex="18" /> &nbsp;<span class="help-inline"><wicket:message key="gb.isFrozenDescription"></wicket:message></span></label></td></tr>

                    <tr><th><wicket:message key="gb.allowForks"></wicket:message></th><td class="edit"><label class="checkbox"><input type="checkbox" wicket:id="allowForks" tabindex="19" /> &nbsp;<span class="help-inline"><wicket:message key="gb.allowForksDescription"></wicket:message></span></label></td></tr>

                    <tr><th><wicket:message key="gb.verifyCommitter"></wicket:message></th><td class="edit"><label class="checkbox"><input type="checkbox" wicket:id="verifyCommitter" tabindex="20" /> &nbsp;<span class="help-inline"><wicket:message key="gb.verifyCommitterDescription"></wicket:message></span><br/><span class="help-inline" style="padding-left:10px;"><wicket:message key="gb.verifyCommitterNote"></wicket:message></span></label></td></tr>

                    <tr><th colspan="2"><hr/></th></tr>

                    <tr><th><wicket:message key="gb.userPermissions"></wicket:message></th><td style="padding:2px;"><span wicket:id="users"></span></td></tr>

                    <tr><th colspan="2"><hr/></th></tr>

@@ -69,7 +71,7 @@
        <div class="tab-pane" id="federation">

            <table class="plain">

                <tbody class="settings">

                    <tr><th><wicket:message key="gb.federationStrategy"></wicket:message></th><td class="edit"><select class="span4" wicket:id="federationStrategy" tabindex="19" /></td></tr>

                    <tr><th><wicket:message key="gb.federationStrategy"></wicket:message></th><td class="edit"><select class="span4" wicket:id="federationStrategy" tabindex="21" /></td></tr>

                    <tr><th><wicket:message key="gb.federationSets"></wicket:message></th><td style="padding:2px;"><span wicket:id="federationSets"></span></td></tr>

                </tbody>

            </table>


 src/com/gitblit/wicket/pages/EditRepositoryPage.java

@@ -379,6 +379,10 @@
        }

        form.add(new DropDownChoice<String>("HEAD", availableRefs).setEnabled(availableRefs.size() > 0));



        List<String> gcPeriods = Arrays.asList("1 day", "2 days", "3 days", "4 days", "5 days", "7 days", "10 days", "14 days");

        form.add(new DropDownChoice<String>("gcPeriod", gcPeriods));

        form.add(new TextField<String>("gcThreshold"));



        // federation strategies - remove ORIGIN choice if this repository has

        // no origin.

        List<FederationStrategy> federationStrategies = new ArrayList<FederationStrategy>(


 src/com/gitblit/wicket/pages/RepositoryPage.java

@@ -92,6 +92,18 @@
        }

        objectId = WicketUtils.getObject(params);

        

        if (StringUtils.isEmpty(repositoryName)) {

            error(MessageFormat.format(getString("gb.repositoryNotSpecifiedFor"), getPageName()), true);

        }



        if (!getRepositoryModel().hasCommits) {

            setResponsePage(EmptyRepositoryPage.class, params);

        }

		
        if (getRepositoryModel().isCollectingGarbage) {

            error(MessageFormat.format(getString("gb.busyCollectingGarbage"), getRepositoryModel().name), true);

        }



        if (objectId != null) {

            RefModel branch = null;

            if ((branch = JGitUtils.getBranch(getRepository(), objectId)) != null) {

@@ -103,16 +115,9 @@
                boolean canAccess = user.hasBranchPermission(repositoryName,

                                branch.reference.getName());

                if (!canAccess) {

                    error("Access denied", true);

                    error(getString("gb.accessDeined"), true);

                }

            }

        }

        if (StringUtils.isEmpty(repositoryName)) {

            error(MessageFormat.format(getString("gb.repositoryNotSpecifiedFor"), getPageName()), true);

        }



        if (!getRepositoryModel().hasCommits) {

            setResponsePage(EmptyRepositoryPage.class, params);

        }



        // register the available page links for this page and user


 src/com/gitblit/wicket/panels/BranchesPanel.java

@@ -166,6 +166,14 @@
            @Override

            public void onClick() {

                Repository r = GitBlit.self().getRepository(repositoryModel.name);

                if (r == null) {

                    if (GitBlit.self().isCollectingGarbage(repositoryModel.name)) {

                        error(MessageFormat.format(getString("gb.busyCollectingGarbage"), repositoryModel.name));

                    } else {

                        error(MessageFormat.format("Failed to find repository {0}", repositoryModel.name));

                    }

                    return;

                }

                boolean success = JGitUtils.deleteBranchRef(r, entry.getName());

                r.close();

                if (success) {